apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-09-26 20:41:53 +08:00
Code Issues Actions 5 Packages Projects Releases Wiki Activity

更新文档 (#3996)

Browse Source
This commit is contained in:
yangjianfengo1
2025-09-09 10:23:51 +08:00
committed by GitHub
parent 934071578a
commit 14df2c59da
4 changed files with 65 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
README.md
View File

@@ -26,6 +26,8 @@ English | [简体中文](README_CN.md)
# FastDeploy : Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
## News
**[2025-09] 🔥 FastDeploy v2.2 is newly released!** It now offers compatibility with models in the HuggingFace ecosystem, has further optimized performance, and newly adds support for [baidu/ERNIE-21B-A3B-Thinking](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking)!
**[2025-08] 🔥 Released FastDeploy v2.1:** A brand-new KV Cache scheduling strategy has been introduced, and expanded support for PD separation and CUDA Graph across more models. Enhanced hardware support has been added for platforms like Kunlun and Hygon, along with comprehensive optimizations to improve the performance of both the service and inference engine.
**[2025-07] The FastDeploy 2.0 Inference Deployment Challenge is now live!** Complete the inference deployment task for the ERNIE 4.5 series open-source models to win official FastDeploy 2.0 merch and generous prizes! 🎁 You're welcome to try it out and share your feedback! 📌[Sign up here](https://www.wjx.top/vm/meSsp3L.aspx#) 📌[Event details](https://github.com/PaddlePaddle/FastDeploy/discussions/2728)
@@ -57,8 +59,9 @@ FastDeploy supports inference deployment on **NVIDIA GPUs**, **Kunlunxin XPUs**,
- [Iluvatar GPU](./docs/get_started/installation/iluvatar_gpu.md)
- [Enflame GCU](./docs/get_started/installation/Enflame_gcu.md)
- [Hygon DCU](./docs/get_started/installation/hygon_dcu.md)
- [MetaX GPU](./docs/get_started/installation/metax_gpu.md.md)
**Note:** We are actively working on expanding hardware support. Additional hardware platforms including Ascend NPU and MetaX GPU are currently under development and testing. Stay tuned for updates!
**Note:** We are actively working on expanding hardware support. Additional hardware platforms including Ascend NPU are currently under development and testing. Stay tuned for updates!
## Get Started
@@ -68,20 +71,12 @@ Learn how to use FastDeploy through our documentation:
- [ERNIE-4.5-VL Multimodal Model Deployment](./docs/get_started/ernie-4.5-vl.md)
- [Offline Inference Development](./docs/offline_inference.md)
- [Online Service Deployment](./docs/online_serving/README.md)
- [Full Supported Models List](./docs/supported_models.md)
- [Best Practices](./docs/best_practices/README.md)
## Supported Models
| Model | Data Type | PD Disaggregation | Chunked Prefill | Prefix Caching | MTP | CUDA Graph | Maximum Context Length |
|:--- | :------- | :---------- | :-------- | :-------- | :----- | :----- | :----- |
|ERNIE-4.5-300B-A47B | BF16/WINT4/WINT8/W4A8C8/WINT2/FP8 | ✅| ✅ | ✅|✅| ✅ |128K |
|ERNIE-4.5-300B-A47B-Base| BF16/WINT4/WINT8 | ✅| ✅ | ✅|❌| ✅ | 128K |
|ERNIE-4.5-VL-424B-A47B | BF16/WINT4/WINT8 | WIP | ✅ | WIP | ❌ | WIP |128K |
|ERNIE-4.5-VL-28B-A3B | BF16/WINT4/WINT8 | ❌ | ✅ | WIP | ❌ | WIP |128K |
|ERNIE-4.5-21B-A3B | BF16/WINT4/WINT8/FP8 | ❌ | ✅ | ✅ | ✅ | ✅|128K |
|ERNIE-4.5-21B-A3B-Base | BF16/WINT4/WINT8/FP8 | ✅ | ✅ | ✅ | ❌ | ✅|128K |
|ERNIE-4.5-0.3B | BF16/WINT8/FP8 | ✅ | ✅ | ✅ | ❌ | ✅| 128K |
Learn how to download models, enable using the torch format, and more:
- [Full Supported Models List](./docs/supported_models.md)
## Advanced Usage

19
README_CN.md
View File

@@ -26,7 +26,9 @@
# FastDeploy ：基于飞桨的大语言模型与视觉语言模型推理部署工具包
## 最新活动
**[2025-08] 🔥 FastDeploy v2.1 全新发布:** 全新的KV Cache调度策略，更多模型支持PD分离和CUDA Graph，昆仑、海光等更多硬件支持增强，全方面优化服务和推理引擎的性能。
**[2025-09] 🔥 FastDeploy v2.2 全新发布**: HuggingFace生态模型兼容，性能进一步优化，更新增对[baidu/ERNIE-21B-A3B-Thinking](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking)支持!
**[2025-08] FastDeploy v2.1 发布**:全新的KV Cache调度策略，更多模型支持PD分离和CUDA Graph，昆仑、海光等更多硬件支持增强，全方面优化服务和推理引擎的性能。
**[2025-07] 《FastDeploy2.0推理部署实测》专题活动已上线!** 完成文心4.5系列开源模型的推理部署等任务，即可获得骨瓷马克杯等FastDeploy2.0官方周边及丰富奖金！🎁 欢迎大家体验反馈～ 📌[报名地址](https://www.wjx.top/vm/meSsp3L.aspx#) 📌[活动详情](https://github.com/PaddlePaddle/FastDeploy/discussions/2728)
@@ -55,8 +57,9 @@ FastDeploy 支持在**英伟达（NVIDIA）GPU**、**昆仑芯（Kunlunxin）XPU
- [天数 CoreX](./docs/zh/get_started/installation/iluvatar_gpu.md)
- [燧原 S60](./docs/zh/get_started/installation/Enflame_gcu.md)
- [海光 DCU](./docs/zh/get_started/installation/hygon_dcu.md)
- [沐曦 GPU](./docs/zh/get_started/installation/metax_gpu.md.md)
**注意:** 我们正在积极拓展硬件支持范围。目前，包括昇腾（Ascend）NPU 和 沐曦（MetaX）GPU 在内的其他硬件平台正在开发测试中。敬请关注更新！
**注意:** 我们正在积极拓展硬件支持范围。目前，包括昇腾（Ascend）NPU 等其他硬件平台正在开发测试中。敬请关注更新！
## 入门指南
@@ -66,20 +69,12 @@ FastDeploy 支持在**英伟达（NVIDIA）GPU**、**昆仑芯（Kunlunxin）XPU
- [ERNIE-4.5-VL 部署](./docs/zh/get_started/ernie-4.5-vl.md)
- [离线推理](./docs/zh/offline_inference.md)
- [在线服务](./docs/zh/online_serving/README.md)
- [模型支持列表](./docs/zh/supported_models.md)
- [最佳实践](./docs/zh/best_practices/README.md)
## 支持模型列表
| Model | Data Type | PD Disaggregation | Chunked Prefill | Prefix Caching | MTP | CUDA Graph | Maximum Context Length |
|:--- | :------- | :---------- | :-------- | :-------- | :----- | :----- | :----- |
|ERNIE-4.5-300B-A47B | BF16/WINT4/WINT8/W4A8C8/WINT2/FP8 | ✅| ✅ | ✅|✅| ✅ |128K |
|ERNIE-4.5-300B-A47B-Base| BF16/WINT4/WINT8 | ✅| ✅ | ✅|❌| ✅ | 128K |
|ERNIE-4.5-VL-424B-A47B | BF16/WINT4/WINT8 | WIP | ✅ | WIP | ❌ | WIP |128K |
|ERNIE-4.5-VL-28B-A3B | BF16/WINT4/WINT8 | ❌ | ✅ | WIP | ❌ | WIP |128K |
|ERNIE-4.5-21B-A3B | BF16/WINT4/WINT8/FP8 | ❌ | ✅ | ✅ | ✅ | ✅|128K |
|ERNIE-4.5-21B-A3B-Base | BF16/WINT4/WINT8/FP8 | ✅ | ✅ | ✅ | ❌ | ✅|128K |
|ERNIE-4.5-0.3B | BF16/WINT8/FP8 | ✅ | ✅ | ✅ | ❌ | ✅| 128K |
通过我们的文档了解如何下载模型，如何支持torch格式等：
- [模型支持列表](./docs/zh/supported_models.md)
## 进阶用法

6
dockerfiles/Dockerfile.gpu
View File

@@ -1,6 +1,6 @@
FROM ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/fastdeploy-cuda-12.6:2.1.0
ARG PADDLE_VERSION=3.1.1
ARG FD_VERSION=2.1.0
FROM ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/fastdeploy-cuda-12.6:2.2.0
ARG PADDLE_VERSION=3.2.0
ARG FD_VERSION=2.2.0
ENV DEBIAN_FRONTEND=noninteractive

87
mkdocs.yml
View File

@@ -2,11 +2,13 @@ site_name: 'FastDeploy : Large Language Model Deployement'
repo_url: https://github.com/PaddlePaddle/FastDeploy
repo_name: FastDeploy
copyright: Copyright © 2025 Maintained by FastDeploy
theme:
name: material
highlightjs: true
icon:
repo: fontawesome/brands/github
favicon: assets/images/favicon.ico
logo: assets/images/logo.jpg
palette:
- media: "(prefers-color-scheme: light)" # 浅色
scheme: default
@@ -50,14 +52,17 @@ plugins:
HYGON DCU: 海光 DCU
Enflame S60: 燧原 S60
Iluvatar CoreX: 天数 CoreX
Metax C550: 沐曦 C550
Quick Deployment For ERNIE-4.5-0.3B: ERNIE-4.5-0.3B快速部署
Quick Deployment for ERNIE-4.5-VL-28B-A3B: ERNIE-4.5-VL-28B-A3B快速部署
ERNIE-4.5-300B-A47B: ERNIE-4.5-300B-A47B快速部署
ERNIE-4.5-VL-424B-A47B: ERNIE-4.5-VL-424B-A47B快速部署
Quick Deployment For QWEN: Qwen3-0.6b快速部署
Online Serving: 在线服务
OpenAI-Compitable API Server: 兼容 OpenAI 协议的服务化部署
Monitor Metrics: 监控Metrics
Scheduler: 调度器
Graceful Shutdown: 服务优雅关闭
Offline Inference: 离线推理
Best Practices: 最佳实践
ERNIE-4.5-0.3B: ERNIE-4.5-0.3B
@@ -83,6 +88,8 @@ plugins:
Sampling: 采样策略
MultiNode Deployment: 多机部署
Graph Optimization: 图优化
Data Parallelism: 数据并行
PLAS: PLAS
Supported Models: 支持模型列表
Benchmark: 基准测试
Usage: 用法
@@ -91,23 +98,26 @@ plugins:
Environment Variables: 环境变量
nav:
- 'FastDeploy': index.md
- 'Quick Start':
- FastDeploy: index.md
- Quick Start:
- Installation:
- 'Nvidia GPU': get_started/installation/nvidia_gpu.md
- 'KunlunXin XPU': get_started/installation/kunlunxin_xpu.md
- 'HYGON DCU': get_started/installation/hygon_dcu.md
- 'Enflame S60': get_started/installation/Enflame_gcu.md
- 'Iluvatar CoreX': get_started/installation/iluvatar_gpu.md
- 'Quick Deployment For ERNIE-4.5-0.3B': get_started/quick_start.md
- 'Quick Deployment for ERNIE-4.5-VL-28B-A3B': get_started/quick_start_vl.md
- 'ERNIE-4.5-300B-A47B': get_started/ernie-4.5.md
- 'ERNIE-4.5-VL-424B-A47B': get_started/ernie-4.5-vl.md
- 'Online Serving':
- 'OpenAI-Compitable API Server': online_serving/README.md
- 'Monitor Metrics': online_serving/metrics.md
- 'Scheduler': online_serving/scheduler.md
- 'Offline Inference': offline_inference.md
- Nvidia GPU: get_started/installation/nvidia_gpu.md
- KunlunXin XPU: get_started/installation/kunlunxin_xpu.md
- HYGON DCU: get_started/installation/hygon_dcu.md
- Enflame S60: get_started/installation/Enflame_gcu.md
- Iluvatar CoreX: get_started/installation/iluvatar_gpu.md
- Metax C550: get_started/installation/metax_gpu.md
- Quick Deployment For ERNIE-4.5-0.3B: get_started/quick_start.md
- Quick Deployment for ERNIE-4.5-VL-28B-A3B: get_started/quick_start_vl.md
- ERNIE-4.5-300B-A47B: get_started/ernie-4.5.md
- ERNIE-4.5-VL-424B-A47B: get_started/ernie-4.5-vl.md
- Quick Deployment For QWEN: get_started/quick_start_qwen.md
- Online Serving:
- OpenAI-Compitable API Server: online_serving/README.md
- Monitor Metrics: online_serving/metrics.md
- Scheduler: online_serving/scheduler.md
- Graceful Shutdown: online_serving/graceful_shutdown_service.md
- Offline Inference: offline_inference.md
- Best Practices:
- ERNIE-4.5-0.3B: best_practices/ERNIE-4.5-0.3B-Paddle.md
- ERNIE-4.5-21B-A3B: best_practices/ERNIE-4.5-21B-A3B-Paddle.md
@@ -116,26 +126,27 @@ nav:
- ERNIE-4.5-VL-424B-A47B: best_practices/ERNIE-4.5-VL-424B-A47B-Paddle.md
- FAQ: best_practices/FAQ.md
- Quantization:
- 'Overview': quantization/README.md
- 'Online Quantization': quantization/online_quantization.md
- 'WINT2 Quantization': quantization/wint2.md
- Overview: quantization/README.md
- Online Quantization: quantization/online_quantization.md
- WINT2 Quantization: quantization/wint2.md
- Features:
- 'Prefix Caching': features/prefix_caching.md
- 'Disaggregation': features/disaggregated.md
- 'Chunked Prefill': features/chunked_prefill.md
- 'Load Balance': features/load_balance.md
- 'Speculative Decoding': features/speculative_decoding.md
- 'Structured Outputs': features/structured_outputs.md
- 'Reasoning Output': features/reasoning_output.md
- 'Early Stop': features/early_stop.md
- 'Plugins': features/plugins.md
- 'Sampling': features/sampling.md
- 'MultiNode Deployment': features/multi-node_deployment.md
- 'Graph Optimization': features/graph_optimization.md
- 'Supported Models': supported_models.md
- Prefix Caching: features/prefix_caching.md
- Disaggregation: features/disaggregated.md
- Chunked Prefill: features/chunked_prefill.md
- Load Balance: features/load_balance.md
- Speculative Decoding: features/speculative_decoding.md
- Structured Outputs: features/structured_outputs.md
- Reasoning Output: features/reasoning_output.md
- Early Stop: features/early_stop.md
- Plugins: features/plugins.md
- Sampling: features/sampling.md
- MultiNode Deployment: features/multi-node_deployment.md
- Graph Optimization: features/graph_optimization.md
- Data Parallelism: features/data_parallel_service.md
- PLAS: features/plas_attention.md
- Supported Models: supported_models.md
- Benchmark: benchmark.md
- Usage:
- 'Log Description': usage/log.md
- 'Code Overview': usage/code_overview.md
- 'Environment Variables': usage/environment_variables.md
- 'FastDeploy Unit Test Guide': usage/fastdeploy_unit_test_guide.md
- Log Description: usage/log.md
- Code Overview: usage/code_overview.md
- Environment Variables: usage/environment_variables.md