apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-12-24 13:28:13 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

remove load default_v1 since already been as default (#4980)

Browse Source
This commit is contained in:
JYChen
2025-11-12 16:49:48 +08:00
committed by GitHub
parent c45b3ccb52
commit a1218076dc
14 changed files with 10 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/best_practices/ERNIE-4.5-0.3B-Paddle.md
View File

@@ -32,12 +32,10 @@ python -m fastdeploy.entrypoints.openai.api_server \
--tensor-parallel-size 1 \
--quantization wint4 \
--max-model-len 32768 \
--max-num-seqs 128 \
--load-choices "default_v1"
--max-num-seqs 128
```
- `--quantization`: indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed).
- `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency.
- `--load-choices`: indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage.
For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。

4
docs/best_practices/ERNIE-4.5-21B-A3B-Paddle.md
View File

@@ -32,12 +32,10 @@ python -m fastdeploy.entrypoints.openai.api_server \
--tensor-parallel-size 1 \
--quantization wint4 \
--max-model-len 32768 \
--max-num-seqs 128 \
--load-choices "default_v1"
--max-num-seqs 128
```
- `--quantization`: indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed).
- `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency.
- `--load-choices`: indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage.
For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。

2
docs/best_practices/ERNIE-4.5-21B-A3B-Thinking.md
View File

@@ -29,7 +29,6 @@ Start the service by following command:
```bash
python -m fastdeploy.entrypoints.openai.api_server \
--model baidu/ERNIE-4.5-21B-A3B-Thinking \
--load-choices "default_v1" \
--tensor-parallel-size 1 \
--max-model-len 131072 \
--quantization wint8 \
@@ -39,7 +38,6 @@ python -m fastdeploy.entrypoints.openai.api_server \
```
- `--quantization`: Indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed).
- `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency.
- `--load-choices`: Indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage.
- `--reasoning-parser`, `--tool-call-parser`: Indicates the corresponding reasoning content and tool call parser.
For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。

4
docs/best_practices/ERNIE-4.5-300B-A47B-Paddle.md
View File

@@ -29,12 +29,10 @@ python -m fastdeploy.entrypoints.openai.api_server \
--tensor-parallel-size 8 \
--quantization wint4 \
--max-model-len 32768 \
--max-num-seqs 128 \
--load-choices "default_v1"
--max-num-seqs 128
```
- `--quantization`: indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed).
- `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency.
- `--load-choices`: indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage.
For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。

1
docs/get_started/installation/iluvatar_gpu.md
View File

@@ -573,7 +573,6 @@ export FD_DEBUG=1
python3 -m fastdeploy.entrypoints.openai.api_server \
--model baidu/ERNIE-4.5-21B-A3B-Thinking \
--port 8180 \
--load-choices "default_v1" \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--quantization wint8 \

6
docs/get_started/quick_start_qwen.md
View File

@@ -17,9 +17,6 @@ For more information about how to install FastDeploy, refer to the [installation
## 1. Launch Service
After installing FastDeploy, execute the following command in the terminal to start the service. For the configuration method of the startup command, refer to [Parameter Description](../parameters.md)
> ⚠️ **Note:**
> When using HuggingFace models (torch format), you need to enable `--load-choices "default_v1"`.
```
export ENABLE_V1_KVCACHE_SCHEDULER=1
python -m fastdeploy.entrypoints.openai.api_server \
@@ -28,8 +25,7 @@ python -m fastdeploy.entrypoints.openai.api_server \
--metrics-port 8181 \
--engine-worker-queue-port 8182 \
--max-model-len 32768 \
--max-num-seqs 32 \
--load-choices "default_v1"
--max-num-seqs 32
```
> 💡 Note: In the path specified by ```--model```, if the subdirectory corresponding to the path does not exist in the current directory, it will try to query whether AIStudio has a preset model based on the specified model name (such as ```Qwen/QWEN3-0.6b```). If it exists, it will automatically start downloading. The default download path is: ```~/xx```. For instructions and configuration on automatic model download, see [Model Download](../supported_models.md).

5
docs/supported_models.md
View File

@@ -15,8 +15,6 @@ export FD_MODEL_SOURCE=AISTUDIO # "AISTUDIO", "MODELSCOPE" or "HUGGINGFACE"
export FD_MODEL_CACHE=/ssd1/download_models
```
> ⭐ **Note**: Models marked with an asterisk can directly use **HuggingFace Torch weights** and support **FP8/WINT8/WINT4** as well as **BF16**. When running inference, you need to enable **`--load-choices "default_v1"`**.
> Example launch Command using baidu/ERNIE-4.5-21B-A3B-PT:
```
python -m fastdeploy.entrypoints.openai.api_server \
@@ -25,8 +23,7 @@ python -m fastdeploy.entrypoints.openai.api_server \
--metrics-port 8181 \
--engine-worker-queue-port 8182 \
--max-model-len 32768 \
--max-num-seqs 32 \
--load-choices "default_v1"
--max-num-seqs 32
```
## Large Language Models

4
docs/zh/best_practices/ERNIE-4.5-0.3B-Paddle.md
View File

@@ -32,13 +32,11 @@ python -m fastdeploy.entrypoints.openai.api_server \
--tensor-parallel-size 1 \
--quantization wint4 \
--max-model-len 32768 \
--max-num-seqs 128 \
--load-choices "default_v1"
--max-num-seqs 128
```
其中：
- `--quantization`: 表示模型采用的量化策略。不同量化策略，模型的性能和精度也会不同。可选值包括：`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。
- `--max-model-len`：表示当前部署的服务所支持的最长Token数量。设置得越大，模型可支持的上下文长度也越大，但相应占用的显存也越多，可能影响并发数。
- `--load-choices`: 表示loader的版本，"default_v1"表示启用v1版本的loader，具有更快的加载速度和更少的内存使用。
更多的参数含义与默认设置，请参见[FastDeploy参数说明](../parameters.md)。

4
docs/zh/best_practices/ERNIE-4.5-21B-A3B-Paddle.md
View File

@@ -32,13 +32,11 @@ python -m fastdeploy.entrypoints.openai.api_server \
--tensor-parallel-size 1 \
--quantization wint4 \
--max-model-len 32768 \
--max-num-seqs 128 \
--load-choices "default_v1"
--max-num-seqs 128
```
其中：
- `--quantization`: 表示模型采用的量化策略。不同量化策略，模型的性能和精度也会不同。可选值包括：`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。
- `--max-model-len`：表示当前部署的服务所支持的最长Token数量。设置得越大，模型可支持的上下文长度也越大，但相应占用的显存也越多，可能影响并发数。
- `--load-choices`: 表示loader的版本，"default_v1"表示启用v1版本的loader，具有更快的加载速度和更少的内存使用。
更多的参数含义与默认设置，请参见[FastDeploy参数说明](../parameters.md)。

2
docs/zh/best_practices/ERNIE-4.5-21B-A3B-Thinking.md
View File

@@ -29,7 +29,6 @@ ERNIE-4.5-21B-A3B 各量化精度，在下列硬件上部署所需要的最小
```bash
python -m fastdeploy.entrypoints.openai.api_server \
--model baidu/ERNIE-4.5-21B-A3B-Thinking \
--load-choices "default_v1" \
--tensor-parallel-size 1 \
--max-model-len 131072 \
--quantization wint8 \
@@ -40,7 +39,6 @@ python -m fastdeploy.entrypoints.openai.api_server \
其中：
- `--quantization`: 表示模型采用的量化策略。不同量化策略，模型的性能和精度也会不同。可选值包括：`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。
- `--max-model-len`：表示当前部署的服务所支持的最长Token数量。设置得越大，模型可支持的上下文长度也越大，但相应占用的显存也越多，可能影响并发数。
- `--load-choices`: 表示loader的版本，"default_v1"表示启用v1版本的loader，具有更快的加载速度和更少的内存使用。
- `--reasoning-parser` 、 `--tool-call-parser`: 表示对应调用的思考内容和工具调用解析器
更多的参数含义与默认设置，请参见[FastDeploy参数说明](../parameters.md)。

4
docs/zh/best_practices/ERNIE-4.5-300B-A47B-Paddle.md
View File

@@ -29,13 +29,11 @@ python -m fastdeploy.entrypoints.openai.api_server \
--tensor-parallel-size 8 \
--quantization wint4 \
--max-model-len 32768 \
--max-num-seqs 128 \
--load-choices "default_v1"
--max-num-seqs 128
```
其中：
- `--quantization`: 表示模型采用的量化策略。不同量化策略，模型的性能和精度也会不同。可选值包括：`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。
- `--max-model-len`：表示当前部署的服务所支持的最长Token数量。设置得越大，模型可支持的上下文长度也越大，但相应占用的显存也越多，可能影响并发数。
- `--load-choices`: 表示loader的版本，"default_v1"表示启用v1版本的loader，具有更快的加载速度和更少的内存使用。
更多的参数含义与默认设置，请参见[FastDeploy参数说明](../parameters.md)。

1
docs/zh/get_started/installation/iluvatar_gpu.md
View File

@@ -573,7 +573,6 @@ export FD_DEBUG=1
python3 -m fastdeploy.entrypoints.openai.api_server \
--model baidu/ERNIE-4.5-21B-A3B-Thinking \
--port 8180 \
--load-choices "default_v1" \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--quantization wint8 \

6
docs/zh/get_started/quick_start_qwen.md
View File

@@ -16,9 +16,6 @@
## 1. 启动服务
安装FastDeploy后，在终端执行如下命令，启动服务，其中启动命令配置方式参考[参数说明](../parameters.md)
> ⚠️ **注意:**
> 当使用HuggingFace 模型(torch格式)时, 需要开启 `--load-choices "default_v1"`
```shell
export ENABLE_V1_KVCACHE_SCHEDULER=1
python -m fastdeploy.entrypoints.openai.api_server \
@@ -27,8 +24,7 @@ python -m fastdeploy.entrypoints.openai.api_server \
--metrics-port 8181 \
--engine-worker-queue-port 8182 \
--max-model-len 32768 \
--max-num-seqs 32 \
--load-choices "default_v1"
--max-num-seqs 32
```
>💡 注意：在 ```--model``` 指定的路径中，若当前目录下不存在该路径对应的子目录，则会尝试根据指定的模型名称（如 ```Qwen/Qwen3-0.6B```）查询AIStudio是否存在预置模型，若存在，则自动启动下载。默认的下载路径为：```~/xx```。关于模型自动下载的说明和配置参阅[模型下载](../supported_models.md)。

5
docs/zh/supported_models.md
View File

@@ -15,8 +15,6 @@ export FD_MODEL_SOURCE=AISTUDIO # "AISTUDIO", "MODELSCOPE" or "HUGGINGFACE"
export FD_MODEL_CACHE=/ssd1/download_models
```
> ⭐ **说明**：带星号的模型可直接使用 **HuggingFace Torch 权重**，支持 **FP8/WINT8/WINT4 动态量化** 和 **BF16 精度** 推理，推理时需启用 **`--load-choices "default_v1"`**。
> 以baidu/ERNIE-4.5-21B-A3B-PT为例启动命令如下
```
python -m fastdeploy.entrypoints.openai.api_server \
@@ -25,8 +23,7 @@ python -m fastdeploy.entrypoints.openai.api_server \
--metrics-port 8181 \
--engine-worker-queue-port 8182 \
--max-model-len 32768 \
--max-num-seqs 32 \
--load-choices "default_v1"
--max-num-seqs 32
```
## 纯文本模型列表