diff --git a/docs/best_practices/ERNIE-4.5-0.3B-Paddle.md b/docs/best_practices/ERNIE-4.5-0.3B-Paddle.md index 5644f3840..7aa8a2e6a 100644 --- a/docs/best_practices/ERNIE-4.5-0.3B-Paddle.md +++ b/docs/best_practices/ERNIE-4.5-0.3B-Paddle.md @@ -32,12 +32,10 @@ python -m fastdeploy.entrypoints.openai.api_server \ --tensor-parallel-size 1 \ --quantization wint4 \ --max-model-len 32768 \ - --max-num-seqs 128 \ - --load-choices "default_v1" + --max-num-seqs 128 ``` - `--quantization`: indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed). - `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency. -- `--load-choices`: indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage. For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。 diff --git a/docs/best_practices/ERNIE-4.5-21B-A3B-Paddle.md b/docs/best_practices/ERNIE-4.5-21B-A3B-Paddle.md index 071dbe8e2..851657f46 100644 --- a/docs/best_practices/ERNIE-4.5-21B-A3B-Paddle.md +++ b/docs/best_practices/ERNIE-4.5-21B-A3B-Paddle.md @@ -32,12 +32,10 @@ python -m fastdeploy.entrypoints.openai.api_server \ --tensor-parallel-size 1 \ --quantization wint4 \ --max-model-len 32768 \ - --max-num-seqs 128 \ - --load-choices "default_v1" + --max-num-seqs 128 ``` - `--quantization`: indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed). - `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency. -- `--load-choices`: indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage. For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。 diff --git a/docs/best_practices/ERNIE-4.5-21B-A3B-Thinking.md b/docs/best_practices/ERNIE-4.5-21B-A3B-Thinking.md index a67be76fe..49e78c4b9 100644 --- a/docs/best_practices/ERNIE-4.5-21B-A3B-Thinking.md +++ b/docs/best_practices/ERNIE-4.5-21B-A3B-Thinking.md @@ -29,7 +29,6 @@ Start the service by following command: ```bash python -m fastdeploy.entrypoints.openai.api_server \ --model baidu/ERNIE-4.5-21B-A3B-Thinking \ - --load-choices "default_v1" \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --quantization wint8 \ @@ -39,7 +38,6 @@ python -m fastdeploy.entrypoints.openai.api_server \ ``` - `--quantization`: Indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed). - `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency. -- `--load-choices`: Indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage. - `--reasoning-parser`, `--tool-call-parser`: Indicates the corresponding reasoning content and tool call parser. For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。 diff --git a/docs/best_practices/ERNIE-4.5-300B-A47B-Paddle.md b/docs/best_practices/ERNIE-4.5-300B-A47B-Paddle.md index f658477e2..537bed67c 100644 --- a/docs/best_practices/ERNIE-4.5-300B-A47B-Paddle.md +++ b/docs/best_practices/ERNIE-4.5-300B-A47B-Paddle.md @@ -29,12 +29,10 @@ python -m fastdeploy.entrypoints.openai.api_server \ --tensor-parallel-size 8 \ --quantization wint4 \ --max-model-len 32768 \ - --max-num-seqs 128 \ - --load-choices "default_v1" + --max-num-seqs 128 ``` - `--quantization`: indicates the quantization strategy used by the model. Different quantization strategies will result in different performance and accuracy of the model. It could be one of `wint8` / `wint4` / `block_wise_fp8`(Hopper is needed). - `--max-model-len`: Indicates the maximum number of tokens supported by the currently deployed service. The larger the value, the longer the context length the model can support, but the more GPU memory is occupied, which may affect the concurrency. -- `--load-choices`: indicates the version of the loader. "default_v1" means enabling the v1 version of the loader, which has faster loading speed and less memory usage. For more parameter meanings and default settings, see [FastDeploy Parameter Documentation](../parameters.md)。 diff --git a/docs/get_started/installation/iluvatar_gpu.md b/docs/get_started/installation/iluvatar_gpu.md index 1a205323d..e5720015c 100644 --- a/docs/get_started/installation/iluvatar_gpu.md +++ b/docs/get_started/installation/iluvatar_gpu.md @@ -573,7 +573,6 @@ export FD_DEBUG=1 python3 -m fastdeploy.entrypoints.openai.api_server \ --model baidu/ERNIE-4.5-21B-A3B-Thinking \ --port 8180 \ - --load-choices "default_v1" \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --quantization wint8 \ diff --git a/docs/get_started/quick_start_qwen.md b/docs/get_started/quick_start_qwen.md index 4f6d834ef..79bfc4081 100644 --- a/docs/get_started/quick_start_qwen.md +++ b/docs/get_started/quick_start_qwen.md @@ -17,9 +17,6 @@ For more information about how to install FastDeploy, refer to the [installation ## 1. Launch Service After installing FastDeploy, execute the following command in the terminal to start the service. For the configuration method of the startup command, refer to [Parameter Description](../parameters.md) -> ⚠️ **Note:** -> When using HuggingFace models (torch format), you need to enable `--load-choices "default_v1"`. - ``` export ENABLE_V1_KVCACHE_SCHEDULER=1 python -m fastdeploy.entrypoints.openai.api_server \ @@ -28,8 +25,7 @@ python -m fastdeploy.entrypoints.openai.api_server \ --metrics-port 8181 \ --engine-worker-queue-port 8182 \ --max-model-len 32768 \ - --max-num-seqs 32 \ - --load-choices "default_v1" + --max-num-seqs 32 ``` > 💡 Note: In the path specified by ```--model```, if the subdirectory corresponding to the path does not exist in the current directory, it will try to query whether AIStudio has a preset model based on the specified model name (such as ```Qwen/QWEN3-0.6b```). If it exists, it will automatically start downloading. The default download path is: ```~/xx```. For instructions and configuration on automatic model download, see [Model Download](../supported_models.md). diff --git a/docs/supported_models.md b/docs/supported_models.md index e4e7915a1..b0684affc 100644 --- a/docs/supported_models.md +++ b/docs/supported_models.md @@ -15,8 +15,6 @@ export FD_MODEL_SOURCE=AISTUDIO # "AISTUDIO", "MODELSCOPE" or "HUGGINGFACE" export FD_MODEL_CACHE=/ssd1/download_models ``` -> ⭐ **Note**: Models marked with an asterisk can directly use **HuggingFace Torch weights** and support **FP8/WINT8/WINT4** as well as **BF16**. When running inference, you need to enable **`--load-choices "default_v1"`**. - > Example launch Command using baidu/ERNIE-4.5-21B-A3B-PT: ``` python -m fastdeploy.entrypoints.openai.api_server \ @@ -25,8 +23,7 @@ python -m fastdeploy.entrypoints.openai.api_server \ --metrics-port 8181 \ --engine-worker-queue-port 8182 \ --max-model-len 32768 \ - --max-num-seqs 32 \ - --load-choices "default_v1" + --max-num-seqs 32 ``` ## Large Language Models diff --git a/docs/zh/best_practices/ERNIE-4.5-0.3B-Paddle.md b/docs/zh/best_practices/ERNIE-4.5-0.3B-Paddle.md index 869bf783f..40bb723b3 100644 --- a/docs/zh/best_practices/ERNIE-4.5-0.3B-Paddle.md +++ b/docs/zh/best_practices/ERNIE-4.5-0.3B-Paddle.md @@ -32,13 +32,11 @@ python -m fastdeploy.entrypoints.openai.api_server \ --tensor-parallel-size 1 \ --quantization wint4 \ --max-model-len 32768 \ - --max-num-seqs 128 \ - --load-choices "default_v1" + --max-num-seqs 128 ``` 其中: - `--quantization`: 表示模型采用的量化策略。不同量化策略,模型的性能和精度也会不同。可选值包括:`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。 - `--max-model-len`:表示当前部署的服务所支持的最长Token数量。设置得越大,模型可支持的上下文长度也越大,但相应占用的显存也越多,可能影响并发数。 -- `--load-choices`: 表示loader的版本,"default_v1"表示启用v1版本的loader,具有更快的加载速度和更少的内存使用。 更多的参数含义与默认设置,请参见[FastDeploy参数说明](../parameters.md)。 diff --git a/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Paddle.md b/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Paddle.md index 54b528176..1fc72f748 100644 --- a/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Paddle.md +++ b/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Paddle.md @@ -32,13 +32,11 @@ python -m fastdeploy.entrypoints.openai.api_server \ --tensor-parallel-size 1 \ --quantization wint4 \ --max-model-len 32768 \ - --max-num-seqs 128 \ - --load-choices "default_v1" + --max-num-seqs 128 ``` 其中: - `--quantization`: 表示模型采用的量化策略。不同量化策略,模型的性能和精度也会不同。可选值包括:`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。 - `--max-model-len`:表示当前部署的服务所支持的最长Token数量。设置得越大,模型可支持的上下文长度也越大,但相应占用的显存也越多,可能影响并发数。 -- `--load-choices`: 表示loader的版本,"default_v1"表示启用v1版本的loader,具有更快的加载速度和更少的内存使用。 更多的参数含义与默认设置,请参见[FastDeploy参数说明](../parameters.md)。 diff --git a/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Thinking.md b/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Thinking.md index c2648ceb3..d904d9129 100644 --- a/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Thinking.md +++ b/docs/zh/best_practices/ERNIE-4.5-21B-A3B-Thinking.md @@ -29,7 +29,6 @@ ERNIE-4.5-21B-A3B 各量化精度,在下列硬件上部署所需要的最小 ```bash python -m fastdeploy.entrypoints.openai.api_server \ --model baidu/ERNIE-4.5-21B-A3B-Thinking \ - --load-choices "default_v1" \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --quantization wint8 \ @@ -40,7 +39,6 @@ python -m fastdeploy.entrypoints.openai.api_server \ 其中: - `--quantization`: 表示模型采用的量化策略。不同量化策略,模型的性能和精度也会不同。可选值包括:`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。 - `--max-model-len`:表示当前部署的服务所支持的最长Token数量。设置得越大,模型可支持的上下文长度也越大,但相应占用的显存也越多,可能影响并发数。 -- `--load-choices`: 表示loader的版本,"default_v1"表示启用v1版本的loader,具有更快的加载速度和更少的内存使用。 - `--reasoning-parser` 、 `--tool-call-parser`: 表示对应调用的思考内容和工具调用解析器 更多的参数含义与默认设置,请参见[FastDeploy参数说明](../parameters.md)。 diff --git a/docs/zh/best_practices/ERNIE-4.5-300B-A47B-Paddle.md b/docs/zh/best_practices/ERNIE-4.5-300B-A47B-Paddle.md index 6cfe79aa2..184439b74 100644 --- a/docs/zh/best_practices/ERNIE-4.5-300B-A47B-Paddle.md +++ b/docs/zh/best_practices/ERNIE-4.5-300B-A47B-Paddle.md @@ -29,13 +29,11 @@ python -m fastdeploy.entrypoints.openai.api_server \ --tensor-parallel-size 8 \ --quantization wint4 \ --max-model-len 32768 \ - --max-num-seqs 128 \ - --load-choices "default_v1" + --max-num-seqs 128 ``` 其中: - `--quantization`: 表示模型采用的量化策略。不同量化策略,模型的性能和精度也会不同。可选值包括:`wint8` / `wint4` / `block_wise_fp8`(需要Hopper架构)。 - `--max-model-len`:表示当前部署的服务所支持的最长Token数量。设置得越大,模型可支持的上下文长度也越大,但相应占用的显存也越多,可能影响并发数。 -- `--load-choices`: 表示loader的版本,"default_v1"表示启用v1版本的loader,具有更快的加载速度和更少的内存使用。 更多的参数含义与默认设置,请参见[FastDeploy参数说明](../parameters.md)。 diff --git a/docs/zh/get_started/installation/iluvatar_gpu.md b/docs/zh/get_started/installation/iluvatar_gpu.md index 7125dda6e..06ad189ed 100644 --- a/docs/zh/get_started/installation/iluvatar_gpu.md +++ b/docs/zh/get_started/installation/iluvatar_gpu.md @@ -573,7 +573,6 @@ export FD_DEBUG=1 python3 -m fastdeploy.entrypoints.openai.api_server \ --model baidu/ERNIE-4.5-21B-A3B-Thinking \ --port 8180 \ - --load-choices "default_v1" \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --quantization wint8 \ diff --git a/docs/zh/get_started/quick_start_qwen.md b/docs/zh/get_started/quick_start_qwen.md index c678a7b6c..ca395666c 100644 --- a/docs/zh/get_started/quick_start_qwen.md +++ b/docs/zh/get_started/quick_start_qwen.md @@ -16,9 +16,6 @@ ## 1. 启动服务 安装FastDeploy后,在终端执行如下命令,启动服务,其中启动命令配置方式参考[参数说明](../parameters.md) -> ⚠️ **注意:** -> 当使用HuggingFace 模型(torch格式)时, 需要开启 `--load-choices "default_v1"` - ```shell export ENABLE_V1_KVCACHE_SCHEDULER=1 python -m fastdeploy.entrypoints.openai.api_server \ @@ -27,8 +24,7 @@ python -m fastdeploy.entrypoints.openai.api_server \ --metrics-port 8181 \ --engine-worker-queue-port 8182 \ --max-model-len 32768 \ - --max-num-seqs 32 \ - --load-choices "default_v1" + --max-num-seqs 32 ``` >💡 注意:在 ```--model``` 指定的路径中,若当前目录下不存在该路径对应的子目录,则会尝试根据指定的模型名称(如 ```Qwen/Qwen3-0.6B```)查询AIStudio是否存在预置模型,若存在,则自动启动下载。默认的下载路径为:```~/xx```。关于模型自动下载的说明和配置参阅[模型下载](../supported_models.md)。 diff --git a/docs/zh/supported_models.md b/docs/zh/supported_models.md index b131cbe3d..1424d2320 100644 --- a/docs/zh/supported_models.md +++ b/docs/zh/supported_models.md @@ -15,8 +15,6 @@ export FD_MODEL_SOURCE=AISTUDIO # "AISTUDIO", "MODELSCOPE" or "HUGGINGFACE" export FD_MODEL_CACHE=/ssd1/download_models ``` -> ⭐ **说明**:带星号的模型可直接使用 **HuggingFace Torch 权重**,支持 **FP8/WINT8/WINT4 动态量化** 和 **BF16 精度** 推理,推理时需启用 **`--load-choices "default_v1"`**。 - > 以baidu/ERNIE-4.5-21B-A3B-PT为例启动命令如下 ``` python -m fastdeploy.entrypoints.openai.api_server \ @@ -25,8 +23,7 @@ python -m fastdeploy.entrypoints.openai.api_server \ --metrics-port 8181 \ --engine-worker-queue-port 8182 \ --max-model-len 32768 \ - --max-num-seqs 32 \ - --load-choices "default_v1" + --max-num-seqs 32 ``` ## 纯文本模型列表