apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-05 08:37:06 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[BugFix]Fix ep size (#3092)

Browse Source 
* fix ep

* fix num_layer
This commit is contained in:
YuanRisheng
2025-07-30 21:03:12 +08:00
committed by GitHub
parent d17886de19
commit 7dfdd157ac
4 changed files with 10 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
fastdeploy/cache_manager/prefix_cache_manager.py
View File

@@ -168,7 +168,7 @@ class PrefixCacheManager:
+ f" --device_id {int(device_ids[i])}"
+ f" --rank {i}"
+ f" --splitwise_role {self.splitwise_role}"
+ f" --num_layers {cache_config.model_cfg.num_layers}"
+ f" --num_layers {cache_config.model_cfg.num_hidden_layers}"
+ f" --head_dim {cache_config.model_cfg.head_dim}"
+ f" --kv_num_head {kv_num_head}"
+ f" --mp_num {tensor_parallel_size}"