apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-05 00:33:03 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[Feature] refactor metax_gpu attention and moe and remove some useless code (#3688)

Browse Source 
Co-authored-by: yongqiangma <xing.wo@163.com>
This commit is contained in:
SuperNova
2025-09-12 14:40:25 +08:00
committed by GitHub
parent cab7a633fe
commit 805f29a06c
5 changed files with 389 additions and 289 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
fastdeploy/config.py
View File

@@ -894,7 +894,7 @@ class CacheConfig:
self.kv_cache_ratio = 1.0
else:
self.kv_cache_ratio = 0.75
self.enc_dec_block_num = 0 if current_platform.is_iluvatar() else 2
self.enc_dec_block_num = 0 if current_platform.is_iluvatar() or current_platform.is_maca() else 2
self.prealloc_dec_block_slot_num_threshold = 12
self.cache_dtype = "bfloat16"
self.model_cfg = None