apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-12-24 13:28:13 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[Speculative Decoding][MTP]Support mtp in epdptp mode (#4614)

Browse Source 
* support mtp many features

* support mtp reshard in rl mode

* fix function

* support mtp ep

* support mtp in hybird-dp-tp mode

* default open scheduler_v1 in mtp
This commit is contained in:
freeliuzc
2025-10-28 16:02:47 +08:00
committed by GitHub
parent b4014834a9
commit c63361fd1d
10 changed files with 124 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
fastdeploy/model_executor/model_loader/default_loader_v1.py
View File

@@ -65,6 +65,11 @@ class DefaultModelLoaderV1(BaseModelLoader):
# register rl model
import fastdeploy.rl # noqa
if fd_config.speculative_config.model_type != "mtp":
architectures = architectures.replace("Ernie5ForCausalLM", "Ernie5MoeForCausalLM")
else:
architectures = architectures.replace("Ernie5ForCausalLM", "Ernie5MTPForCausalLM")
architectures = architectures + "RL"
enable_cache, _, weight_cache_context = is_weight_cache_enabled(fd_config)