apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-11-03 00:44:23 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[Optimization] Fuse get_max_len and get_kv_max_len (#4369)
Some checks failed
CE Compile Job / ce_job_pre_check (push) Has been cancelled
Details
CE Compile Job / print_ce_job_pre_check_outputs (push) Has been cancelled
Details
CE Compile Job / FD-Clone-Linux (push) Has been cancelled
Details
CE Compile Job / Show Code Archive Output (push) Has been cancelled
Details
CE Compile Job / BUILD_SM8090 (push) Has been cancelled
Details
CE Compile Job / BUILD_SM8689 (push) Has been cancelled
Details
CE Compile Job / CE_UPLOAD (push) Has been cancelled
Details
Deploy GitHub Pages / deploy (push) Has been cancelled
Details

Browse Source 
* opt split_q_block

* fuse max_lens and max kv_len
This commit is contained in:
Sunny-bot1
2025-10-13 20:35:00 +08:00
committed by GitHub
parent 425205b03c
commit a751d977bc
15 changed files with 29 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
fastdeploy/spec_decode/mtp.py
View File

@@ -247,7 +247,6 @@ class MTPProposer(Proposer):
self.model_inputs["kv_num_blocks_x_cpu"] = paddle.zeros_like(
self.target_model_inputs["kv_num_blocks_x_cpu"]
).cpu()
self.model_inputs["max_len_kv_cpu"] = paddle.zeros_like(self.target_model_inputs["max_len_kv_cpu"]).cpu()
# Get the attention backend
attn_cls = get_attention_backend()
@@ -374,7 +373,6 @@ class MTPProposer(Proposer):
self.model_inputs["kv_batch_ids"] = None
self.model_inputs["kv_tile_ids_per_batch"] = None
self.model_inputs["kv_num_blocks_x_cpu"] = None # CPU
self.model_inputs["max_len_kv_cpu"] = None # CPU
# Input tokens
self.model_inputs["draft_tokens"] = paddle.full(
@@ -583,7 +581,6 @@ class MTPProposer(Proposer):
kv_batch_ids=self.model_inputs["kv_batch_ids"],
kv_tile_ids_per_batch=self.model_inputs["kv_tile_ids_per_batch"],
kv_num_blocks_x_cpu=self.model_inputs["kv_num_blocks_x_cpu"],
max_len_kv_cpu=self.model_inputs["max_len_kv_cpu"],
)
# Initialzie attention meta data