apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-05 08:37:06 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[BugFix] fix num_running_requests in cuda_graph (#3457)

Browse Source 
* fix cuda_grpah

* add note

---------

Co-authored-by: RAM <gstian5555@outlook.com>
This commit is contained in:
lizexu123
2025-08-19 10:47:22 +08:00
committed by GitHub
parent beec24fd89
commit a053ab889b
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
fastdeploy/worker/gpu_model_runner.py
View File

@@ -1594,6 +1594,10 @@ class GPUModelRunner(ModelRunnerBase):
In FastDeploy, almost all input tensors have a buffer. So, just keep the buffer clean when replaying the CUDA graph with the padded batch. In FastDeploy, almost all input tensors have a buffer. So, just keep the buffer clean when replaying the CUDA graph with the padded batch.
""" """
# In init_attention_metadata, the decode buffer has already been cleared # In init_attention_metadata, the decode buffer has already been cleared
# To adapt to CUDA Graph, keep the forward pass at the maximum batch size.
if self.use_cudagraph:
self.forward_meta.seq_lens_this_time = self.seq_lens_this_time_buffer
return return
def _init_image_preprocess(self) -> None: def _init_image_preprocess(self) -> None: