apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-05 16:48:03 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[LLM] support send batch data and aggregate data (#2860)

Browse Source 
* [LLM] support send batch data and aggregate data

* [LLM] fix ci bugs

* [LLM] fix ci bugs

* [LLM] fix ci bugs

* [LLM] fix ci bugs

* [LLM] update
This commit is contained in:
ltd0924
2025-07-16 23:42:20 +08:00
committed by GitHub
parent 63d6e7ce06
commit d245d1ca6c
11 changed files with 267 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
fastdeploy/engine/engine.py
View File

@@ -263,10 +263,11 @@ class LLMEngine(object):
try:
results = self.scheduler.get_results()
if len(results) == 0:
time.sleep(0.001)
time.sleep(0.005)
continue
for request_id, contents in results.items():
for result in contents:
self.zmq_server.send_multipart(request_id, result)
self.zmq_server.send_multipart(request_id, contents)
except Exception as e:
llm_logger.error("Unexcepted error happend: {}, {}".format(
e, str(traceback.format_exc())))