313 Commits

Author	SHA1	Message	Date
cmcamdy	6cc3cb4bcf	fix mtp multi batch (#5521)	2025-12-12 14:11:20 +08:00
GoldPancake	909059c60a	[Feature] Support for request-level speculative decoding metrics monitoring. (#5518) ... * support spec metrics monitor per request * fix bug * remove debug log * fix ut bugs	2025-12-12 12:22:18 +08:00
kevin	954a145d57	[Optimization] support mm prefill batch (#5313) ... * support mm prefill batch * update code * update code * update code * update code * fix encoder cache bug * update code * update code * fix bug * fix paddle ocr bug * fix xpu bug * update code	2025-12-11 22:21:14 +08:00
chen	747b16e021	[BugFix] Fix MTP no logprobs when enable_logprob (#5499)	2025-12-11 19:57:22 +08:00
Ryan	e58fed3665	[Graph Optimization][BugFix][CI] Fix 0size bug && add unitest (#5495)	2025-12-11 16:25:26 +08:00
周周周	ff353b922f	[Others] update tbo related code (#5485)	2025-12-11 12:34:46 +08:00
chen	6289cbc434	[BugFix] fix hung when n>1 and --enable-logprob (#5492)	2025-12-11 10:46:27 +08:00
Jiang-Jia-Jun	4b3e41c665	[Optim] Improve task-checking performance in engine-worker-queue (#5376) ... * [Optim] Optimize costtime in checking tasks in engine-worker-queue * Update fastdeploy/engine/common_engine.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/inter_communicator/engine_worker_queue.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [Docs] Add docstring to set_exist_tasks method (#5382) * Initial plan * Add docstring to set_exist_tasks method Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * [Docs] Add docstring documentation to exist_tasks() method (#5381) * Initial plan * Add comprehensive docstring to exist_tasks() method Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * [Optimization] Conditionally initialize shared memory for single-node deployments only (#5383) * Initial plan * Conditionally initialize exist_tasks_intra_signal for single-node deployments Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * Use is_single_node flag for consistent deployment type checking Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * Remove redundant None checks in exist_tasks methods Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * format code --------- Co-authored-by: Jiang-Jia-Jun <jiangjiajun@baidu.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>	2025-12-11 10:33:32 +08:00
Yonghua Li	2ec76352da	[BugFix] fix instability after clearing weight (#5493) ... * [BugFix] fix instability after clearing weight * [chore] add todo	2025-12-11 10:22:35 +08:00
Neil Zhu	4403a21d4b	[Metax] refactor cutlass moe and optimize flash attention (#5361) ... * [Metax] refactor moe and flash attention backend --------- Co-authored-by: zhangchenyi_dl <16219492+zhangchenyidl@user.noreply.gitee.com>	2025-12-10 17:15:17 +08:00
Yonghua Li	419b416376	[BugFix] [RL] remove shutdown_process_group/restart_process_group for RL (#5433) ... * [fix] remove shutdown_process_group/restart_process_group for RL * [chore] remove log * [chore] remove log * [chore] set log to debug level	2025-12-09 20:32:37 +08:00
lizexu123	95eab9f9ee	[Feature] support stop_token_ids (#5399) ... * support stop_token_ids * fix * delete chinese * support both * delete print	2025-12-09 17:49:12 +08:00
lizexu123	b0cf2c4b7a	[Feature] Support prefill batch inference for pooling models. (#5436) ... * fix multi-inputs * fix threshold * fix threshold * fix * support multi-batch * add tests * fix test * test * fix	2025-12-09 16:21:00 +08:00
Nyakku Shigure	e1c4a12e34	[Graph Optimization][CINN] Use CINN in PaddleOCR-VL ViT part (#5223) ... --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-12-09 14:37:00 +08:00
chen	76649b45c1	[Optimization] compulte real max_logprobs in batch (#5430)	2025-12-09 14:15:05 +08:00
xiaozude	c06a6234b9	[Metax] optimize mla attention (#5258)	2025-12-09 11:18:19 +08:00
Daci	2f208db4e9	[Feature] Multimodal Model P / D Separation (#5323) ... * RouterArgs port str -> int * fix race condition [is_fetching] causing multiple fetch requests * bugfix: Delete duplicate input_ids tensor creation * mm pd splitwise json -> pickle5; multimodal_inputs only pos id; debuglog f to %s * fix ENABLE_V1_KVCACHE_SCHEDULER=0 mm model lack pos_id, ... * update cr * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * pre-commit fix * rm multimodal_inputs deepcopy & fix rdma_cache_transfer.py tpsize=0 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2025-12-09 10:47:42 +08:00
周周周	438c9f785a	[BugFix] 0 not into cuda graph to save memory (#5426)	2025-12-08 16:47:44 +08:00
RAM	b2908b8e82	[New][RL] Support Rollout Routing Replay (#5405) ... * [RL] Support Rollout Routing Replay * add routing indices cache * fix config bug and moe forward bug * R3 Support GLM * support eb4.5 * fix merge bug * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add routing replay ci * support glm topk * support orther top_k * fix ci bug * pre-commit * only support chatcmpl * Revert "Revert "[RL] Support Rollout Routing Replay (#5321)" (#5402)" This reverts commit c45e064f3d. * Fix XPU and NPU bug --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-12-05 22:06:26 +08:00
Jiang-Jia-Jun	c45e064f3d	Revert "[RL] Support Rollout Routing Replay (#5321)" (#5402) ... This reverts commit 96d2d4877b.	2025-12-05 20:19:39 +08:00
周周周	94c57e4175	[BugFix]remove _execute_empty_input (#5396)	2025-12-05 20:19:01 +08:00
RAM	96d2d4877b	[RL] Support Rollout Routing Replay (#5321) ... * [RL] Support Rollout Routing Replay * add routing indices cache * fix config bug and moe forward bug * R3 Support GLM * support eb4.5 * fix merge bug * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add routing replay ci * support glm topk * support orther top_k * fix ci bug * pre-commit * only support chatcmpl --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-12-05 20:01:33 +08:00
Ayakouji	a8f8791668	[Optimization] Qwen2.5-VL support multi-batch prefill (#5269) ... * update * fix Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix dict access --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-05 18:22:39 +08:00
周周周	c83dc58105	[Feature] support Two batch overlap, mainly used in Prefill (#5078)	2025-12-05 14:58:50 +08:00
fmiao2372	ebe613ccc8	[Intel HPU] fix bug about RP 5138 (#5380)	2025-12-05 11:33:29 +08:00
zccjjj	e927c65742	[XPU] [Optimization] [EP] EP communication optimization. (#5145)	2025-12-05 10:03:45 +08:00
Yonghua Li	f4119d51b4	[PD Disaggregation] support DP via v1 router and decouple DP and EP (#5197) ... * [fix] support DP via v1 router and decouple DP and EP * [fix] fix scripts * [fix] reset model path * [fix] dp use get_output_ep, fix router port type, update scripts * [merge] merge with latest code * [chore] remove some debug log * [fix] fix code style check * [fix] fix test_multi_api_server for log_dir name * [chore] reduce logs * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-12-04 15:38:43 +08:00
Longzhi Wang	5cd17fd662	[Models] Add forward_meta to moe models' forward function (#5138) ... * [Models] Add forward_meta to moe models' forward function * fix missing param * fix * fix * fix forward_meta * fix test and remove chunked MoE releated in config * fix test * fix * fix	2025-12-04 13:26:58 +08:00
fmiao2372	209006e6a6	[Intel HPU] fix memory fragmentation issue due to warmup process and fix moe all_reduce issue (#5357)	2025-12-04 11:29:41 +08:00
lizexu123	946025480e	[Bug fix] fix pooling models (#5358) ... * fix * fix * fix test * fix gpu_model_runner --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-12-04 11:06:30 +08:00
qwes5s5	a52aea073c	fix logprobs (#5335)	2025-12-04 10:38:51 +08:00
ddchenhao66	4e8096bd0d	[XPU] xpu support mm prefix cache (#5356) ... Co-authored-by: ddchenhao66 <dhaochen163.com>	2025-12-03 19:07:34 +08:00
qw86972190	6048ea37bd	[XPU]add enable_logprob (#5279) ... * [XPU]Update document * [XPU]Update documentation * [XPU]add enable_logprob * Fix code style issues * “doc” * “docs” * “doc” * Fix code style via pre-commit --------- Co-authored-by: root <root@gajl-bbc-onlinec-com-1498354.gajl.baidu.com>	2025-12-02 15:32:28 +08:00
lizexu123	c563eca791	[Feature] support reward model (#5301) ... * Your commit message here * add test * update develop * support reward * support enable_chunk_prefill * support bingfa * support convert is reward * update test * delete print * fix enable_thinking * add document * fix place * fix test * fix * support enable_prefix_caching * add no-enable_prefix-caching test * fix * support enable_prefix_caching * delete print * fix document * fix * fix test * fix document and delete chinese * udpate * enable_thinking * fix test	2025-12-02 14:55:31 +08:00
K11OntheBoat	2e1680838f	[PD Disaggregation] Support PD deployment of DeepSeekv3. (#5251) ... * Support deepseekv3 cache transfer for PD deploy * clean some log info --------- Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-12-02 14:11:50 +08:00
qwes5s5	117980dd4e	[LogProbs]Enable prompt logprobs output and modify data transmission method for the online interface. (#5089) ... * add prompt logprobs * Merge prompt_logprobs_tensors and prompt_logprobs * fix param check * trigger ci * fix unitest * fix logprobs bug	2025-12-02 13:49:51 +08:00
Longzhi Wang	add524d80c	[Feature] support chunked moe (#4575) ... * [Feature] support chunked moe * update * update * fix and add test * update * fix conflict and modity test * fix fused_moe * fix fused_moe * fix docstring * fix * fix typo * fix test * fix * fix * fix test * fix test	2025-12-01 15:17:18 +08:00
cmcamdy	9f4977eb74	[xpu] support mtp for xpu(mix) (#5274) ... * [XPU] support kernel for mtp(base) * [XPU] support kernel for mtp(base) * format * format * format * fix gather next token * fix step && add test * fix * mv pre/post process * add adjust batch / gather next token for mtp * fix code style * fix mtp kenrel name * fix mtp kernel test * mv xpu pre/post process * mv xpu pre/post process * [xpu] support mtp * fix code style	2025-12-01 11:03:14 +08:00
fmiao2372	2c7683d551	[Intel HPU] change MoE weights and scales from list to tensor and add… (#5289) ... * [Intel HPU] change MoE weights and scales from list to tensor and add q/k rms norm * update doc * move HPU_CHUNK_SIZE into envs	2025-11-28 19:17:05 +08:00
周周周	73886204d4	[Others] clean code (#5235)	2025-11-28 15:40:49 +08:00
cmcamdy	5a67a6d960	[XPU] support kernel for mtp(base) (#4748) ... * [XPU] support kernel for mtp(base) * [XPU] support kernel for mtp(base) * format * format * format * fix gather next token * fix step && add test * fix * mv pre/post process * add adjust batch / gather next token for mtp * fix code style * fix mtp kenrel name * fix mtp kernel test * mv xpu pre/post process * mv xpu pre/post process	2025-11-27 15:05:44 +08:00
GoldPancake	cfc5b0ccf9	[BugFix] fix mtp logprob bugs in chunk prefill (#5244) ... * fix mtp logprob bugs in chunk prefill * fix * fix	2025-11-27 11:31:29 +08:00
chen	00d0ef5134	check (#5237)	2025-11-26 17:07:26 +08:00
Daci	f25ee3a26f	[Feature] enable guided decoding ENABLE_V1_KVCACHE_SCHEDULER = 1 (#5140) ... * enable guided decoding ENABLE_V1_KVCACHE_SCHEDULER = 1 * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-26 10:22:35 +08:00
kevin	8e4e3ff510	[Feature] support eplb in api_server (#4782) ... * support eplb in api_server * update code * add eplb test case * update eplb * support tp+dp eplb * update test cese * update code * update code * fix bug * update copilot review * update test case name	2025-11-24 20:22:29 +08:00
xiaozude	d5bd64336a	[Metax] support ENABLE_V1_KVCACHE_SCHEDULER (#5163)	2025-11-24 19:19:49 +08:00
xiaoxiaohehe001	95f3c8c641	[Fix] Fix eplb bug and support fp8 load weight (#5178) ... * fix eplb part2 * fix eplb part2 * fix eplb part2	2025-11-24 15:31:37 +08:00
kevin	c068a4f642	[Feature] dyc8 support prefixcache (#5125) ... * dyc8 support prefixcache * fix cache_trans test case * update code	2025-11-21 19:46:26 +08:00
ddchenhao66	e70e2279ce	[PD Disaggregation][XPU] Add XPU support for PD disaggregation (#5113) ... * [XPU] xpu support PD disaggregation * [XPU] fix the issue of cache KV transfer process startup failure on non-zero XPU cards * [XPU] xpu support PD disaggregation in v1 scheduler --------- Co-authored-by: ddchenhao66 <dhaochen163.com>	2025-11-21 14:09:01 +08:00
Yonghua Li	43097a512a	[BugFix] [PD Disaggregation] fix v1 scheduler prefill node profile run & ipc transfer protocol (#5132) ... * [fix] fix v1 scheduler profile run for append attention in prefill node * [fix] skip send_signal if kv signal not inited for gpu and xpu * [fix] extend fix to flash_attn & mla_attn * [fix] fix v1 pd run in ipc transfer protocol * [ci] add test for v1 pd profile run using ipc transfer protocol * [style] fix code style check * [style] fix code style again * [fix] fix profile run * [update] remove --num-gpu-blocks-override in example script * [chore] rename forward_meta is_profiling to is_dummy_or_profile_run	2025-11-20 21:39:22 +08:00