270 Commits

Author	SHA1	Message	Date
Daci	f25ee3a26f	[Feature] enable guided decoding ENABLE_V1_KVCACHE_SCHEDULER = 1 (#5140) ... * enable guided decoding ENABLE_V1_KVCACHE_SCHEDULER = 1 * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-26 10:22:35 +08:00
kevin	8e4e3ff510	[Feature] support eplb in api_server (#4782) ... * support eplb in api_server * update code * add eplb test case * update eplb * support tp+dp eplb * update test cese * update code * update code * fix bug * update copilot review * update test case name	2025-11-24 20:22:29 +08:00
xiaozude	d5bd64336a	[Metax] support ENABLE_V1_KVCACHE_SCHEDULER (#5163)	2025-11-24 19:19:49 +08:00
xiaoxiaohehe001	95f3c8c641	[Fix] Fix eplb bug and support fp8 load weight (#5178) ... * fix eplb part2 * fix eplb part2 * fix eplb part2	2025-11-24 15:31:37 +08:00
kevin	c068a4f642	[Feature] dyc8 support prefixcache (#5125) ... * dyc8 support prefixcache * fix cache_trans test case * update code	2025-11-21 19:46:26 +08:00
ddchenhao66	e70e2279ce	[PD Disaggregation][XPU] Add XPU support for PD disaggregation (#5113) ... * [XPU] xpu support PD disaggregation * [XPU] fix the issue of cache KV transfer process startup failure on non-zero XPU cards * [XPU] xpu support PD disaggregation in v1 scheduler --------- Co-authored-by: ddchenhao66 <dhaochen163.com>	2025-11-21 14:09:01 +08:00
Yonghua Li	43097a512a	[BugFix] [PD Disaggregation] fix v1 scheduler prefill node profile run & ipc transfer protocol (#5132) ... * [fix] fix v1 scheduler profile run for append attention in prefill node * [fix] skip send_signal if kv signal not inited for gpu and xpu * [fix] extend fix to flash_attn & mla_attn * [fix] fix v1 pd run in ipc transfer protocol * [ci] add test for v1 pd profile run using ipc transfer protocol * [style] fix code style check * [style] fix code style again * [fix] fix profile run * [update] remove --num-gpu-blocks-override in example script * [chore] rename forward_meta is_profiling to is_dummy_or_profile_run	2025-11-20 21:39:22 +08:00
Daci	eab8384da6	[Feature] ThreadPoolExecutor async fill_token_bitmask (#5083) ... * ThreadPoolExecutor async fill_token_bitmask * ThreadPoolExecutor async fill_token_bitmask logging * fix test_guided_decoding * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add fill_bitmask_parallel_batch_size ENV * FD_FILL_BITMASK_BATCH fastdeploy.envs --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-19 10:04:16 +08:00
K11OntheBoat	4a7739ec0b	Fix dummy run when use PD Disaggregation with EP inference. (#5112) ... Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-11-18 21:09:30 +08:00
fmiao2372	74f33efdbf	[Intel HPU] fix bugs caused by other commits (#5074) ... * [Intel HPU] fix bugs caused by other commits * update code by copilot	2025-11-17 15:28:55 +08:00
fmiao2372	e43a5fc055	[Intel HPU] enable level 1 prefix caching and fix some bugs (#4971) ... * [Intel HPU] enable prefix caching and dense tp moe ep and fix some bugs * update code by copilot * remove dense tp and moe ep code	2025-11-14 19:42:50 +08:00
Daci	5fc12eddfe	[Optimization] xgrammar async compile, multi thread, speed up (#4835) ... * xgrammar async compile, multi thread, speed up * fix test_sampler.py & pre-commit err * add redis version check && fix request.llm_engine_recv_req_timestamp * xgrammar prefill & decode & v0 * fix test_gpu_prompt_logprobs.py * add test_guided_decoding.py * Update fastdeploy/scheduler/splitwise_scheduler.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/model_executor/guided_decoding/xgrammar_backend.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/model_executor/guided_decoding/xgrammar_backend.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix torch xgrammar unittest env --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-14 18:05:26 +08:00
周周周	51b1f13547	[Executor]move batch_id_per_token (#4853)	2025-11-14 15:38:48 +08:00
Juncai	36822fa49c	[PD Disaggregation] remove splitwise deployment on single node and refine the code (#4891) ... * remove splitwise deployment on single node and refine the code * up * up * up * add test * up	2025-11-14 09:56:53 +08:00
周周周	6c4ebc5fee	[worker_process.py]modify some var name (#4749)	2025-11-13 14:21:27 +08:00
qwes5s5	a2d06118e1	[Logprobs]Support prompt_logprobs and max_logprobs (#4897) ... * add prompt logprobs * trigger ci * fix unitest * Update fastdeploy/config.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/entrypoints/llm.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/engine/sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/engine/test_sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/engine/test_sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix max_logprobs --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-12 19:29:48 +08:00
ltd0924	5bf48de999	[KVCache] support unified cache backend (#4903) ... * [Feature] support unified cache backend * fix * fix * fix * fix * Update metax_model_runner.py * fix * update * Update test_moba_attention_backend.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-11-12 14:54:52 +08:00
周周周	da6b4c10e5	[ATTENTION] make buffer alloc as a function (#4945)	2025-11-11 19:17:08 +08:00
chen	896ef565cc	[Others] Add Tests for GPU Model Runner and Logprobs Output (#4913)	2025-11-11 18:37:33 +08:00
K11OntheBoat	76be598129	replace paddle.max by numpy to avoid useless error log (#4893) ... Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-11-11 16:28:05 +08:00
SunLei	3098aee05f	[Perf] Support tensor transmission between work and engine with zero-copy to improve efficiency (#4839) ... * feat(zmq): support tensor transmission with zero-copy for improved efficiency * perf: zmq.send disable copy * zmq recv data for debug * convert logprobs tensor to cpu	2025-11-11 15:43:11 +08:00
Yuanle Liu	3dc0ffa46d	[TSP] Support qwen3 moe tsp + cudagraph (#4871) ... * support qwen3_moe tsp mode * fix * fix * update * update * update * fix * support external_rmsnorm * update * fix	2025-11-10 23:37:51 +08:00
chenjian	78895e2c7d	[Bug Fix] fix bug for PD EP (#4823) ... * fix bug for PD EP * fix * optimize perf for engine worker queue * fix bug * fix internode ll two stage * fix for ci * fix bug	2025-11-10 15:33:29 +08:00
chen	80aedb82ce	[BugFix] max_lgprobes=-1 maps to ori_vocab_size (#4884) ... * -1 ori_vobal_size * check * check * check * revert config.py	2025-11-07 22:15:40 +08:00
Neil Zhu	6de1ce3b25	[Metax] support ERNIE-4.5-VL-28B (#4820)	2025-11-07 04:55:49 -08:00
Juncai	08ca0f6aea	[Feature] [PD] add simple router and refine splitwise deployment (#4709) ... * add simple router and refine splitwise deployment * fix	2025-11-06 14:56:02 +08:00
周周周	69fa741763	remove seq_lens_this_time (#4821)	2025-11-06 11:06:28 +08:00
K11OntheBoat	62dfad4a5f	[PD Disaggregation] Support Qwen3-MoE use PD + EP inference. (#4691) ... support Qwen-MoE PD/EP	2025-11-06 10:32:15 +08:00
周周周	876e4a8935	remove input_ids from ForwardMeta (#4793)	2025-11-05 11:55:51 +08:00
zhupengyang	2fd254e5b7	support ep+tp at op layer (#4688)	2025-11-05 11:15:57 +08:00
chen	1c3ca48128	[Feature][Executor] GPU Model Runner Supports prompt_logprobs and max_logprobs (#4769)	2025-11-05 10:43:25 +08:00
李泳桦	1b61d62ecf	[fix] fix v0 pd, let worker step_shm_value create=False (#4780)	2025-11-04 20:37:57 +08:00
lzy	af7e0f27f3	supports internode_ll_two_stage (#4162) ... * supports internode_ll_two_stage * supports internode_ll_two_stage * supports internode_ll_two_stage * supports internode_ll_two_stage * supports D internode_ll_two_stage * fix codestype * fix xpu internode_ll_two_stage * fix xpu internode_ll_two_stage	2025-11-04 16:35:40 +08:00
ddchenhao66	bffa08b74b	[XPU] fix thinking bug where output only contains reasoning_content (#4761) ... Co-authored-by: ddchenhao66 <dhaochen163.com>	2025-11-04 14:32:35 +08:00
Neil Zhu	c95d0740ec	[Metax] adapt cutlass moe for ernie-vl (#4685)	2025-11-03 17:44:27 +08:00
chenjian	25498efcf3	[Optimize] Support and robust for tpN for PD (#4595) ... * [Optimize] Support and robust for tpN for PD * fix * fix * support dpM tpN for cache messager * fix * fix token counter * fix bug for merge develop * fix bug * robust cache messager for v0	2025-11-03 15:38:31 +08:00
chenjian	f83d0cf127	[Feature] Support eplb for fd (#4599) ... * support eplb * support eplb --------- Co-authored-by: kevin <chengyf112@gmail.com>	2025-11-03 14:08:15 +08:00
freeliuzc	11398790d3	[Speculative Decoding][MTP]Support attn mask offset (#4641) ... * [MTP]Merge support attn (#4591) * support mask_offset in speculate decoding * fix dummpy run output * add unit test * fix unit test import * support attn_mask_offset in mtp mode * add update_attn_mask op * fix unit test && fix code-style	2025-11-03 10:08:01 +08:00
lizexu123	4ac6de9a3c	[Feature] support pooling model runner (#4590) ... * support qwen3-embedding * support qwen3-embedding-0.6b * fix * fix bug * fix test_return_token_ids.py and update enable_thinking * fix mtp dummy_run * merge develop * fix np.float32 * delete FD_DISABLE_CHUNKED_PREFILL and FD_USE_GET_SAVE_OUTPUT_V1 * delete and build_stream_transfer_data * fix test_update_v1: * fix * fix * update dummy_run post_process * delete test_update_v1 * fix * fix dummy_run * fix model_path * fix model_path * fix dummy_run	2025-10-31 22:32:05 +08:00
ddchenhao66	3cbca75cc8	[XPU] xpu support neox style ROPE (#4719) ... Co-authored-by: ddchenhao66 <dhaochen163.com>	2025-10-31 18:14:25 +08:00
李泳桦	0f75b62de2	[BugFix] Fix profile run in pd-disaggregated deployment (#4584) ... * [fix] fix pd+dp+ep bug * [fix] fix again * [ci] fix code style	2025-10-31 14:42:00 +08:00
kevin	64e875b460	[Scheduler] update v1 prefill batch (#4611) ... * update v1 prefill batch * update code * update code	2025-10-31 14:03:01 +08:00
kxz2002	a2870ed4a9	[Feature] Unify the registration name recognition for tool_parser and reasoning_parser to “-” (#4668) ... * parser register name unify * change ernie_x1 to ernie-x1 * change ernie4_5_vl to ernie-45-vl * fix unit test	2025-10-31 10:45:27 +08:00
RAM	cd3b7cc392	[Graph Optimization] Add the CUDAGraph usage switch for Draft Model (#4601) ... * add draft model using cudagraph switch * set default as false * capture draft model in ci * fix bug	2025-10-30 11:44:50 +08:00
Lucas	8f40dfa9bf	[XPU] fix pos_emb_type bug (#4638)	2025-10-29 17:14:32 +08:00
RichardWooSJTU	0dde936e93	[BugFix] fix total_block_num init error in worker_process (#4553) ... * fix total_block_num init error in worker_process * fix req and token client * fix req and token client * fix xpu xi * fix xpu ci	2025-10-28 20:42:12 -07:00
李泳桦	a012e3608b	[Feature] support logits processors (#4515) ... * [feat] provide an interface for logits processors and a builtin LogitBiasLogitsProcessor * [chore] fix code style * [fix] add unit test & fix existing bugs * [feat] add engine/worker arg --logits-processors * [fix] redefine user args as logits_processors_args and fix some bugs * [fix] fix test_sampler * Update fastdeploy/model_executor/logits_processor/builtin.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/model_executor/logits_processor/__init__.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/model_executor/test_logits_processor.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [fix] fix typo * Update fastdeploy/engine/sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [fix] fix bracelet * [chore] redefine logits processor interface: pass the entire share_inputs into LP, do not copy share_inputs and logits * [doc] add docs * [fix] fix logit bias processor not applied when decoding is too fast & add docs and tests * [fix] fix redundant code * [feat] skip apply() if no bias is specified --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-10-29 00:08:53 +08:00
ming1753	561b9f38d3	[BugFix] fix paddleocr prefix cache bug (#4625) ... * fix paddleocr prefix cache bug * disable prefix-caching in ocr	2025-10-28 21:38:12 +08:00
Lucas	0a0c74e717	[XPU] Support PaddleOCR-VL model for XPU (#4529) ... * [XPU] support PaddleOCR-VL in XPU * [XPU] fix PaddleOCR-VL pos_emb_type	2025-10-28 20:35:04 +08:00
lizhenyun01	4d2f478d53	[BugFix] fix TPDP mix parallel infer (#4583) ... Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-10-28 16:58:20 +08:00