58 Commits

Author	SHA1	Message	Date
kevin	966297e5d6	[Feature] mm prefix cache (#4554) ... * mm prefix cache * add _revert_match_blocks * update code * update code * update code * fix bugs * add test case * fix bug * update code * update reserved_dec_block_ids	2025-11-19 19:32:14 +08:00
kevin	3ce2c8f754	[Feature] support async download features (#4910) ... * add async download * update code * fix bug * update code * update code * fix bugs * update code --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-11-18 18:37:59 +08:00
kevin	3dbe5596e6	[Feature] Support eplb for ep (#4786) ... * support eplb for ep * update code * update code * update code * update code * update code * update code * update code * update code * update code	2025-11-07 15:42:29 +08:00
Yuan Xiaolan	1e88754133	support set dy-C8 from args (#4475)	2025-11-04 17:01:35 +08:00
lizhenyun01	006c7e5a0d	[Feature] Support attention dp balance for mixed deployment (#4594) ... * [Feature] Support attention dp balance for mixed deployment * add abstractmethod * add config to rl * fix argument style	2025-10-29 15:23:51 +08:00
GoldPancake	9c7187998c	[Feature] support mtp logprob (#4457) ... * support logprob in mtp * remove debug code * fix * feat: add draft_logprobs for Speculative Decode MTP * Revert "feat: add draft_logprobs for Speculative Decode MTP" This reverts commit d5a3c5c933. * fix * feat: add draft_logprobs for Speculative Decode MTP * feat: add draft_logprobs for Speculative Decode MTP * fix some bugs * fix codestyle * fix bugs * fix bugs * fix bugs * fix bus * fix bugs * fix unitest --------- Co-authored-by: sunlei1024 <sunlei5788@gmail.com> Co-authored-by: sunlei18 <sunlei18@sunlei18deMacBook-Pro.local>	2025-10-20 10:18:00 +08:00
freeliuzc	c3499875bd	[MTP]support mtp chunk_prefill_v1 (#4365) ... * support mtp chunk_prefill_v1 * fix mtp chunkprefill output * fix mtp chunkprefill output, fix unit test * fix save_output --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-10-15 15:33:59 +08:00
ltd0924	28aa18bfc1	[Feature] support prefix cache + dp (#4356) ... * fix * fix * fix * [Feature] support clear data * update * fix * fix * fix * fix * [BugFix] fix clear data * Update api_server.py * Update api_server.py * [Feature] support fd decode response * Update engine.py * Update envs.py * Update expert_service.py * Update common_engine.py * [Feature] support prefix cache + ep * fix * add * Update expert_service.py * Update common_engine.py * Update common_engine.py * Update common_engine.py * Update common_engine.py * Update common_engine.py --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-11 10:02:08 +08:00
freeliuzc	e2b68b33c9	fix mtp in rl (#4234)	2025-09-24 16:59:24 +08:00
lzy	be98f6e950	supports internode_ll_two_stage (#4143) ... * supports internode_ll_two_stage * supports internode_ll_two_stage * supports internode_ll_two_stage * supports internode_ll_two_stage	2025-09-22 14:55:06 +08:00
chen	2485333f71	ep support logprob (#4089)	2025-09-12 21:11:16 +08:00
gaoziyuan	10768a4d79	[NewFeture]add ep rollout model init and update/clear ep buffer (#3927) ... * add ep rollout model init && add deep update/clear * fix test	2025-09-12 14:15:13 +08:00
freeliuzc	2f473ba966	[Feature][MTP]Support MTP for rl-model (#4009) ... * qk norm for speculate decode C16 * support mtp in v1_scheduler mode * support mtp rope_3d * support mtp features * add unit test && del some log --------- Co-authored-by: yuanxiaolan <yuanxiaolan01@baidu.com> Co-authored-by: xiaoxiaohehe001 <hiteezsf@163.com>	2025-09-10 13:34:37 +08:00
Yuanle Liu	71a9127e13	Update args_utils.py	2025-09-08 01:41:43 -07:00
chenjian	41cd3e24c9	[Feature] Enable prefix caching as default (#3816) ... * [Feature] Enable prefix caching as default * [Feature] Enable prefix caching as default * Set prefix caching as default * skip dynamic load * fix kill bug * fix kill bug * fix kill bug * fix ci * fix --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-06 09:51:34 +08:00
yinwei	77c1bd0813	[XPU]Fixed the issue of performance degradation caused by enabling ENABLE_V1_KVCACHE_SCHEDULER (#3900) ... * fix bug * fix bug * update * udpate * update	2025-09-05 19:17:25 +08:00
chenjian	8567ada09e	[Fix] disable scheduler v1 in guided decoding (#3877) ... * disable scheduler v1 in guided decoding * disable scheduler v1 in guided decoding	2025-09-04 20:54:55 +08:00
chenjian	fb1e0d6a87	[Feature] Set scheduler v1 as default (#3812) ... * [Feature] Set scheduler v1 as default * [Feature] Set scheduler v1 as default * [Feature] Set scheduler v1 as default * [Feature] Set scheduler v1 as default * [Feature] Set scheduler v1 as default * [Feature] Set scheduler v1 as default	2025-09-04 11:02:10 +08:00
kevin	a86b35ab49	Fix chunked prefill (#3778) ... * update enable chunked_prefill * update code * update code * update code	2025-09-02 13:41:55 +08:00
kevin	753772ace8	default enable chunked prefill (#3731) ... * add error traceback info * update error msg * update code * default enable chunked prefill * update code * update code * add envs * update code --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-31 13:15:13 +08:00
SunLei	b9af95cf1c	[Feature] Add AsyncTokenizerClient&ChatResponseProcessor with remote encode&decode support. (#3674) ... * [Feature] add AsyncTokenizerClient * add decode_image * Add response_processors with remote decode support. * [Feature] add tokenizer_base_url startup argument * Revert comment removal and restore original content. * [Feature] Non-streaming requests now support remote image decoding. * Fix parameter type issue in decode_image call. * Keep completion_token_ids when return_token_ids = False. * add copyright	2025-08-30 17:06:26 +08:00
yangjianfengo1	3754a9906d	[Feature] block sparse attention (#3668) ... * 支持稀疏attn * fix bug * code style * fix moba attn get kv shape * 修复a100编译 * codestyle * code style * code style * code style * fix conflict * 增加单侧 * code style * 增加eblite 加载时间 * fix bug * for ci * for ci * for ci * for ci * 支持mlp block size 128 * 增加小算子单测 * fix 单测 mlp * 将环境变量加入到config里面 * fix rollout config * 修复显存 * add test server * add test server * fix mlp 最后一层使用full attn	2025-08-29 19:46:30 +08:00
Jiang-Jia-Jun	c694fa2879	Revert "[Feature] block sparse attention (#3209)" (#3647) ... This reverts commit 646a0c2fd8.	2025-08-27 17:35:04 +08:00
chen	ce9c0917c5	[Precision] Support lm_head layer running in float32 (#3597) ... * support lm_head fp32 bf16 fp16 * support lm_head fp32 bf16 fp16 * add doc and check code * lm_head_fp32 specify lm_head as fp32 * code check * check doc	2025-08-27 11:34:53 +08:00
yangjianfengo1	646a0c2fd8	[Feature] block sparse attention (#3209) ... * 支持稀疏attn * fix bug * code style * fix moba attn get kv shape * 修复a100编译 * codestyle * code style * code style * code style * fix conflict * 增加单侧 * code style * 增加eblite 加载时间 * fix bug * for ci * for ci * for ci * for ci * 支持mlp block size 128 * 增加小算子单测 * fix 单测 mlp * 将环境变量加入到config里面 * fix rollout config	2025-08-26 07:16:04 -07:00
gaoziyuan	82e64b13e1	[NewFeature]Support dp multi api server && Fix some bug in mixed ep && merge develop (#3598) ... * [Feature] update ep * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix queue ports idx * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * Update engine.py * fix ci * fix some bug in mixed ep * add server fix and op fix * rm some log * fix code style * ltd fix * fix * fix * fix some bug * fix bug * fix bug * fix style * Update config.py * Update splitwise_connector.py * Update cache_messager.py * Update __init__.py * merge and fix * Update engine.py * Update common_engine.py * Update run_ci_xpu.sh * Update ernie_processor.py * Update ernie_processor.py --------- Co-authored-by: ltd0924 <ltd0924@sina.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com>	2025-08-26 19:59:02 +08:00
lizexu123	c43a4bec00	[Features] support hugging face qwen3 dense and qwen2 model (#3574) ... * support qwen2 and qwen3 hugging face * fix moe * defualt_v1 loader * hugging_face_format deprecated * modify hugging_face_foramt to model_format * model_format auto * fix environemt * fix bug * fix qwen3-0.6 bug * model_format is str * fix	2025-08-26 10:54:53 +08:00
zhink	df7c31012b	Modified to support custom all reduce by default (#3538)	2025-08-22 16:59:05 +08:00
YuanRisheng	c389a4013c	Unify server-side and model-side Config(Part-5) (#3497) ... * move config * fix xpu * fix * fix vl * fix vl * fix unitest * fix args * add unitest * fix test	2025-08-21 19:00:21 +08:00
Yzc216	466cbb5a99	[Feature] Models api (#3073) ... * add v1/models interface related * add model parameters * default model verification * unit test * check model err_msg * unit test * type annotation * model parameter in response * modify document description * modify document description * unit test * verification * verification update * model_name * pre-commit * update test case * update test case * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/entrypoints/openai/serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: LiqinruiG <37392159+LiqinruiG@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-08-21 17:02:56 +08:00
luukunn	9c129813f9	[Feature] add custom chat template (#3251) ... * add custom chat_template * add custom chat_template * add unittest * fix * add docs * fix comment * add offline chat * fix unit test * fix unit test * fix * fix pre commit * fix unit test * add unit test * add unit test * add unit test * fix pre_commit * fix enable_thinking * fix pre commit * fix pre commit * fix unit test * add requirements	2025-08-18 16:34:08 +08:00
luukunn	eda83ca672	add Tool Parser (#3272) ... * add tool-parser * add tool-parser * add tool parser * add tool parser * fix * add offline * add offline * fix * parsers:tool&reasoning * 修改tool parser名称· * update * fix reasoning-parser * add requirements * fix finish reason * fix * fix reasoning-parser * fix * fix * fix * fix * fix --------- Co-authored-by: zhuzixuan <zhuzixuan@baidu.com>	2025-08-13 01:06:55 +08:00
chenjian	b21272d9ff	[Bug fix] fix block num setting in scheduler v1 for develop (#3303) ... * fix block num setting in scheduler v1 * fix block num setting in scheduler v1 * fix max_block_num and max_num_batched_tokens setting * fix max_block_num and max_num_batched_tokens setting * fix max_block_num and max_num_batched_tokens setting * fix max_block_num and max_num_batched_tokens setting	2025-08-12 10:38:51 +08:00
chen	46c8491201	merge logprob into batch_output (#3266)	2025-08-11 10:03:00 +08:00
lizexu123	afff4d37ea	[Feature] support seed parameter (#3161) ... * support seed * fix * add SamplingMetadata seed test * The next_tokens values are inconsistent! * add air and rejection seed test * fix * add SamplingParams seed test * fix seed=0 * Default to defualt * fix * fix args_utils * fix review * fix review * fix * fix * add xpu,gcu,iluvatar support seed * fix	2025-08-06 15:20:47 +08:00
ApplEOFDiscord	b71cbb466d	[Feature] remove dependency on enable_mm and refine multimodal's code (#3014) ... * remove dependency on enable_mm * fix codestyle check error * fix codestyle check error * update docs * resolve conflicts on model config * fix unit test error * fix code style check error --------- Co-authored-by: shige <1021937542@qq.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-01 20:01:18 +08:00
bukejiyu	db698bda01	qwen loader (#3057)	2025-07-30 19:09:38 +08:00
YuanRisheng	99a70fc722	unify parallel config (#3070)	2025-07-30 11:41:23 +08:00
Zero Rains	b2f9a42d87	[Feature] Support repetition early stop (#3024) ... * support repetition early stop and support user to set the parameter * remove log * fix codestyle * add the early_stop_config to rollout_config * update config and EarlyStopper class * fix the bug for triton * modify the stop method * update description * modify the usage for stop_flags --------- Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-07-29 22:42:54 +08:00
YuanRisheng	502ee92a0a	Unify server-side and model-side Config (Part3) (#3047) ... * merge model config * fix arch * fix rl	2025-07-29 17:07:44 +08:00
YuanRisheng	bddf403576	Unify server-side and model-side Config (Part2) (#3035) ... * merge speculative and graph opt conifg * add attr	2025-07-28 15:31:48 +08:00
YuanRisheng	6ccc10ad47	Unify server-side and model-side Config (Part1) (#3018) ... * move cache config * fix mtp	2025-07-28 10:51:52 +08:00
Zero Rains	0fb37ab7e4	update flake8 version to support pre-commit in python3.12 (#3000) ... * update flake8 version to support pre-commit in python3.12 * polish code	2025-07-24 01:43:31 -07:00
ltd0924	f935d6f862	[BugFix] fix multinode deployment (#2977)	2025-07-24 15:04:04 +08:00
Yzc216	e14587a954	[Feature] multi-source download (#2986) ... * multi-source download * multi-source download * huggingface download revision * requirement * style * add revision arg * test * pre-commit	2025-07-24 14:26:37 +08:00
chenjian	85a78d695d	[Feature] Support block scheduler v1 for FD (#2928) ... * Support FD block scheduler v1 * Support FD block scheduler v1 * Support FD block scheduler v1 * Fix according to copilot review * Fix according to review * Remove is_dummy * Fix bug when real_bsz=1 * Fix infer first token cost time --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-07-23 20:31:31 +08:00
Zero Rains	89a485b69f	[Feature] Support using prefix-caching + cudagraph for inference (#2924) ... * fix the bug in cudagraph+prefix-caching but still have some bug with profile Change-Id: Ibf2ba3f2e3b08641d03f4b1391d7c862c3efa397 * add the signal to make sure cache manager launched * fix judge condition * reomove useless control * update control stream * update * fix xpu * change the do_profile flag * update * add new threads to init cache_manager --------- Co-authored-by: RAM <gstian5555@outlook.com>	2025-07-22 00:59:45 -07:00
zhink	0262ef7eb3	custom all reduce support cuda graph (#2938) ... * Support enabling cuda graph and custom all reduce at the same time, and fix the overwritten custom all reduce flag * rename communication_op to communication	2025-07-21 22:52:03 +08:00
Zero Rains	25698d56d1	polish code with new pre-commit rule (#2923)	2025-07-19 23:19:27 +08:00
ltd0924	9c25dcca0b	[LLM] Update Multinode Deployment (#2830) ... * [LLM] fix multinode bugs * [LLM] update multinode deployment * [LLM] update multinode deployment * [LLM] update multinode deployment * [LLM] update multinode deployment * [LLM] update multinode deployment * [LLM] fix ci bugs * Update fastdeploy/engine/args_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [LLM] update random port * [LLM] update random port * [LLM] fix ci bugs * fix ci bugs --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-16 23:42:54 +08:00