65 Commits

Author	SHA1	Message	Date
Juncai	80efe98f8d	[PD Disaggregation] Add timestamp for analyzing splitwise deployment (#5317) ... * Add timestamp for analyzing splitwise deployment * up * up * up * up * up * up * fix format * fix	2025-12-08 10:08:44 +08:00
chenjian	3878a99b69	[Fearture] Support cache kv cache for output tokens (#4535) ... * [Fearture] Support cache kv cache for output tokens * fix bug * fix ci bug * improve coverage * enable output caching by default * fix ci --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-04 20:53:08 +08:00
Yonghua Li	f4119d51b4	[PD Disaggregation] support DP via v1 router and decouple DP and EP (#5197) ... * [fix] support DP via v1 router and decouple DP and EP * [fix] fix scripts * [fix] reset model path * [fix] dp use get_output_ep, fix router port type, update scripts * [merge] merge with latest code * [chore] remove some debug log * [fix] fix code style check * [fix] fix test_multi_api_server for log_dir name * [chore] reduce logs * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-12-04 15:38:43 +08:00
qw86972190	6048ea37bd	[XPU]add enable_logprob (#5279) ... * [XPU]Update document * [XPU]Update documentation * [XPU]add enable_logprob * Fix code style issues * “doc” * “docs” * “doc” * Fix code style via pre-commit --------- Co-authored-by: root <root@gajl-bbc-onlinec-com-1498354.gajl.baidu.com>	2025-12-02 15:32:28 +08:00
qwes5s5	117980dd4e	[LogProbs]Enable prompt logprobs output and modify data transmission method for the online interface. (#5089) ... * add prompt logprobs * Merge prompt_logprobs_tensors and prompt_logprobs * fix param check * trigger ci * fix unitest * fix logprobs bug	2025-12-02 13:49:51 +08:00
cmcamdy	9f4977eb74	[xpu] support mtp for xpu(mix) (#5274) ... * [XPU] support kernel for mtp(base) * [XPU] support kernel for mtp(base) * format * format * format * fix gather next token * fix step && add test * fix * mv pre/post process * add adjust batch / gather next token for mtp * fix code style * fix mtp kenrel name * fix mtp kernel test * mv xpu pre/post process * mv xpu pre/post process * [xpu] support mtp * fix code style	2025-12-01 11:03:14 +08:00
GoldPancake	cfc5b0ccf9	[BugFix] fix mtp logprob bugs in chunk prefill (#5244) ... * fix mtp logprob bugs in chunk prefill * fix * fix	2025-11-27 11:31:29 +08:00
SunLei	c424e08dc5	[Speculative Decoding] split draft_tokens into standalone post-processing path (#5205) ... * refactor(mtp): split draft_tokens into standalone post-processing path for MTP + logprobs * Restore Request.__repr__ implementation * ci * add envs * fix unittest	2025-11-27 11:22:41 +08:00
Yonghua Li	cead6b26fa	[Metrics] Update time_to_first_token to include tokenization & queue time, and remove redundant metrics (#4993) ... * [update] update time_to_first_tokens to include queue time, and remove first_token_latency and infer_latency * [doc] update docs * [ci] fix test * [chore] delete redundant code --------- Co-authored-by: Jiaxin Sui <95567040+plusNew001@users.noreply.github.com>	2025-11-26 14:42:17 +08:00
GoldPancake	ab3a2e45ff	fix mtp reschedule (#5165)	2025-11-21 19:31:35 +08:00
chenjian	3ea1b44a58	[Optimization] Improve perf for fd response token with internal adapter (#4992) ... * [Optimize] Improve perf for fd response token with internal adapter * fix * fix bug * fix ci * fix ci * fix ci * fix ci	2025-11-21 19:02:03 +08:00
SunLei	d9f64adb0e	fix: Fix block allocation issue when MTP and logprobs are enabled (#5077)	2025-11-17 17:50:07 +08:00
qwes5s5	36216e62f0	[Log] Add trace log and add loggingInstrumentor tool (#4692) ... * add trace logger and trace print * trigger ci * fix unittest * translate notes and add copyright --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-11-17 11:08:57 +08:00
Juncai	36822fa49c	[PD Disaggregation] remove splitwise deployment on single node and refine the code (#4891) ... * remove splitwise deployment on single node and refine the code * up * up * up * add test * up	2025-11-14 09:56:53 +08:00
Yonghua Li	6c5ab727c1	[BugFix] fix num_requests_running after clear_data (#4927) ... * [BugFix] fix num_requests_running after clear_data * [fix] fix tasks_list and stop flags not cleared when _free_blocks failed	2025-11-13 13:50:21 +08:00
qwes5s5	a2d06118e1	[Logprobs]Support prompt_logprobs and max_logprobs (#4897) ... * add prompt logprobs * trigger ci * fix unitest * Update fastdeploy/config.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/entrypoints/llm.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/engine/sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/engine/test_sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/engine/test_sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix max_logprobs --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-12 19:29:48 +08:00
SunLei	3098aee05f	[Perf] Support tensor transmission between work and engine with zero-copy to improve efficiency (#4839) ... * feat(zmq): support tensor transmission with zero-copy for improved efficiency * perf: zmq.send disable copy * zmq recv data for debug * convert logprobs tensor to cpu	2025-11-11 15:43:11 +08:00
chen	6871aad03d	[BugFix] fix token_processor zmq (#4827) ... * fix_token_processor_zmq * check pooling_model's token_ids is None * revert	2025-11-07 19:43:25 +08:00
Juncai	08ca0f6aea	[Feature] [PD] add simple router and refine splitwise deployment (#4709) ... * add simple router and refine splitwise deployment * fix	2025-11-06 14:56:02 +08:00
chenjian	cc8f5312f5	[Feature] Add timestamp for profiler (#4726) ... * [Feature] Add timestamp for profiler * fix bug for offine inference * fix for ci * fix * fix ci	2025-11-05 12:04:59 +08:00
chen	1c3ca48128	[Feature][Executor] GPU Model Runner Supports prompt_logprobs and max_logprobs (#4769)	2025-11-05 10:43:25 +08:00
lizexu123	4ac6de9a3c	[Feature] support pooling model runner (#4590) ... * support qwen3-embedding * support qwen3-embedding-0.6b * fix * fix bug * fix test_return_token_ids.py and update enable_thinking * fix mtp dummy_run * merge develop * fix np.float32 * delete FD_DISABLE_CHUNKED_PREFILL and FD_USE_GET_SAVE_OUTPUT_V1 * delete and build_stream_transfer_data * fix test_update_v1: * fix * fix * update dummy_run post_process * delete test_update_v1 * fix * fix dummy_run * fix model_path * fix model_path * fix dummy_run	2025-10-31 22:32:05 +08:00
ApplEOFDiscord	14f8cddaf1	[Feature] add mm token usage (#4570) ... * add mm token usage * fix unit test * fix unit test * fix unit test * fix model path * fix unit test * fix unit test * fix unit test * remove uncomment * change var name * fix code style * fix code style * fix code style * fix code style * fix unit test	2025-10-29 14:37:12 +08:00
freeliuzc	c63361fd1d	[Speculative Decoding][MTP]Support mtp in epdptp mode (#4614) ... * support mtp many features * support mtp reshard in rl mode * fix function * support mtp ep * support mtp in hybird-dp-tp mode * default open scheduler_v1 in mtp	2025-10-28 16:02:47 +08:00
SunLei	809c1ac7ec	feat: add post-processing step for pool_output (#4462) ... * feat: add post-processing step for pool_output * bugfix * fix: test_serving_embedding * fix test_request_to_batch_dicts * fix: code style	2025-10-21 20:24:26 +08:00
ltd0924	fb76cdfb4f	[Fearture] Support mm model close prefix cache (#4459) ... * [Feature] support prefix cache in DP * fix * Update common_engine.py * Update common_engine.py * Update common_engine.py * Update common_engine.py * [BugFix] fix workers more than 1 * fix * Update api_server.py * fix * Update api_server.py * fix * [Fearture] Support mm model close prefix cache * Update api_server.py * Update engine_client.py * Update engine_client.py * add test * Update test_chat.py * fix * fix * Update test_chat.py * Update test_chat.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-10-21 15:37:59 +08:00
SunLei	ee915220bd	[Speculative Decoding] Add draft_logprobs Support for Speculative Decode MTP (#4467) ... * feat: add draft_logprobs for Speculative Decode MTP * feat: add draft_logprobs for Speculative Decode MTP * feat: add draft_logprobs for Speculative Decode MTP * fix: postprocess for speculative decode * test: test_speculative_decoding_use_logprobs * fix: test_completion_echo * fix test_max_streaming_tokens --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-10-21 14:57:50 +08:00
李泳桦	ffe7af8a97	[fix] fix requests & block metrics (#4404) ... * [fix] fix requests & block metrics * [chore] rename variables	2025-10-15 11:49:24 +08:00
RichardWooSJTU	3740e33fea	【Feature】ResourceManagerV1 support need block num notifying (#4220) ... * support need block num notifying * adapt t2i * fix unexpected change	2025-09-29 11:11:51 +08:00
fmiao2372	f1b5392e20	[Intel HPU] Support intel hpu platform (#4161) ... * [Intel HPU] Support intel hpu platform * fix some issues * apply precommit and move AttentionBackend_HPU * fix format issue * correct ops import * fix ci issue * update code in layers * fix code style issue * remove dense tp moe ep mode * fix enc_dec_block_num * fix rebase issue * rename hpu to gaudi in readme * rename ForwardMeta_HPU to HPUForwardMeta	2025-09-24 12:27:50 +08:00
ltd0924	83720da79f	[Feature] support clear data (#3601) ... * [Feature] support clear data * update * fix * fix * fix * fix * fix * fix * fix	2025-09-23 10:20:02 +08:00
chenjian	918ccdb123	[Feature] Support pd ep deployment with yiyan adapter (#4029) ... * [Feature] Support mixed deployment with yiyan adapter in release2.2 * fix metrics * add unit test * add unit test * add unit test * Support pd ep deployment with yiyan adapter * Support pd ep deployment with yiyan adapter * refactor cache messager * support scheduler v1 in PD * suppport pd v1 + chunk prefill * suppport pd v1 + chunk prefill * add eplb * support eplb * support eplb * support eplb * support v1 * fix * fix * fix bug * remove eplb support * support prefix cache in P * fix bug * fix bug * support one stop in V1 * fix bug * fix ci * fix ci * fix * fix * fix * fix * fix --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-22 16:41:38 +08:00
lizexu123	c86945ef49	[Feature] support pool (#3827) ... * support pool * update pooling * add pooler_config and check * update * support AutoWeightsLoader load weight * fix * update * delete print * update pre-commit * fix * fix xpu * fix ModelRegistry->model_registry * fix Copilot review * fix pooler.py * delete StepPooler * fix abstract * fix default_loader_v1 * fix Pre Commit * support torch qwen3 dense * add test and fix torch-qwen * fix * fix * adapter ci: * fix review * fix pooling_params.py * fix * fix tasks.py 2025 * fix print and logger * Modefy ModelRegistry and delete AutoWeightsLoader * fix logger * fix test_embedding * fix ci bug * ernie4_5 model_registry * fix test * support Qwen3-Embedding-0.6B tp=1 load * fix extra code * fix * delete fix vocab_size * delete prepare_params_dict * fix:	2025-09-22 14:09:09 +08:00
chen	66a98b44ed	ep support logprob (#4089) (#4151)	2025-09-19 14:07:31 +08:00
chenjian	618ccdbfba	[Feature] Support mixed deployment with yiyan adapter in develop (#3976) ... * [Feature] Support mixed deployment with yiyan adapter in release2.2 * fix metrics * add unit test * add unit test * add unit test * fix ci * fix for eb5 * fix ci * fix ci * fix ci --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-18 01:52:20 +08:00
RichardWooSJTU	2adca04f1f	Reconstruct streaming data transfer with zmq (#3836) ... * reconstruct USE_GET_SAVE_OUTPUT_V1 * fix ut * use dp rank * fix ci	2025-09-17 14:30:39 +08:00
zhupengyang	9409665713	[xpu] support ep (#4067)	2025-09-15 13:53:11 +08:00
co63oc	8466219ec8	fix typos (#3840) ... * fix typos * ci --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-12 11:04:38 +08:00
qwes5s5	17169a14f2	[metrics] Add serveral observability metrics (#3868) ... * Add several observability metrics * [wenxin-tools-584] 【可观测性】支持查看本节点的并发数、剩余block_size、排队请求数等信息 * adjust some metrics and md files * trigger ci * adjust ci file * trigger ci * trigger ci --------- Co-authored-by: K11OntheBoat <your_email@example.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-08 14:13:13 +08:00
Longzhi Wang	e0c9a6c76c	[Feat] Support streaming transfer data using ZMQ (#3521) ... * Support streaming transfer data of ZMQ * fix typo * fix typo * support tp * add unittest * update * update * fix typo * fix typo * fix tp_num in ci machine --------- Co-authored-by: Wanglongzhi2001 <>	2025-09-02 19:52:19 +08:00
co63oc	d6369b4d51	fix typos (#3684)	2025-09-01 17:50:17 +08:00
李泳桦	98e03fb4ea	[feat] add metrics for yiyan adapter (#3219) (#3614) ... * [feat] add metrics for yiyan adapter * [fix] fix metrics num_requests_waiting and num_requests_running * [fix] fix metrics gpu_cache_usage_perc * [refactor] change where requests_number increases * [chore] rename xxx_block_num as xxx_gpu_block_num, and update their values accordingly * [chore] delete useless code	2025-08-30 23:20:58 +08:00
freeliuzc	52eda7fdb3	[Feature][MTP]support new speculative decoding method named hybrid mtp with ngram (#3610)	2025-08-26 14:29:22 +08:00
YuanRisheng	c389a4013c	Unify server-side and model-side Config(Part-5) (#3497) ... * move config * fix xpu * fix * fix vl * fix vl * fix unitest * fix args * add unitest * fix test	2025-08-21 19:00:21 +08:00
kevin	67298cf4c0	add error traceback info (#3419) ... * add error traceback info * update error msg * update code --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-19 19:32:04 +08:00
Zero Rains	b23af29d0b	Launch expert_service before kv_cache initialization in worker_process (#3045) ... * launch expert_service before kv_cache initialization * add two signal make sure model loading and expert_service lauching finished * fix the EP bug * fix ep * update launching way * fix ep * update * roback ep * pre-commit all files --------- Co-authored-by: RAM <gstian5555@outlook.com> Co-authored-by: Divano <dddivano@outlook.com>	2025-08-11 19:38:46 +08:00
chenjian	b88537a456	fix bug for scheduler v0 (#3308)	2025-08-11 13:07:04 +08:00
chen	46c8491201	merge logprob into batch_output (#3266)	2025-08-11 10:03:00 +08:00
chenjian	c011cb8b16	[Bug Fix] Fix scheduler bug in develop (#3292) ... * Fix scheduler bug in develop * Fix scheduler bug in develop * Fix scheduler bug in develop	2025-08-10 13:55:38 +08:00
chenjian	5f0b30f6d0	support logprob in scheduler v1 (#3249) ... Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-07 20:14:01 +08:00