100 Commits

Author	SHA1	Message	Date
ming1753	e4e3cede7f	[Feature] Support Paddle-OCR (#4396) ... * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com>	2025-10-24 23:34:30 +08:00
RichardWooSJTU	5a8c60454e	[BugFix] Fix decode_type which has been deleted in req and optimize token client retry scheme (#4564)	2025-10-23 05:08:10 -07:00
luukunn	bbf06b9ff7	[BugFix]Fix finish reason (#4543) ... * fix finish reason * add unit test * add unit test * fix unie test * fix unit test	2025-10-23 14:04:43 +08:00
SunLei	809c1ac7ec	feat: add post-processing step for pool_output (#4462) ... * feat: add post-processing step for pool_output * bugfix * fix: test_serving_embedding * fix test_request_to_batch_dicts * fix: code style	2025-10-21 20:24:26 +08:00
SunLei	ee915220bd	[Speculative Decoding] Add draft_logprobs Support for Speculative Decode MTP (#4467) ... * feat: add draft_logprobs for Speculative Decode MTP * feat: add draft_logprobs for Speculative Decode MTP * feat: add draft_logprobs for Speculative Decode MTP * fix: postprocess for speculative decode * test: test_speculative_decoding_use_logprobs * fix: test_completion_echo * fix test_max_streaming_tokens --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-10-21 14:57:50 +08:00
ltd0924	a498736af5	[APIServer] support define gunicorn timeout (#4496) ... * [BUGFIX] clear request #4286 * [BugFix] support define gunicorn timeout * Update utils.py * Update utils.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-20 23:36:07 +08:00
Yuanle Liu	cef3164c3b	Optimizing the performance of think length limit using custom operators (#4279) ... * delete impl * delete min_length&max_length * support limit thinking content strategy * fix * fix * fix * update * fix set_value_by_flags_and_idx * fix * fix * fix * fix * update * fix * fix * fix typo * fix ci * fix * fix * support mtp * fix * fix * update * update	2025-10-20 21:09:13 +08:00
kxz2002	b5b993e48e	【feature】support n parameter (#4273) ... * support n parameter * pre-commit check * pre-commit check * restore format_and_add_data * update n_param * bug fix index - str to int * bug fix del child_task * bug fix metrics * add debug info * add debug info2 * remove debug info * change connecting symbol to '-' * bugfix change connecting symbol * bugfix change connecting symbol2 * unit tests fix * unit test fix2 * unittest add param n=2 * n param add unit tests and adapt to echo * pre-commit fix * resolve review * adjust stop reason * add unittest for _create_chat_completion_choice * modify unittest * solve confict * solve conflict * resolve conflict --------- Co-authored-by: LiqinruiG <37392159+LiqinruiG@users.noreply.github.com> Co-authored-by: gaoziyuan <m13689897706@163.com>	2025-10-17 20:51:59 +08:00
LiqinruiG	4251ac5e95	【Fix】 remove text_after_process & raw_prediction (#4421) ... * remove text_after_process & raw_prediction * remove text_after_process & raw_prediction	2025-10-16 19:00:18 +08:00
SunLei	b4b579a7ed	Feature：Add support for Pooling Model Embedding and provide an OpenAI-compatible API. (#4344) ... * feat: add OpenAIServing * feat: add ZmqOpenAIServing & OpenAIServingEmbedding * feat: Refine the basic ServingEngine class and introduce ServingContext * fix: codestyle * fix: request * fix: pooling_params * feat: _process_chat_template_kwargs * feat: support batch request * feat: pooling_params verify & default parameters --------- Co-authored-by: sunlei1024 <sunlei1024@example.com>	2025-10-15 19:42:59 +08:00
ltd0924	fbdb056de0	[BUGFIX] clear request #4286 (#4402) ... Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-15 17:43:28 +08:00
ltd0924	d8841b7b40	[BugFix] fix workers=1 (#4364) ... * [Feature] support prefix cache in DP * fix * Update common_engine.py * Update common_engine.py * Update common_engine.py * Update common_engine.py * [BugFix] fix workers more than 1 * fix * Update api_server.py * fix * Update api_server.py * fix --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-15 17:06:25 +08:00
qwes5s5	abb62624b8	[fix] Fixed the issue of excessive/redundant spans being returned for streaming requests. (#4375) ... * fix stream span * fix stream span	2025-10-15 11:47:47 +08:00
ltd0924	28d1b6cd97	[BugFix] fix multinode bugs (#4377) ... * [BugFix] fix multinode bugs * Update test_config.py * Update test_config.py * Update test_config.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-15 11:43:39 +08:00
李泳桦	6265f4385f	[feat] support prefix cache clearing when /clear_load_weight is called (#4008) ... * [feat] support clearing prefix cache (cherry-picked from release/2.1) * [fix] fix ipc suffix, use port instead * [fix] fix prefix caching not enabled * [fix] fix key/value_cache_scales indent * [fix] fix ep group all-reduce * [fix] fix clear/update lock not working when workers > 1 * [chore] add preemption triggered info log * [fix] fix code style * [fix] fix max_num_seqs config * [fix] do not force enable_prefix_caching=False in dynamic loading * [fix] fix ci * Revert "[fix] fix ci" This reverts commit 0bc6d55cc8. * [fix] initialize available_gpu_block_num with max_gpu_block_num * [fix] fix config splitwise_role * [fix] fix clearing caches synchronization and add more logs * [chore] print cache_ready_signal in log * [fix] fix scheduler_config.splitwise_role * [fix] fix cache_messager cache_ready_signal create=True * [fix] stop cache messager from launching in mixed deployment	2025-09-28 19:42:53 +08:00
xiaolei373	55124f8491	Add cli run batch (#4237) ... * feat(log):add_request_and_response_log * [cli] add run batch cli --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-26 14:27:25 +08:00
zhuzixuan	12a3587cca	[Supplements and upgrades]Improvement of X1 parsers (#4172) ... * reasoning_parser * reasoning_parser * reasoning_parser * reasoning_parser * reasoning_parser * reasoning_parser * reasoning_parser	2025-09-26 13:37:37 +08:00
memoryCoderC	4ec00df2b0	[Feature] add config api (#4254)	2025-09-26 11:21:02 +08:00
memoryCoderC	8b0ce8e3ab	[Feature] add cli command serve (#4226)	2025-09-24 14:50:45 +08:00
ltd0924	83720da79f	[Feature] support clear data (#3601) ... * [Feature] support clear data * update * fix * fix * fix * fix * fix * fix * fix	2025-09-23 10:20:02 +08:00
luukunn	ee9d8a840a	[fix]Modify follow-up push parameters and Modify the verification method for thinking length (#4086) ... * 续推参数 generated_token_ids 修改成 completion_token_ids;修改思考长度校验方式 * 续推参数 generated_token_ids 修改成 completion_token_ids;修改思考长度校验方式 * 续推参数 generated_token_ids 修改成 completion_token_ids;修改思考长度校验方式 * 续推参数 generated_token_ids 修改成 completion_token_ids;修改思考长度校验方式 * add completion_token_ids * add logger * fix reasoning_max_tokens ParameterError * add unittest * add unittest * add unittest * add unittest * add unittest * add unit test	2025-09-19 14:26:01 +08:00
xiaolei373	ddf5606263	Bugfix test exception (#4171) ... * feat(log):add_request_and_response_log * modify default error type	2025-09-19 11:48:49 +08:00
xiaolei373	98447beb4d	Add param valid log (#4113) ... * feat(log):add_request_and_response_log * [bugfix] add param valid log --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-18 10:39:24 +08:00
Jiang-Jia-Jun	a04365a0c7	Update api_server.py	2025-09-15 21:31:33 +08:00
xiaolei373	9ac539471d	[format] Valid para format error info (#4035) ... * feat(log):add_request_and_response_log * 报错信息与OpenAI对齐	2025-09-12 19:05:17 +08:00
zhuzixuan	a47976e82d	[Echo] Support more types of prompt echo (#4022) ... * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. * wenxin-tools-700 When the prompt type is list[int] or list[list[int]], it needs to support echoing after decoding. --------- Co-authored-by: luukunn <83932082+luukunn@users.noreply.github.com>	2025-09-11 19:34:44 +08:00
ltd0924	684e93269b	[Fix] fix multi api server log dir (#3967) ... * [BugFix] fix max streaming tokens invalid * fix scheduler bug * fix scheduler bug * Update multi_api_server.py	2025-09-10 17:15:30 +08:00
zhuzixuan	83bd55100b	[Optimize]Error messages about Model api. (#3839) ... * add v1/models interface related * add model parameters * default model verification * unit test * check model err_msg * unit test * type annotation * model parameter in response * modify document description * modify document description * unit test * verification * verification update * model_name * pre-commit * update test case * update test case * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/entrypoints/openai/serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * 优化报错信息。 --------- Co-authored-by: yangzichao01 <yangzichao01@baidu.com> Co-authored-by: Yzc216 <101054010+Yzc216@users.noreply.github.com> Co-authored-by: LiqinruiG <37392159+LiqinruiG@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-09-08 15:52:26 +08:00
ltd0924	0c45e225d3	mv connection_manager init (#3901) ... Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-05 21:11:48 +08:00
ltd0924	bd7d15f7ea	[Feature] support controller port in multi api server (#3898) ... * Update serving_chat.py * Update serving_completion.py * Update serving_completion.py * Update multi_api_server.py	2025-09-05 17:16:31 +08:00
SunLei	29628de6a7	Support for async processor added. (#3869) ... * Support for async processor added. * remove yappi code --------- Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-04 19:58:53 +08:00
xiaolei373	ed97cf8396	Graceful shut down (#3785) ... * feat(log):add_request_and_response_log * 优雅退出-接口增加退出时长参数	2025-09-04 19:33:50 +08:00
RichardWooSJTU	f36a388ffe	fix response processsors (#3826) ... * fix response processsors * fix ci * fix ut --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-04 16:01:25 +08:00
yyssys	abde903813	Automatically configure workers based on max-num-seqs (#3846) ... Automatically configure workers based on max-num-seqs	2025-09-03 21:12:42 +08:00
luukunn	fc598d4c5a	add reasoning parser plugin (#3811) ... * add reasoning parser plugin * fix finish reason	2025-09-03 18:31:27 +08:00
ltd0924	2c9b169c0e	[BugFix] fix scheduler invalid (#3803) ... * [BugFix] fix max streaming tokens invalid * fix scheduler bug * fix scheduler bug	2025-09-02 20:28:51 +08:00
Jiang-Jia-Jun	0e4df5a6f4	[Feature] Setting number of apiserver workers automatically (#3790) ... Co-authored-by: Jiang-Jia-Jun <jiangjiajun@baidu.com>	2025-09-02 14:17:48 +08:00
ltd0924	bf0cf5167a	[BugFix] fix max streaming tokens invalid (#3789)	2025-09-02 13:57:32 +08:00
SunLei	b9af95cf1c	[Feature] Add AsyncTokenizerClient&ChatResponseProcessor with remote encode&decode support. (#3674) ... * [Feature] add AsyncTokenizerClient * add decode_image * Add response_processors with remote decode support. * [Feature] add tokenizer_base_url startup argument * Revert comment removal and restore original content. * [Feature] Non-streaming requests now support remote image decoding. * Fix parameter type issue in decode_image call. * Keep completion_token_ids when return_token_ids = False. * add copyright	2025-08-30 17:06:26 +08:00
luukunn	9a7c231f2c	[Feature]support chat_template.jinja (#3721) ... * add support chat_template.jinja * add support chat_template.jinja	2025-08-30 17:05:34 +08:00
李泳桦	88297240e7	[feat] completion api supports passing input token ids in either prompt or prompt_token_ids (#3311) ... * [feat] completion api supports passing input token ids in either `prompt` or `prompt_token_ids` * [fix] update comment * [fix] fix type error * [test] add a unittest file for serving api test * [test] try to fix ci error * [chore] rename test function names * [test] try to fix ci error * [test] try to fix ci error * [test] add tests for qwen	2025-08-29 14:19:42 +08:00
Yuanle Liu	4957908275	add input_processor plugin (#3657) ... * add input_processor plugin * update * update * update * update * update * update * update * update * update * update * update	2025-08-28 22:53:57 +08:00
ltd0924	e5015eea05	[BugFix] fix logger (#3666)	2025-08-28 17:08:00 +08:00
gaoziyuan	82e64b13e1	[NewFeature]Support dp multi api server && Fix some bug in mixed ep && merge develop (#3598) ... * [Feature] update ep * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix queue ports idx * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * Update engine.py * fix ci * fix some bug in mixed ep * add server fix and op fix * rm some log * fix code style * ltd fix * fix * fix * fix some bug * fix bug * fix bug * fix style * Update config.py * Update splitwise_connector.py * Update cache_messager.py * Update __init__.py * merge and fix * Update engine.py * Update common_engine.py * Update run_ci_xpu.sh * Update ernie_processor.py * Update ernie_processor.py --------- Co-authored-by: ltd0924 <ltd0924@sina.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com>	2025-08-26 19:59:02 +08:00
SunLei	2f28f40d90	fix: replace list * n initialization with list comprehension to avoid shared references (#3618)	2025-08-26 17:53:31 +08:00
Sunny-bot1	c68c3c4b8b	[Feature] bad words support v1 scheduler and specifiy token ids (#3608) ... * support bad_words_token_ids * docs * fix test * fix * bad words support kvcache v1 and token ids * fix	2025-08-25 20:14:51 -07:00
ltd0924	66c5addce4	[Bugfix] fix api server control signal bugs (#3531) ... * Update serving_chat.py * Update serving_completion.py * Update serving_completion.py	2025-08-25 21:13:04 +08:00
chen	9cab3f47ff	[Feature] Add temp_scaled_logprobs and top_p_normalized_logprobs parameters for logits and logprobs post processing (#3552) ... * [feature] Add temp_scaled_logprobs and top_p_normalized_logprobs parameters for logits and logprobs post processing * infer engine support temp_scaled_logprobs and top_p_normalized_logprobs * delete some code * code check * code check and add doc * fix tokenizer.decoder(-1), return 'Invalid Token' * add ci for temp_scaled and top_p logprobs * check test * check seq len time shape * logprob clip inf --------- Co-authored-by: sunlei1024 <sunlei5788@gmail.com>	2025-08-25 14:11:49 +08:00
李泳桦	8bea4b1e25	[fix] fix output tokens count in streaming completion api (#3507)	2025-08-21 18:19:13 +08:00
李泳桦	e4f0b755b4	[fix] setting disable_chat_template while passing prompt_token_ids led to response error (#3228) ... * [fix] setting disable_chat_template while passing prompt_token_ids led to response error * [fix] code syntax * [test] add test case for this bug * [test] add test case for empty message list * [test] fix test case for empty message list	2025-08-21 17:30:51 +08:00