86 Commits

Author	SHA1	Message	Date
李泳桦	0fa28b1068	[fix] fix ep group all-reduce (#4140) ... * [fix] fix ep group all-reduce * [fix] fix clear/update lock not working when workers > 1 * [chore] add preemption triggered info log * [fix] fix code style * fix model_weights_signal (#4092) * fix model_weights_signal --------- Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-18 10:34:49 +08:00
K11OntheBoat	7f9a9b37f3	Support limit thinking lengths (#4070) ... Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-09-17 12:40:08 +08:00
李泳桦	7ccbcc5a62	[feat] support prefix cache clearing when /clear_load_weight is called (#4091) ... * [feat] support clearing prefix cache (cherry-picked from release/2.1) * [fix] fix ipc suffix, use port instead * [fix] fix prefix caching not enabled * [fix] fix code style * [fix] wait for rank0 to update weight status	2025-09-16 11:11:20 +08:00
chenjian	4f8ff478b3	[Feature] Support mixed deployment with yiyan adapter in release22 (#3974) ... * [Feature] Support mixed deployment with yiyan adapter in release2.2 * [Feature] Support mixed deployment with yiyan adapter in release2.2 * fix metrics * add unit test * add unit test * add unit test * add unit test * add unit test * add unit test	2025-09-10 16:01:13 +08:00
ltd0924	a6b161b007	[Fix] fix multi api server log dir (#3966) ... * fix scheduler bug * fix * Update api_server.py * Update multi_api_server.py * [Fix]	2025-09-10 13:48:17 +08:00
zhuzixuan	d43c2f2577	[Optimize]Error messages about Model api. (#3839) (#3972) ... * add v1/models interface related * add model parameters * default model verification * unit test * check model err_msg * unit test * type annotation * model parameter in response * modify document description * modify document description * unit test * verification * verification update * model_name * pre-commit * update test case * update test case * Update tests/entrypoints/openai/test_serving_models.py * Update tests/entrypoints/openai/test_serving_models.py * Update tests/entrypoints/openai/test_serving_models.py * Update tests/entrypoints/openai/test_serving_models.py * Update fastdeploy/entrypoints/openai/serving_models.py * 优化报错信息。 --------- Co-authored-by: yangzichao01 <yangzichao01@baidu.com> Co-authored-by: Yzc216 <101054010+Yzc216@users.noreply.github.com> Co-authored-by: LiqinruiG <37392159+LiqinruiG@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-09-09 10:58:11 +08:00
chenjian	8915c8411d	Revert "[Feature] Setting number of apiserver workers automatically (#3794)" (#3918) ... This reverts commit d1d063e4af.	2025-09-05 21:06:50 +08:00
ltd0924	173e4df982	[Fix] mv connection_manager init (#3902) ... * Update serving_chat.py * Update serving_completion.py * Update serving_completion.py * mv connection_manager init --------- Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-05 17:42:36 +08:00
ltd0924	55ebe855c0	[Feature] support controller port in multi api server (#3895) ... * fix scheduler bug * fix * Update api_server.py * Update multi_api_server.py	2025-09-05 13:38:58 +08:00
luukunn	b8d0f1c081	[bug] fix finish reason (#3858) ... * add reasoning parser plugin * fix finish reason --------- Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-04 14:36:03 +08:00
ltd0924	8550e19008	[bugfix] scheduler (#3871) ... * fix scheduler bug * fix * Update api_server.py	2025-09-04 11:34:12 +08:00
SunLei	8c0e7d6fe9	Support for async processor added. (#3870) ... * Support for async processor added. * remove yappi code	2025-09-04 10:35:08 +08:00
ltd0924	37cb37b7f2	[BugFix] fix scheduler (#3818) ... * fix scheduler bug * fix	2025-09-03 11:16:49 +08:00
ltd0924	cd09384a14	[BugFix] fix max streaming tokens invalid (#3799) ... * Update serving_chat.py * Update serving_completion.py * Update serving_completion.py	2025-09-02 21:03:13 +08:00
Jiang-Jia-Jun	d1d063e4af	[Feature] Setting number of apiserver workers automatically (#3794) ... Co-authored-by: Jiang-Jia-Jun <jiangjiajun@baidu.com>	2025-09-02 17:19:07 +08:00
SunLei	b9af95cf1c	[Feature] Add AsyncTokenizerClient&ChatResponseProcessor with remote encode&decode support. (#3674) ... * [Feature] add AsyncTokenizerClient * add decode_image * Add response_processors with remote decode support. * [Feature] add tokenizer_base_url startup argument * Revert comment removal and restore original content. * [Feature] Non-streaming requests now support remote image decoding. * Fix parameter type issue in decode_image call. * Keep completion_token_ids when return_token_ids = False. * add copyright	2025-08-30 17:06:26 +08:00
luukunn	9a7c231f2c	[Feature]support chat_template.jinja (#3721) ... * add support chat_template.jinja * add support chat_template.jinja	2025-08-30 17:05:34 +08:00
李泳桦	88297240e7	[feat] completion api supports passing input token ids in either prompt or prompt_token_ids (#3311) ... * [feat] completion api supports passing input token ids in either `prompt` or `prompt_token_ids` * [fix] update comment * [fix] fix type error * [test] add a unittest file for serving api test * [test] try to fix ci error * [chore] rename test function names * [test] try to fix ci error * [test] try to fix ci error * [test] add tests for qwen	2025-08-29 14:19:42 +08:00
Yuanle Liu	4957908275	add input_processor plugin (#3657) ... * add input_processor plugin * update * update * update * update * update * update * update * update * update * update * update	2025-08-28 22:53:57 +08:00
ltd0924	e5015eea05	[BugFix] fix logger (#3666)	2025-08-28 17:08:00 +08:00
ltd0924	3d92fb09f7	[BugFix] fix parameter is 0 (#3592) ... * Update engine_client.py * fix * Update common_engine.py	2025-08-28 09:52:36 +08:00
ltd0924	2974016103	[BugFix] fix ce bugs (#3641) ... * [BugFix] fix tp8 client refuse * fix engine port bug * Update utils.py	2025-08-27 20:38:15 +08:00
gaoziyuan	82e64b13e1	[NewFeature]Support dp multi api server && Fix some bug in mixed ep && merge develop (#3598) ... * [Feature] update ep * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix queue ports idx * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * Update engine.py * fix ci * fix some bug in mixed ep * add server fix and op fix * rm some log * fix code style * ltd fix * fix * fix * fix some bug * fix bug * fix bug * fix style * Update config.py * Update splitwise_connector.py * Update cache_messager.py * Update __init__.py * merge and fix * Update engine.py * Update common_engine.py * Update run_ci_xpu.sh * Update ernie_processor.py * Update ernie_processor.py --------- Co-authored-by: ltd0924 <ltd0924@sina.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com>	2025-08-26 19:59:02 +08:00
SunLei	2f28f40d90	fix: replace list * n initialization with list comprehension to avoid shared references (#3618)	2025-08-26 17:53:31 +08:00
Sunny-bot1	c68c3c4b8b	[Feature] bad words support v1 scheduler and specifiy token ids (#3608) ... * support bad_words_token_ids * docs * fix test * fix * bad words support kvcache v1 and token ids * fix	2025-08-25 20:14:51 -07:00
ltd0924	66c5addce4	[Bugfix] fix api server control signal bugs (#3531) ... * Update serving_chat.py * Update serving_completion.py * Update serving_completion.py	2025-08-25 21:13:04 +08:00
chen	9cab3f47ff	[Feature] Add temp_scaled_logprobs and top_p_normalized_logprobs parameters for logits and logprobs post processing (#3552) ... * [feature] Add temp_scaled_logprobs and top_p_normalized_logprobs parameters for logits and logprobs post processing * infer engine support temp_scaled_logprobs and top_p_normalized_logprobs * delete some code * code check * code check and add doc * fix tokenizer.decoder(-1), return 'Invalid Token' * add ci for temp_scaled and top_p logprobs * check test * check seq len time shape * logprob clip inf --------- Co-authored-by: sunlei1024 <sunlei5788@gmail.com>	2025-08-25 14:11:49 +08:00
YuanRisheng	c389a4013c	Unify server-side and model-side Config(Part-5) (#3497) ... * move config * fix xpu * fix * fix vl * fix vl * fix unitest * fix args * add unitest * fix test	2025-08-21 19:00:21 +08:00
李泳桦	8bea4b1e25	[fix] fix output tokens count in streaming completion api (#3507)	2025-08-21 18:19:13 +08:00
李泳桦	e4f0b755b4	[fix] setting disable_chat_template while passing prompt_token_ids led to response error (#3228) ... * [fix] setting disable_chat_template while passing prompt_token_ids led to response error * [fix] code syntax * [test] add test case for this bug * [test] add test case for empty message list * [test] fix test case for empty message list	2025-08-21 17:30:51 +08:00
luukunn	371fb3f853	[Feature] add tool parser (#3483) ... * add tool parser * add x1 enable_thinking * restart ci * fix vl reasoning parser * modify call style * modify call style * add offline enablethinking * fix completion * fix * fix unit test * fix unit test * fix unit test * fix vl reasoning parser * fix vl reasoning parser	2025-08-21 17:25:44 +08:00
Yzc216	466cbb5a99	[Feature] Models api (#3073) ... * add v1/models interface related * add model parameters * default model verification * unit test * check model err_msg * unit test * type annotation * model parameter in response * modify document description * modify document description * unit test * verification * verification update * model_name * pre-commit * update test case * update test case * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/entrypoints/openai/serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: LiqinruiG <37392159+LiqinruiG@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-08-21 17:02:56 +08:00
ltd0924	51f68ae593	[Feature] add dealer manager to reuse the connection (#3471) ... * [BugFix] fix control signal release failed * [BugFix] fix control signal release failed * update * update * update * [Feature] add dealer manager to reuse the connection * fix * fix * fix * fix * fix * fix * Create test_dealer_connection_manager.py * Delete test/entrypoints/openai directory * Update test_dealer_connection_manager.py * Update test_dealer_connection_manager.py	2025-08-21 13:11:13 +08:00
memoryCoderC	31f639f10b	[Feature] add prompt_tokens and completion_tokens (#3504)	2025-08-21 10:23:27 +08:00
kevin	67298cf4c0	add error traceback info (#3419) ... * add error traceback info * update error msg * update code --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-19 19:32:04 +08:00
ltd0924	bca8905b40	[BugFix] fix control signal release failed (#3390) ... * [BugFix] fix control signal release failed * [BugFix] fix control signal release failed * update * update * update	2025-08-19 13:51:38 +08:00
luukunn	3a7a20d191	[Feature] Pass through the chat_template_kwargs to the data processing module (#3421) ... * fix chat_template_args * fix args * add offline * add offline * fix * fix * fix default enable_thinking value * fix default enable_thinking value * modify condition * Revert "modify condition" This reverts commit 26430bdeb1. * fix unit test	2025-08-19 10:50:01 +08:00
zhuzixuan	c95b3395e9	【BugFix】completion接口echo回显支持 (#3245) ... * wenxin-tools-511,修复v1/completion无法回显的问题。 * 支持多prompt的回显 * 支持多prompt情况下的流式回显 * 补充了 completion 接口支持 echo 的单元测试 * pre-commit * 移除了多余的test文件 * 修复了completion接口echo支持的单测方法 * 补充了单元测试文件 * 补充单测 * unittest * 补充单测 * 修复单测 * 删除不必要的assert. * 重新提交 * 更新测试方法 * ut * 验证是否是正确思路单测 * 验证是否是正确思路单测 * 验证是否是正确思路单测3 * 优化单测代码，有针对性地缩小单测范围。 * 优化单测代码2，有针对性地缩小单测范围。 * 优化单测代码3，有针对性地缩小单测范围。 * support 'echo' in chat/completion. * update * update * update * update * update * update * 补充了关于tokenid的单元测试 * update * 修正index错误 * 修正index错误	2025-08-19 10:41:51 +08:00
luukunn	9c129813f9	[Feature] add custom chat template (#3251) ... * add custom chat_template * add custom chat_template * add unittest * fix * add docs * fix comment * add offline chat * fix unit test * fix unit test * fix * fix pre commit * fix unit test * add unit test * add unit test * add unit test * fix pre_commit * fix enable_thinking * fix pre commit * fix pre commit * fix unit test * add requirements	2025-08-18 16:34:08 +08:00
gaoziyuan	6fdd83da10	fix some bug (#3434)	2025-08-18 14:39:13 +08:00
xiaolei373	d4f610e4cd	feat(log):add_request_and_response_log (#3373)	2025-08-13 23:27:41 +08:00
luukunn	eda83ca672	add Tool Parser (#3272) ... * add tool-parser * add tool-parser * add tool parser * add tool parser * fix * add offline * add offline * fix * parsers:tool&reasoning * 修改tool parser名称· * update * fix reasoning-parser * add requirements * fix finish reason * fix * fix reasoning-parser * fix * fix * fix * fix * fix --------- Co-authored-by: zhuzixuan <zhuzixuan@baidu.com>	2025-08-13 01:06:55 +08:00
memoryCoderC	2d1a4cacdf	Completion add raw_prediction/text_after_process (#3356)	2025-08-12 23:06:45 +08:00
memoryCoderC	c575611a5b	[BugFix] v1/completions add finish_reason (#3246) ... * [BugFix] v1/completions add finish_reason * update TestOpenAIServingCompletion for merge --------- Co-authored-by: YUNSHEN XIE <1084314248@qq.com>	2025-08-12 19:40:26 +08:00
ltd0924	31d4fcb425	[BugFix] fix too many open files problem (#3256) ... * Update cache_messager.py * fix too many open files problem * fix too many open files problem * fix too many open files problem * fix ci bugs * Update api_server.py * add parameter * format * format * format * format * Update parameters.md * Update parameters.md * Update serving_completion.py * Update serving_chat.py * Update envs.py --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-08 20:10:11 +08:00
JYChen	9423c577fe	[stop_seq] fix out-bound value for stop sequence (#3216) ... * fix out-bound value for stop sequence * catch error if there are out-of-bounds value * check in offline mode * add ut tests	2025-08-07 15:40:21 +08:00
李泳桦	09cc4e2802	[fix] fix completion stream api output_tokens not in usage (#3247)	2025-08-07 10:36:00 +08:00
sg263	841e831575	[Trace]add trace when fd start (#3174) ... * add opentelemetry * add opentelemetry * add opentelemetry on dequeue * add opentelemetry on dequeue * add opentelemetry on dequeue * fix annotation * fix annotation when add opentelemetry * fix opentelemetry-instrumentation-fastapi * fix pentelemetry-bootstrap * fix opentelemetry can not work in uvicorn * move conf to env * fd start add trace * fix pre-commit * fix pre-commit * change FD_JOB_ID --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: shige <shige@baidu.com>	2025-08-05 21:18:27 +08:00
lizhenyun01	fe540f6caa	[plugin] Custom model_runner/model support (#3186) ... * support custom model&&model_runner * fix merge * add test && update doc * fix codestyle * fix unittest * load model in rl	2025-08-04 18:52:39 -07:00
SunLei	68bc1d12c0	[Bugfix] Fix uninitialized decoded_token and add corresponding unit test. (#3195)	2025-08-04 19:23:58 +08:00