67 Commits

Author	SHA1	Message	Date
copilot-swe-agent[bot]	5badb9df39	Use consistent signal name resolution across all files ... - Use signal.Signals(signum).name consistently in both api_server.py files - Add fallback for unknown signals with Signal(n) format - This ensures accurate signal name reporting in logs Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-23 12:16:43 +00:00
copilot-swe-agent[bot]	9e4eb339b8	Address code review feedback ... - Translate Chinese comments to English for consistency - Add subprocess import to api_server.py for TimeoutExpired handling - Improve signal name detection in worker_process.py using signal.Signals - Add better docstring comments for signal handlers and cleanup functions Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-23 12:15:24 +00:00
copilot-swe-agent[bot]	cd62cc2df9	Improve error handling in cleanup functions ... - Add check for already terminated processes before attempting cleanup - Add ProcessLookupError handling for race conditions - Add success logging for better debugging - Check if cache manager processes are alive before terminating Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-23 12:13:11 +00:00
copilot-swe-agent[bot]	e09d6363d3	Add signal handlers for graceful process termination ... - Added SIGINT/SIGTERM signal handlers in api_server.py (both OpenAI and simple versions) - Added cleanup_processes() function to properly terminate worker processes - Enhanced StandaloneApplication with worker exit hooks and cleanup - Added signal handling in worker_process.py for graceful worker shutdown - Added shutdown_event to coordinate graceful shutdown across threads - Improved worker monitor to respect shutdown event Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-23 12:10:20 +00:00
Yonghua Li	0c8c6369ed	[Feature] [PD Disaggregation] simplify configuration for pd-disaggregated deployment, and refactor post-init and usage for all ports (#5415) ... * [feat] simplify configuration for pd-disaggregated deployment, and refactor post-init and usage for all ports * [fix] fix some bugs * [fix] fix rdma port for cache manager/messager * [fix] temporarily cancel port availability check to see if it can pass ci test * [feat] simplify args for multi api server * [fix] fix dp * [fix] fix port for xpu * [fix] add tests for ports post processing & fix ci * [test] fix test_multi_api_server * [fix] fix rdma_comm_ports args for multi_api_server * [fix] fix test_common_engine * [fix] fix test_cache_transfer_manager * [chore] automatically setting FD_ENABLE_MULTI_API_SERVER * [fix] avoid api server from creating engine_args twice * [fix] fix test_run_batch * [fix] fix test_metrics * [fix] fix splitwise connector init * [test] add test_rdma_transfer and test_expert_service * [fix] fix code syntax * [fix] fix test_rdma_transfer and build wheel with rdma script	2025-12-17 15:50:42 +08:00
xiaolei373	a30b4da260	[Feature] Tracing: Fine-Grained Tracing for Request Latency Part1 (#5458)	2025-12-16 16:36:09 +08:00
xiaolei373	a4bb3e9960	[bugfix]remove metrics middleware (#5332)	2025-12-03 17:07:45 +08:00
qwes5s5	117980dd4e	[LogProbs]Enable prompt logprobs output and modify data transmission method for the online interface. (#5089) ... * add prompt logprobs * Merge prompt_logprobs_tensors and prompt_logprobs * fix param check * trigger ci * fix unitest * fix logprobs bug	2025-12-02 13:49:51 +08:00
Yonghua Li	a535050b11	[FDConfig] remove engine client args, use fd_config instead (#5217) ... * [refactor] remove engine client args, use fd_config instead * [chore] update * [fix] fix * [fix] fix * [chore] rename config to fd_config * [fix] fix run_batch * [ci] add ci case for engine client --------- Co-authored-by: Jiaxin Sui <95567040+plusNew001@users.noreply.github.com>	2025-11-28 01:20:54 -08:00
fl0w2o48	e63d715fc3	[BugFix][Metrics] Fix Prometheus Multiprocess Metrics Issues and Add ZMQ Communication Metrics (#5185) ... * [Feature] add metrics for ZMQ and fix multiprocess metrics * fix test_metrics.py --------- Co-authored-by: Jiaxin Sui <95567040+plusNew001@users.noreply.github.com>	2025-11-27 15:05:09 +08:00
Yonghua Li	09379183e2	[BugFix] fix work metrics not returned by metrics api (#4912) ... * [BugFix] fix work metrics not returned by metrics api * [fix] fix conflict * [fix] fix ci	2025-11-25 19:12:29 +08:00
kevin	8e4e3ff510	[Feature] support eplb in api_server (#4782) ... * support eplb in api_server * update code * add eplb test case * update eplb * support tp+dp eplb * update test cese * update code * update code * fix bug * update copilot review * update test case name	2025-11-24 20:22:29 +08:00
zhouchong	5444af6ff6	[APIServer] metrics use port the same as api_port (#5016) ... * metrics use port the same as api_port * Be tolerant to tests that monkeypatch/partially mock args. * Reduce code redundancy --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-11-17 10:42:45 +08:00
kxz2002	87911b7cf1	[Feature] Enable FastDeploy to support adding the “--api-key” authentication parameter. (#4806) ... * add api key initial commit * add unit test * modify unit test * move middleware to a single file and add unit tests	2025-11-08 18:24:02 +08:00
xiaolei373	14e7d88ea4	[feature] support reward api (#4518) ... Co-authored-by: SunLei <sunlei5788@gmail.com>	2025-10-29 00:20:28 +08:00
Daci	6426414a0f	[Feature] EngineWorkerQueue anonymous port (#4597) ... * EngineWorkerQueue 支持匿名端口设置 * EngineWorkerQueue 支持匿名端口设置 * EngineWorkerQueue 支持匿名端口设置 * EngineWorkerQueue 支持匿名端口设置 * EngineWorkerQueue 支持匿名端口设置	2025-10-28 10:22:37 +08:00
kevin	8aab4e367f	[Feature] mm support prefix cache (#4134) ... * support mm prefix caching * update code * fix mm_hashes * support encoder cache * add encoder cache * update code * update encoder cache * fix features bug * fix worker bug * support processor cache, need to optimize yet * refactor multimodal data cache * update code * update code * update v1 scheduler * update code * update code * update codestyle * support turn off processor cache and encoder cache * update pre-commit * fix code * solve review * update code * update code * update test case * set processor cache in GiB * update test case * support mm prefix caching for qwen model * fix code style check * update pre-commit * fix unit test * fix unit test * add ci test case * fix rescheduled bug * change text_after_process to prompt_tokens * fix unit test * fix chat template * change model path * [EP] fix adapter bugs (#4572) * Update expert_service.py * Update common_engine.py * Update expert_service.py * fix v1 hang bug (#4573) * fix import image_ops error on some platforms (#4559) * [CLI]Update parameters in bench latecy cli tool and fix collect-env cli tool (#4558) * add collect-env * del files * [Graph Optimization] Add dy_runnable and introduce cudagraph_switch_threshold for cudagraph mode switching (#4578) * add new branch for sot * reorder * fix batch bug * [XPU]Moe uses a new operator (#4585) * [XPU]Moe uses a new operator * [XPU]Moe uses a new operator * update response * [Feature] Support Paddle-OCR (#4396) * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> * [DataProcessor] add reasoning_tokens into usage info (#4520) * add reasoning_tokens into usage info initial commit * add unit tests * modify unit test * modify and add unit tests * fix unit test * move steam usage to processor * modify processor * modify test_logprobs * modify test_logprobs.py * modify stream reasoning tokens accumulation * fix unit test * perf: Optimize task queue communication from engine to worker (#4531) * perf: Optimize task queue communication from engine to worker * perf: get_tasks to numpy * perf: get_tasks remove to_numpy * fix: request & replace ENV * remove test_e2w_perf.py * fix code style --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * Clean up ports after processing results (#4587) * [CI] Add /re-run command in PR comments to restart failed CI workflows (#4593) * [Others] api server exits when worker process is dead (#3271) * [fix] fix terminal hangs when worker process is dead * [chore] change sleep time of monitor * [chore] remove redundant comments * update docs --------- Co-authored-by: ApplEOFDiscord <wwy640130@163.com> Co-authored-by: ApplEOFDiscord <31272106+ApplEOFDiscord@users.noreply.github.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com> Co-authored-by: yinwei <yinwei_hust@163.com> Co-authored-by: JYChen <zoooo0820@qq.com> Co-authored-by: qwes5s5 <45442318+qwes5s5@users.noreply.github.com> Co-authored-by: Ryan <zihaohuang@aliyun.com> Co-authored-by: yyssys <atyangshuang@foxmail.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com> Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> Co-authored-by: kxz2002 <115912648+kxz2002@users.noreply.github.com> Co-authored-by: SunLei <sunlei5788@gmail.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: Zhang Yulong <35552275+ZhangYulongg@users.noreply.github.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: 李泳桦 <39643373+liyonghua0910@users.noreply.github.com>	2025-10-27 17:39:51 +08:00
李泳桦	cdc40cdc2a	[Others] api server exits when worker process is dead (#3271) ... * [fix] fix terminal hangs when worker process is dead * [chore] change sleep time of monitor * [chore] remove redundant comments	2025-10-27 10:23:48 +08:00
ming1753	e4e3cede7f	[Feature] Support Paddle-OCR (#4396) ... * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com>	2025-10-24 23:34:30 +08:00
ltd0924	a498736af5	[APIServer] support define gunicorn timeout (#4496) ... * [BUGFIX] clear request #4286 * [BugFix] support define gunicorn timeout * Update utils.py * Update utils.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-20 23:36:07 +08:00
Yuanle Liu	cef3164c3b	Optimizing the performance of think length limit using custom operators (#4279) ... * delete impl * delete min_length&max_length * support limit thinking content strategy * fix * fix * fix * update * fix set_value_by_flags_and_idx * fix * fix * fix * fix * update * fix * fix * fix typo * fix ci * fix * fix * support mtp * fix * fix * update * update	2025-10-20 21:09:13 +08:00
SunLei	b4b579a7ed	Feature：Add support for Pooling Model Embedding and provide an OpenAI-compatible API. (#4344) ... * feat: add OpenAIServing * feat: add ZmqOpenAIServing & OpenAIServingEmbedding * feat: Refine the basic ServingEngine class and introduce ServingContext * fix: codestyle * fix: request * fix: pooling_params * feat: _process_chat_template_kwargs * feat: support batch request * feat: pooling_params verify & default parameters --------- Co-authored-by: sunlei1024 <sunlei1024@example.com>	2025-10-15 19:42:59 +08:00
ltd0924	fbdb056de0	[BUGFIX] clear request #4286 (#4402) ... Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-15 17:43:28 +08:00
ltd0924	d8841b7b40	[BugFix] fix workers=1 (#4364) ... * [Feature] support prefix cache in DP * fix * Update common_engine.py * Update common_engine.py * Update common_engine.py * Update common_engine.py * [BugFix] fix workers more than 1 * fix * Update api_server.py * fix * Update api_server.py * fix --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-15 17:06:25 +08:00
qwes5s5	abb62624b8	[fix] Fixed the issue of excessive/redundant spans being returned for streaming requests. (#4375) ... * fix stream span * fix stream span	2025-10-15 11:47:47 +08:00
李泳桦	6265f4385f	[feat] support prefix cache clearing when /clear_load_weight is called (#4008) ... * [feat] support clearing prefix cache (cherry-picked from release/2.1) * [fix] fix ipc suffix, use port instead * [fix] fix prefix caching not enabled * [fix] fix key/value_cache_scales indent * [fix] fix ep group all-reduce * [fix] fix clear/update lock not working when workers > 1 * [chore] add preemption triggered info log * [fix] fix code style * [fix] fix max_num_seqs config * [fix] do not force enable_prefix_caching=False in dynamic loading * [fix] fix ci * Revert "[fix] fix ci" This reverts commit 0bc6d55cc8. * [fix] initialize available_gpu_block_num with max_gpu_block_num * [fix] fix config splitwise_role * [fix] fix clearing caches synchronization and add more logs * [chore] print cache_ready_signal in log * [fix] fix scheduler_config.splitwise_role * [fix] fix cache_messager cache_ready_signal create=True * [fix] stop cache messager from launching in mixed deployment	2025-09-28 19:42:53 +08:00
memoryCoderC	4ec00df2b0	[Feature] add config api (#4254)	2025-09-26 11:21:02 +08:00
memoryCoderC	8b0ce8e3ab	[Feature] add cli command serve (#4226)	2025-09-24 14:50:45 +08:00
ltd0924	83720da79f	[Feature] support clear data (#3601) ... * [Feature] support clear data * update * fix * fix * fix * fix * fix * fix * fix	2025-09-23 10:20:02 +08:00
Jiang-Jia-Jun	a04365a0c7	Update api_server.py	2025-09-15 21:31:33 +08:00
xiaolei373	9ac539471d	[format] Valid para format error info (#4035) ... * feat(log):add_request_and_response_log * 报错信息与OpenAI对齐	2025-09-12 19:05:17 +08:00
ltd0924	0c45e225d3	mv connection_manager init (#3901) ... Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-05 21:11:48 +08:00
xiaolei373	ed97cf8396	Graceful shut down (#3785) ... * feat(log):add_request_and_response_log * 优雅退出-接口增加退出时长参数	2025-09-04 19:33:50 +08:00
yyssys	abde903813	Automatically configure workers based on max-num-seqs (#3846) ... Automatically configure workers based on max-num-seqs	2025-09-03 21:12:42 +08:00
ltd0924	2c9b169c0e	[BugFix] fix scheduler invalid (#3803) ... * [BugFix] fix max streaming tokens invalid * fix scheduler bug * fix scheduler bug	2025-09-02 20:28:51 +08:00
Jiang-Jia-Jun	0e4df5a6f4	[Feature] Setting number of apiserver workers automatically (#3790) ... Co-authored-by: Jiang-Jia-Jun <jiangjiajun@baidu.com>	2025-09-02 14:17:48 +08:00
SunLei	b9af95cf1c	[Feature] Add AsyncTokenizerClient&ChatResponseProcessor with remote encode&decode support. (#3674) ... * [Feature] add AsyncTokenizerClient * add decode_image * Add response_processors with remote decode support. * [Feature] add tokenizer_base_url startup argument * Revert comment removal and restore original content. * [Feature] Non-streaming requests now support remote image decoding. * Fix parameter type issue in decode_image call. * Keep completion_token_ids when return_token_ids = False. * add copyright	2025-08-30 17:06:26 +08:00
luukunn	9a7c231f2c	[Feature]support chat_template.jinja (#3721) ... * add support chat_template.jinja * add support chat_template.jinja	2025-08-30 17:05:34 +08:00
Yuanle Liu	4957908275	add input_processor plugin (#3657) ... * add input_processor plugin * update * update * update * update * update * update * update * update * update * update * update	2025-08-28 22:53:57 +08:00
ltd0924	e5015eea05	[BugFix] fix logger (#3666)	2025-08-28 17:08:00 +08:00
gaoziyuan	82e64b13e1	[NewFeature]Support dp multi api server && Fix some bug in mixed ep && merge develop (#3598) ... * [Feature] update ep * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix queue ports idx * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * Update engine.py * fix ci * fix some bug in mixed ep * add server fix and op fix * rm some log * fix code style * ltd fix * fix * fix * fix some bug * fix bug * fix bug * fix style * Update config.py * Update splitwise_connector.py * Update cache_messager.py * Update __init__.py * merge and fix * Update engine.py * Update common_engine.py * Update run_ci_xpu.sh * Update ernie_processor.py * Update ernie_processor.py --------- Co-authored-by: ltd0924 <ltd0924@sina.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com>	2025-08-26 19:59:02 +08:00
Yzc216	466cbb5a99	[Feature] Models api (#3073) ... * add v1/models interface related * add model parameters * default model verification * unit test * check model err_msg * unit test * type annotation * model parameter in response * modify document description * modify document description * unit test * verification * verification update * model_name * pre-commit * update test case * update test case * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/entrypoints/openai/test_serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/entrypoints/openai/serving_models.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: LiqinruiG <37392159+LiqinruiG@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-08-21 17:02:56 +08:00
ltd0924	51f68ae593	[Feature] add dealer manager to reuse the connection (#3471) ... * [BugFix] fix control signal release failed * [BugFix] fix control signal release failed * update * update * update * [Feature] add dealer manager to reuse the connection * fix * fix * fix * fix * fix * fix * Create test_dealer_connection_manager.py * Delete test/entrypoints/openai directory * Update test_dealer_connection_manager.py * Update test_dealer_connection_manager.py	2025-08-21 13:11:13 +08:00
kevin	67298cf4c0	add error traceback info (#3419) ... * add error traceback info * update error msg * update code --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-19 19:32:04 +08:00
ltd0924	bca8905b40	[BugFix] fix control signal release failed (#3390) ... * [BugFix] fix control signal release failed * [BugFix] fix control signal release failed * update * update * update	2025-08-19 13:51:38 +08:00
luukunn	9c129813f9	[Feature] add custom chat template (#3251) ... * add custom chat_template * add custom chat_template * add unittest * fix * add docs * fix comment * add offline chat * fix unit test * fix unit test * fix * fix pre commit * fix unit test * add unit test * add unit test * add unit test * fix pre_commit * fix enable_thinking * fix pre commit * fix pre commit * fix unit test * add requirements	2025-08-18 16:34:08 +08:00
gaoziyuan	6fdd83da10	fix some bug (#3434)	2025-08-18 14:39:13 +08:00
xiaolei373	d4f610e4cd	feat(log):add_request_and_response_log (#3373)	2025-08-13 23:27:41 +08:00
luukunn	eda83ca672	add Tool Parser (#3272) ... * add tool-parser * add tool-parser * add tool parser * add tool parser * fix * add offline * add offline * fix * parsers:tool&reasoning * 修改tool parser名称· * update * fix reasoning-parser * add requirements * fix finish reason * fix * fix reasoning-parser * fix * fix * fix * fix * fix --------- Co-authored-by: zhuzixuan <zhuzixuan@baidu.com>	2025-08-13 01:06:55 +08:00
ltd0924	31d4fcb425	[BugFix] fix too many open files problem (#3256) ... * Update cache_messager.py * fix too many open files problem * fix too many open files problem * fix too many open files problem * fix ci bugs * Update api_server.py * add parameter * format * format * format * format * Update parameters.md * Update parameters.md * Update serving_completion.py * Update serving_chat.py * Update envs.py --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-08 20:10:11 +08:00