317 Commits

Author	SHA1	Message	Date
K11OntheBoat	64d1aa973b	[Unitest]Add unitest of Attention Layer (#4494)	2025-10-27 19:18:50 +08:00
kevin	8aab4e367f	[Feature] mm support prefix cache (#4134) ... * support mm prefix caching * update code * fix mm_hashes * support encoder cache * add encoder cache * update code * update encoder cache * fix features bug * fix worker bug * support processor cache, need to optimize yet * refactor multimodal data cache * update code * update code * update v1 scheduler * update code * update code * update codestyle * support turn off processor cache and encoder cache * update pre-commit * fix code * solve review * update code * update code * update test case * set processor cache in GiB * update test case * support mm prefix caching for qwen model * fix code style check * update pre-commit * fix unit test * fix unit test * add ci test case * fix rescheduled bug * change text_after_process to prompt_tokens * fix unit test * fix chat template * change model path * [EP] fix adapter bugs (#4572) * Update expert_service.py * Update common_engine.py * Update expert_service.py * fix v1 hang bug (#4573) * fix import image_ops error on some platforms (#4559) * [CLI]Update parameters in bench latecy cli tool and fix collect-env cli tool (#4558) * add collect-env * del files * [Graph Optimization] Add dy_runnable and introduce cudagraph_switch_threshold for cudagraph mode switching (#4578) * add new branch for sot * reorder * fix batch bug * [XPU]Moe uses a new operator (#4585) * [XPU]Moe uses a new operator * [XPU]Moe uses a new operator * update response * [Feature] Support Paddle-OCR (#4396) * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> * [DataProcessor] add reasoning_tokens into usage info (#4520) * add reasoning_tokens into usage info initial commit * add unit tests * modify unit test * modify and add unit tests * fix unit test * move steam usage to processor * modify processor * modify test_logprobs * modify test_logprobs.py * modify stream reasoning tokens accumulation * fix unit test * perf: Optimize task queue communication from engine to worker (#4531) * perf: Optimize task queue communication from engine to worker * perf: get_tasks to numpy * perf: get_tasks remove to_numpy * fix: request & replace ENV * remove test_e2w_perf.py * fix code style --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * Clean up ports after processing results (#4587) * [CI] Add /re-run command in PR comments to restart failed CI workflows (#4593) * [Others] api server exits when worker process is dead (#3271) * [fix] fix terminal hangs when worker process is dead * [chore] change sleep time of monitor * [chore] remove redundant comments * update docs --------- Co-authored-by: ApplEOFDiscord <wwy640130@163.com> Co-authored-by: ApplEOFDiscord <31272106+ApplEOFDiscord@users.noreply.github.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com> Co-authored-by: yinwei <yinwei_hust@163.com> Co-authored-by: JYChen <zoooo0820@qq.com> Co-authored-by: qwes5s5 <45442318+qwes5s5@users.noreply.github.com> Co-authored-by: Ryan <zihaohuang@aliyun.com> Co-authored-by: yyssys <atyangshuang@foxmail.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com> Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> Co-authored-by: kxz2002 <115912648+kxz2002@users.noreply.github.com> Co-authored-by: SunLei <sunlei5788@gmail.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: Zhang Yulong <35552275+ZhangYulongg@users.noreply.github.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: 李泳桦 <39643373+liyonghua0910@users.noreply.github.com>	2025-10-27 17:39:51 +08:00
chen	5c63a089f6	[Feature] Support logprobs_mode (#4567)	2025-10-27 14:27:48 +08:00
CSWYF3634076	acd331780c	[V1 loader] Qwen25 VL support v1 loader and torch style safetensors load (#4388) ... * [BugFix] qwen2.5vl enable_thinking=true and image_patch_id bug fix * [Docs]offine infer add apply_chat_template add_generation_prompt parameter * [Model]qwen2.5VL support --use-cudagraph * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v2 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v3 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v4 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v5 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v6 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v7 * qwen25vl v1 loader * qwen25vl v1 loader v2 * qwen25vl v1 loader v3 * qwen25vl v1 loader fix tp2 weight PySafeSlice * qwen25vl v1 loader no test * qwen25vl v1 loader add unit test * qwen25vl v1 loader add unit test v2 * qwen25vl v1 loader add torch unit test v3 * qwen25vl v1 loader add torch unit test v4 * qwen25vl v1 loader add torch unit test v5 * qwen25vl v1 loader add torch unit test v6	2025-10-27 10:54:15 +08:00
Zhang Yulong	83b720804b	Clean up ports after processing results (#4587)	2025-10-27 10:13:24 +08:00
SunLei	dc1a9c7287	perf: Optimize task queue communication from engine to worker (#4531) ... * perf: Optimize task queue communication from engine to worker * perf: get_tasks to numpy * perf: get_tasks remove to_numpy * fix: request & replace ENV * remove test_e2w_perf.py * fix code style --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-10-25 22:45:38 +08:00
kxz2002	327fa4c255	[DataProcessor] add reasoning_tokens into usage info (#4520) ... * add reasoning_tokens into usage info initial commit * add unit tests * modify unit test * modify and add unit tests * fix unit test * move steam usage to processor * modify processor * modify test_logprobs * modify test_logprobs.py * modify stream reasoning tokens accumulation * fix unit test	2025-10-25 16:57:58 +08:00
yyssys	822dea8d5f	[XPU]Moe uses a new operator (#4585) ... * [XPU]Moe uses a new operator * [XPU]Moe uses a new operator * update response	2025-10-24 23:01:46 +08:00
Ryan	f42ed6d5f2	[Graph Optimization] Add dy_runnable and introduce cudagraph_switch_threshold for cudagraph mode switching (#4578) ... * add new branch for sot * reorder * fix batch bug	2025-10-24 18:36:52 +08:00
qwes5s5	e02a812880	[CLI]Update parameters in bench latecy cli tool and fix collect-env cli tool (#4558) ... * add collect-env * del files	2025-10-24 16:46:45 +08:00
Sunny-bot1	4ffe41a747	WINT4/WINT8 dense gemm default use Machete (#4451)	2025-10-23 17:57:59 +08:00
luukunn	bbf06b9ff7	[BugFix]Fix finish reason (#4543) ... * fix finish reason * add unit test * add unit test * fix unie test * fix unit test	2025-10-23 14:04:43 +08:00
RAM	8a02ab43a8	[FDConfig]Turn on the CUDAGraph + RL switch (#4508) ... * Turn on the CUDAGraph + RL switch * reduce max_num_seqs and number of request	2025-10-23 11:08:07 +08:00
周周周	d7bcedf421	small change in test_fusedmoe.py (#4538)	2025-10-22 17:49:18 +08:00
Yuanle Liu	8e02a509c3	[CI] stable test_rollout_model.py (#4536) ... * stable test_rollout_model.py * update baseline * update baseline	2025-10-22 01:59:44 -07:00
zhouchong	dce988824d	[Feature] Support AsyncLLM (#4458) ... * add async_llm * apply review * update engine config * Adapt to latest engine.py changes * add more unit tests * Increase unit test coverage --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-10-22 15:50:12 +08:00
yzwu	dc7facaa7f	[Iluvatar GPU] fix ci error caused by rebuild_padding param and cuda graph (#4504)	2025-10-21 21:41:41 +08:00
SunLei	809c1ac7ec	feat: add post-processing step for pool_output (#4462) ... * feat: add post-processing step for pool_output * bugfix * fix: test_serving_embedding * fix test_request_to_batch_dicts * fix: code style	2025-10-21 20:24:26 +08:00
plusNew001	2bd3fb6315	[XPU]add xpu ci ep case (#4432) ... * add xpu ci case * Add xDeepEP download and build steps Download and build xDeepEP before running tests. * Fix formatting and add missing sleep command * Update Docker image version in CI workflow * Modify run_ci_xpu.sh for log cleanup and error handling Clean up log files before running tests and output worker log on failure. * Enhance test_ep.py with process management and assertions Refactor test function to include process cleanup and assertions. * Replace test_fastdeploy_llm with test_fd_ep * Fix conditional statement in run_ci_xpu.sh * Update test_ep.py for string handling and formatting Fix string encoding issues and improve readability. * Rename test_ep.py to run_ep.py * Change test script from test_ep.py to run_ep.py	2025-10-21 19:19:40 +08:00
Copilot	175391389f	Add comprehensive unit tests for limit_thinking_content_length operators (#4510) ... * Initial plan * Add comprehensive unit tests for limit_thinking_content_length functions Co-authored-by: yuanlehome <23653004+yuanlehome@users.noreply.github.com> * fix (#4514) --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: yuanlehome <23653004+yuanlehome@users.noreply.github.com> Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-10-21 18:55:57 +08:00
ltd0924	fb76cdfb4f	[Fearture] Support mm model close prefix cache (#4459) ... * [Feature] support prefix cache in DP * fix * Update common_engine.py * Update common_engine.py * Update common_engine.py * Update common_engine.py * [BugFix] fix workers more than 1 * fix * Update api_server.py * fix * Update api_server.py * fix * [Fearture] Support mm model close prefix cache * Update api_server.py * Update engine_client.py * Update engine_client.py * add test * Update test_chat.py * fix * fix * Update test_chat.py * Update test_chat.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-10-21 15:37:59 +08:00
Divano	2b53c4d684	【CI】Add test cases for n parameter and streaming validation (#4503) ... * add repitation early stop cases * add repitation early stop cases * add structure test openai * add n parameters case	2025-10-21 15:33:29 +08:00
SunLei	ee915220bd	[Speculative Decoding] Add draft_logprobs Support for Speculative Decode MTP (#4467) ... * feat: add draft_logprobs for Speculative Decode MTP * feat: add draft_logprobs for Speculative Decode MTP * feat: add draft_logprobs for Speculative Decode MTP * fix: postprocess for speculative decode * test: test_speculative_decoding_use_logprobs * fix: test_completion_echo * fix test_max_streaming_tokens --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-10-21 14:57:50 +08:00
RAM	775edcc09a	[Executor] Default use CUDAGraph (#3594) ... * add start intercept * Adjustment GraphOptConfig * pre-commit * default use cudagraph * set default value * default use cuda graph * pre-commit * fix test case bug * disable rl * fix moba attention * only support gpu * Temporarily disable PD Disaggregation * set max_num_seqs of test case as 1 * set max_num_seqs and temperature * fix max_num_batched_tokens bug * close cuda graph * success run wint2 * profile run with max_num_batched_tokens * 1.add c++ memchecker 2.success run wint2 * updatee a800 yaml * update docs * 1. delete check 2. fix plas attn test case * default use use_unique_memory_pool * add try-except for warmup * ban mtp, mm, rl * fix test case mock * fix ci bug * fix form_model_get_output_topp0 bug * fix ci bug * refine deepseek ci * refine code * Disable PD * fix sot yaml	2025-10-21 14:25:45 +08:00
YuBaoku	70a29ec49e	[CI] update ernie-4_5-vl baseline (#4495) ... * [CI] update ernie-4_5-vl baseline * [CI] update Qwen2.5-VL-7B-Instruct baseline	2025-10-21 10:18:29 +08:00
Yuanle Liu	cef3164c3b	Optimizing the performance of think length limit using custom operators (#4279) ... * delete impl * delete min_length&max_length * support limit thinking content strategy * fix * fix * fix * update * fix set_value_by_flags_and_idx * fix * fix * fix * fix * update * fix * fix * fix typo * fix ci * fix * fix * support mtp * fix * fix * update * update	2025-10-20 21:09:13 +08:00
Ryan	36af88ff3f	[BugFix][CI] Clean up SOT code cache using tearDown in CINN unitest (#4491) ... * fix CINN BUG * 1e-3 -> 1e-2	2025-10-20 20:45:00 +08:00
GoldPancake	47595a2480	[Feature] support mtp logprob (#4464) ... * support mtp logprob * fix unitest	2025-10-20 15:18:12 +08:00
Haonan Luo	1b9f351d21	Support GPT-OSS-BF16 (#4240) ... * [Feature] AppendAtten support sinks & HEAD_DIM=64 * fix bug * fix bug * fix bug * fix bug * [Feature] support gpt-oss * fix bug * add mask * support-gpt-oss * support-gpt-oss * fix long seq * support wint8 * support wint8 * support wint8 * update test * change sliding windows init pos --------- Co-authored-by: ming1753 <ideaminghp@163.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com>	2025-10-20 14:44:58 +08:00
zhuzixuan	1e59905e34	Optimization of ‘tools’ in request fields (#4380) ... * Remove multiple 'tools' * Remove multiple 'tools' * Remove multiple 'tools' * Remove multiple 'tools'	2025-10-20 11:04:08 +08:00
YuBaoku	c4fc0073cf	[CI] Handle unit test issues (#4483)	2025-10-20 10:13:21 +08:00
周周周	817210e47f	[ATTN]delete code and add ffn and moe layer level test (#4440) ... * delete code * delete code * delete code * commit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit	2025-10-19 16:23:11 +08:00
kxz2002	b5b993e48e	【feature】support n parameter (#4273) ... * support n parameter * pre-commit check * pre-commit check * restore format_and_add_data * update n_param * bug fix index - str to int * bug fix del child_task * bug fix metrics * add debug info * add debug info2 * remove debug info * change connecting symbol to '-' * bugfix change connecting symbol * bugfix change connecting symbol2 * unit tests fix * unit test fix2 * unittest add param n=2 * n param add unit tests and adapt to echo * pre-commit fix * resolve review * adjust stop reason * add unittest for _create_chat_completion_choice * modify unittest * solve confict * solve conflict * resolve conflict --------- Co-authored-by: LiqinruiG <37392159+LiqinruiG@users.noreply.github.com> Co-authored-by: gaoziyuan <m13689897706@163.com>	2025-10-17 20:51:59 +08:00
kxz2002	8ccfd975b5	LLM.chat add "tools" param (#4415) ... * llm add tools param initial commit * llm add tools param bugfix * offline add tools add unittests * fix preprocessor * move tools paramter into tasks * change variable name	2025-10-17 20:25:03 +08:00
yinwei	a64c0408b9	[XPU]Fix w4a8 precision bug && rollback moe algo (#4463) ... * fix w4a8 precision bug * add env * code stype check	2025-10-17 18:27:53 +08:00
yzwu	4b661512ca	[Iluvatar GPU] Adapt VL model (#4313)	2025-10-17 16:13:38 +08:00
YuBaoku	01510876ab	[CI] Fix partial instability issues (#4461)	2025-10-17 14:17:06 +08:00
chen	b134e6afe6	[BugFix]Dev fix custom ar unstable result (#4437)	2025-10-17 11:47:16 +08:00
Ryan	6160145f82	[SOT] Change warnings to errors and remove fallback operations (#4378) ... * Change warnings to errors and remove fallback operations * fix unitest * fix codestyle	2025-10-17 11:27:04 +08:00
Ryan	49cea8fb1c	[SOT][Cudagraph] Remove BreakGraph of #3302 && update CustomOp (#3694) ... * rm inplace info && to(gpu) * update append_attention * unpin paddle version * add full_cuda_graph=False * add blank line --------- Co-authored-by: SigureMo <sigure.qaq@gmail.com>	2025-10-17 10:57:55 +08:00
chen	db82e9a022	[BugFix]Fix wfp8afp8 triton moe group_topk renormalized=True (#4449) ... * fix group_topk renormalized=True * check test	2025-10-16 23:17:48 +08:00
xiaolei373	dbca63f862	[bugfix] kill cache_transfer_manager process (#4401)	2025-10-16 20:45:24 +08:00
LiqinruiG	4251ac5e95	【Fix】 remove text_after_process & raw_prediction (#4421) ... * remove text_after_process & raw_prediction * remove text_after_process & raw_prediction	2025-10-16 19:00:18 +08:00
SunLei	b4b579a7ed	Feature：Add support for Pooling Model Embedding and provide an OpenAI-compatible API. (#4344) ... * feat: add OpenAIServing * feat: add ZmqOpenAIServing & OpenAIServingEmbedding * feat: Refine the basic ServingEngine class and introduce ServingContext * fix: codestyle * fix: request * fix: pooling_params * feat: _process_chat_template_kwargs * feat: support batch request * feat: pooling_params verify & default parameters --------- Co-authored-by: sunlei1024 <sunlei1024@example.com>	2025-10-15 19:42:59 +08:00
freeliuzc	744287e1a9	fix param (#4419)	2025-10-15 18:44:24 +08:00
AIbin	6938df9c23	【Fix CI Bug】Fix ci bug (#4413) ... * Support DSK-v3.2 model * Support DSK-v3.2 * Support DSK-v3.2 * Support DSK-3.2 * fix CI bug * fix_CI_BUG * update ci bug	2025-10-15 14:19:04 +08:00
freeliuzc	582aebd48b	[MTP]support mtp chunk_prefill_v1 (#4366) ... * support mtp chunk_prefill_v1 * fix mtp chunkprefill output, fix unit test * fix unit test * fix save_output	2025-10-15 13:21:32 +08:00
qwes5s5	abb62624b8	[fix] Fixed the issue of excessive/redundant spans being returned for streaming requests. (#4375) ... * fix stream span * fix stream span	2025-10-15 11:47:47 +08:00
ltd0924	28d1b6cd97	[BugFix] fix multinode bugs (#4377) ... * [BugFix] fix multinode bugs * Update test_config.py * Update test_config.py * Update test_config.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-10-15 11:43:39 +08:00
YuBaoku	4b647d17de	[CI] Fix partial instability issues (#4418)	2025-10-15 11:13:43 +08:00