50 Commits

Author	SHA1	Message	Date
kevin	2d69d91ab8	add aksk check (#5273)	2025-11-28 14:28:24 +08:00
Daci	7dc06cac6e	[BugFix] race condition [is_fetching] causing multiple fetch requests (#5238) ... * RouterArgs port str -> int * fix race condition [is_fetching] causing multiple fetch requests * bugfix: Delete duplicate input_ids tensor creation	2025-11-28 13:41:36 +08:00
Yonghua Li	cead6b26fa	[Metrics] Update time_to_first_token to include tokenization & queue time, and remove redundant metrics (#4993) ... * [update] update time_to_first_tokens to include queue time, and remove first_token_latency and infer_latency * [doc] update docs * [ci] fix test * [chore] delete redundant code --------- Co-authored-by: Jiaxin Sui <95567040+plusNew001@users.noreply.github.com>	2025-11-26 14:42:17 +08:00
Juncai	af03da5127	[BugFix] fix release block ids (#5184) ... * fix release block ids * up	2025-11-24 16:48:09 +08:00
Juncai	f9b0545a7f	[PD Disaggregation] [Refine] Refine splitwise deployment (#5151) ... * Refine splitwise deployment * up	2025-11-21 15:30:24 +08:00
kevin	7454480e07	[Feature] support bos download retry (#5137) ... * support bos download retry * update code * update code	2025-11-21 10:18:32 +08:00
Yonghua Li	43097a512a	[BugFix] [PD Disaggregation] fix v1 scheduler prefill node profile run & ipc transfer protocol (#5132) ... * [fix] fix v1 scheduler profile run for append attention in prefill node * [fix] skip send_signal if kv signal not inited for gpu and xpu * [fix] extend fix to flash_attn & mla_attn * [fix] fix v1 pd run in ipc transfer protocol * [ci] add test for v1 pd profile run using ipc transfer protocol * [style] fix code style check * [style] fix code style again * [fix] fix profile run * [update] remove --num-gpu-blocks-override in example script * [chore] rename forward_meta is_profiling to is_dummy_or_profile_run	2025-11-20 21:39:22 +08:00
Juncai	01c30f6b87	Fix schedule error in splitwise deployment (#5149)	2025-11-20 21:18:10 +08:00
yangjianfengo1	af715db763	[Scheduler] Support chunk prefill for video input (#5107) ... * add video chunk prefill * add vit_merge=True for test_tokenizer_client.py --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-11-20 16:29:13 +08:00
kevin	109d48e456	[Feature] support async download features (#5003) ... * support async download features * add test case * update code	2025-11-19 22:23:36 +08:00
Yonghua Li	6c5ab727c1	[BugFix] fix num_requests_running after clear_data (#4927) ... * [BugFix] fix num_requests_running after clear_data * [fix] fix tasks_list and stop flags not cleared when _free_blocks failed	2025-11-13 13:50:21 +08:00
kevin	cc34487810	[Feature] support mm disable_chunked (#4803) ... * support mm disable_chunked * update code * update code * update code	2025-11-06 21:32:25 +08:00
kevin	64e875b460	[Scheduler] update v1 prefill batch (#4611) ... * update v1 prefill batch * update code * update code	2025-10-31 14:03:01 +08:00
李泳桦	a012e3608b	[Feature] support logits processors (#4515) ... * [feat] provide an interface for logits processors and a builtin LogitBiasLogitsProcessor * [chore] fix code style * [fix] add unit test & fix existing bugs * [feat] add engine/worker arg --logits-processors * [fix] redefine user args as logits_processors_args and fix some bugs * [fix] fix test_sampler * Update fastdeploy/model_executor/logits_processor/builtin.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/model_executor/logits_processor/__init__.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/model_executor/test_logits_processor.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [fix] fix typo * Update fastdeploy/engine/sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [fix] fix bracelet * [chore] redefine logits processor interface: pass the entire share_inputs into LP, do not copy share_inputs and logits * [doc] add docs * [fix] fix logit bias processor not applied when decoding is too fast & add docs and tests * [fix] fix redundant code * [feat] skip apply() if no bias is specified --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-10-29 00:08:53 +08:00
ming1753	7681375a19	[BugFix] PaddleOCR-VL fix FD_DEBUG type and support v1 loader (#4605) ... * [Bug Fix] PaddleOCRVL fix FD_DEBUG type and support HF model * fix bug * fix bug * fix bug	2025-10-28 09:47:47 +08:00
kevin	8aab4e367f	[Feature] mm support prefix cache (#4134) ... * support mm prefix caching * update code * fix mm_hashes * support encoder cache * add encoder cache * update code * update encoder cache * fix features bug * fix worker bug * support processor cache, need to optimize yet * refactor multimodal data cache * update code * update code * update v1 scheduler * update code * update code * update codestyle * support turn off processor cache and encoder cache * update pre-commit * fix code * solve review * update code * update code * update test case * set processor cache in GiB * update test case * support mm prefix caching for qwen model * fix code style check * update pre-commit * fix unit test * fix unit test * add ci test case * fix rescheduled bug * change text_after_process to prompt_tokens * fix unit test * fix chat template * change model path * [EP] fix adapter bugs (#4572) * Update expert_service.py * Update common_engine.py * Update expert_service.py * fix v1 hang bug (#4573) * fix import image_ops error on some platforms (#4559) * [CLI]Update parameters in bench latecy cli tool and fix collect-env cli tool (#4558) * add collect-env * del files * [Graph Optimization] Add dy_runnable and introduce cudagraph_switch_threshold for cudagraph mode switching (#4578) * add new branch for sot * reorder * fix batch bug * [XPU]Moe uses a new operator (#4585) * [XPU]Moe uses a new operator * [XPU]Moe uses a new operator * update response * [Feature] Support Paddle-OCR (#4396) * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> * [DataProcessor] add reasoning_tokens into usage info (#4520) * add reasoning_tokens into usage info initial commit * add unit tests * modify unit test * modify and add unit tests * fix unit test * move steam usage to processor * modify processor * modify test_logprobs * modify test_logprobs.py * modify stream reasoning tokens accumulation * fix unit test * perf: Optimize task queue communication from engine to worker (#4531) * perf: Optimize task queue communication from engine to worker * perf: get_tasks to numpy * perf: get_tasks remove to_numpy * fix: request & replace ENV * remove test_e2w_perf.py * fix code style --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * Clean up ports after processing results (#4587) * [CI] Add /re-run command in PR comments to restart failed CI workflows (#4593) * [Others] api server exits when worker process is dead (#3271) * [fix] fix terminal hangs when worker process is dead * [chore] change sleep time of monitor * [chore] remove redundant comments * update docs --------- Co-authored-by: ApplEOFDiscord <wwy640130@163.com> Co-authored-by: ApplEOFDiscord <31272106+ApplEOFDiscord@users.noreply.github.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com> Co-authored-by: yinwei <yinwei_hust@163.com> Co-authored-by: JYChen <zoooo0820@qq.com> Co-authored-by: qwes5s5 <45442318+qwes5s5@users.noreply.github.com> Co-authored-by: Ryan <zihaohuang@aliyun.com> Co-authored-by: yyssys <atyangshuang@foxmail.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com> Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> Co-authored-by: kxz2002 <115912648+kxz2002@users.noreply.github.com> Co-authored-by: SunLei <sunlei5788@gmail.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: Zhang Yulong <35552275+ZhangYulongg@users.noreply.github.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: 李泳桦 <39643373+liyonghua0910@users.noreply.github.com>	2025-10-27 17:39:51 +08:00
ming1753	e4e3cede7f	[Feature] Support Paddle-OCR (#4396) ... * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com>	2025-10-24 23:34:30 +08:00
chenjian	670aaa3f83	[Bug fix] Fix pd for x1 thinking (#4433)	2025-10-16 12:03:45 +08:00
李泳桦	ffe7af8a97	[fix] fix requests & block metrics (#4404) ... * [fix] fix requests & block metrics * [chore] rename variables	2025-10-15 11:49:24 +08:00
AIbin	48fd5d757d	Support MLA_CACHE & Fix V1_Schedule Bug (#4318) ... Support MLA_CACHE & Fix V1_Schedule Bug	2025-10-09 12:11:25 +08:00
RichardWooSJTU	3740e33fea	【Feature】ResourceManagerV1 support need block num notifying (#4220) ... * support need block num notifying * adapt t2i * fix unexpected change	2025-09-29 11:11:51 +08:00
李泳桦	6265f4385f	[feat] support prefix cache clearing when /clear_load_weight is called (#4008) ... * [feat] support clearing prefix cache (cherry-picked from release/2.1) * [fix] fix ipc suffix, use port instead * [fix] fix prefix caching not enabled * [fix] fix key/value_cache_scales indent * [fix] fix ep group all-reduce * [fix] fix clear/update lock not working when workers > 1 * [chore] add preemption triggered info log * [fix] fix code style * [fix] fix max_num_seqs config * [fix] do not force enable_prefix_caching=False in dynamic loading * [fix] fix ci * Revert "[fix] fix ci" This reverts commit 0bc6d55cc8. * [fix] initialize available_gpu_block_num with max_gpu_block_num * [fix] fix config splitwise_role * [fix] fix clearing caches synchronization and add more logs * [chore] print cache_ready_signal in log * [fix] fix scheduler_config.splitwise_role * [fix] fix cache_messager cache_ready_signal create=True * [fix] stop cache messager from launching in mixed deployment	2025-09-28 19:42:53 +08:00
chenjian	3cef851468	[Bug fix] Fix bug for running ep (#4245) ... * fix bug for ep * fix bug	2025-09-28 14:56:18 +08:00
Lucas	87179cb744	[XPU] support XPU VL model inference (#4030) ... * [XPU] support XPU VL model inference * fix image op import and device check * rebase develop * fix perf	2025-09-25 14:34:15 +08:00
ltd0924	83720da79f	[Feature] support clear data (#3601) ... * [Feature] support clear data * update * fix * fix * fix * fix * fix * fix * fix	2025-09-23 10:20:02 +08:00
chenjian	918ccdb123	[Feature] Support pd ep deployment with yiyan adapter (#4029) ... * [Feature] Support mixed deployment with yiyan adapter in release2.2 * fix metrics * add unit test * add unit test * add unit test * Support pd ep deployment with yiyan adapter * Support pd ep deployment with yiyan adapter * refactor cache messager * support scheduler v1 in PD * suppport pd v1 + chunk prefill * suppport pd v1 + chunk prefill * add eplb * support eplb * support eplb * support eplb * support v1 * fix * fix * fix bug * remove eplb support * support prefix cache in P * fix bug * fix bug * support one stop in V1 * fix bug * fix ci * fix ci * fix * fix * fix * fix * fix --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-22 16:41:38 +08:00
YuanRisheng	2e9e53ff7e	[FDConfig]Remove max_num_batched_tokens/max_num_seqs in parallel config (#4116) ... * remove max_num_batched_tokens in parallel config * remove max_num_seqs * update test case * fix test * fix --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-17 10:43:35 +08:00
co63oc	17a27170bc	fix typos (#4093)	2025-09-15 18:33:30 +08:00
chenjian	37f1632732	[Optimize] optimize prefix cache in develop (#3890) ... * optimize prefix cache in release22 * fix * fix * fix * add ci for v1 * add unit test --------- Co-authored-by: xiegegege <46314656+xiegegege@users.noreply.github.com>	2025-09-12 10:15:59 +08:00
yinwei	7833f2f6cb	[XPU]Fixed the issue of performance degradation caused by enabling ENABLE_V1_KVCACHE_SCHEDULER (#3897) ... * fix bug * fix bug * update * update * update	2025-09-08 10:34:46 +08:00
freeliuzc	88d44a2c93	support mtp in v1_scheduler mode (#3695)	2025-09-04 17:39:59 +08:00
chenjian	22c165d6dd	[Feature] Set v1 scheduler as default in develop (#3807) ... * Set scheduler v1 as default * Set scheduler v1 as default * Set scheduler v1 as default * Set scheduler v1 as default * Set scheduler v1 as default * close V1 in guided_decoding * fix vl ci * close V1 in guided_decoding	2025-09-04 15:16:56 +08:00
RichardWooSJTU	0989788b29	support extend block tables (#3824)	2025-09-04 14:39:04 +08:00
Yuanle Liu	306c024ff3	[BugFix] fix error of import paddle.base.core.Config (#3761) ... * 延迟 import Config * support chunked_prefill * support chunked_prefill	2025-09-02 17:23:27 +08:00
co63oc	d6369b4d51	fix typos (#3684)	2025-09-01 17:50:17 +08:00
chenjian	465065cd19	[Bug fix] Fix prefix cache in V1 (#3715) ... * [Bug fix] Fix prefix cache in V1 * fix code style	2025-08-31 21:29:33 +08:00
李泳桦	98e03fb4ea	[feat] add metrics for yiyan adapter (#3219) (#3614) ... * [feat] add metrics for yiyan adapter * [fix] fix metrics num_requests_waiting and num_requests_running * [fix] fix metrics gpu_cache_usage_perc * [refactor] change where requests_number increases * [chore] rename xxx_block_num as xxx_gpu_block_num, and update their values accordingly * [chore] delete useless code	2025-08-30 23:20:58 +08:00
Yuanle Liu	68f87240da	fix key error in mm (#3702)	2025-08-29 14:35:12 +08:00
Yuanle Liu	2fb2c0f46a	fix MultimodalRegistry (#3699)	2025-08-29 11:01:30 +08:00
Yuanle Liu	4957908275	add input_processor plugin (#3657) ... * add input_processor plugin * update * update * update * update * update * update * update * update * update * update * update	2025-08-28 22:53:57 +08:00
YuanRisheng	5b66462f0e	Fix fdconfig bugs (#3528) ... * fix config * fix parallel * fix ips * fix rl * open code	2025-08-22 16:17:15 +08:00
kevin	67298cf4c0	add error traceback info (#3419) ... * add error traceback info * update error msg * update code --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-08-19 19:32:04 +08:00
ming1753	396dba0d62	[Bug Fix] Fix V1 video bug (#3388)	2025-08-13 23:04:07 +08:00
ming1753	f5164215be	[Bug Fix] fix vl V1 schedule bug (#3323) ... * [Bug Fix] fix vl V1 schedule bug * fix format	2025-08-12 11:31:39 +08:00
Zero Rains	b23af29d0b	Launch expert_service before kv_cache initialization in worker_process (#3045) ... * launch expert_service before kv_cache initialization * add two signal make sure model loading and expert_service lauching finished * fix the EP bug * fix ep * update launching way * fix ep * update * roback ep * pre-commit all files --------- Co-authored-by: RAM <gstian5555@outlook.com> Co-authored-by: Divano <dddivano@outlook.com>	2025-08-11 19:38:46 +08:00
chenjian	c011cb8b16	[Bug Fix] Fix scheduler bug in develop (#3292) ... * Fix scheduler bug in develop * Fix scheduler bug in develop * Fix scheduler bug in develop	2025-08-10 13:55:38 +08:00
kevin	22cab724e8	[Feature] block scheduler v1 support prefix caching (#3061) ... * block scheduler v1 support prefix cache * update code * update code * fix code bug * add timeout time --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-07-31 19:29:19 +08:00
chenjian	fe0e3f508b	[BUG FIX] Fix bug when preempted request rescheduled (#3080) ... * Fix bug when preempted request rescheduled * Fix bug when preempted request rescheduled * Fix bug when preempted request rescheduled	2025-07-30 22:25:47 +08:00
ming1753	5acde4eb43	[Feature] Multimodal Scheduler V1 (#3019) ... * [Feature] Support multimodal scheduler v1 * remove debug log * fix bug * fix format * modify code * fix bug * fix bug * fix bug * modify code	2025-07-30 16:05:55 +08:00
chenjian	85a78d695d	[Feature] Support block scheduler v1 for FD (#2928) ... * Support FD block scheduler v1 * Support FD block scheduler v1 * Support FD block scheduler v1 * Fix according to copilot review * Fix according to review * Remove is_dummy * Fix bug when real_bsz=1 * Fix infer first token cost time --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-07-23 20:31:31 +08:00