152 Commits

Author	SHA1	Message	Date
Ryan	9a647cb61c	[OP] Add InferShape&InferDtype for per_token_quant_padding (#4667) (#4683) ... * add InferShape&InferDtype for per_token_quant_padding * fix codestyle	2025-10-31 16:42:52 +08:00
Ryan	07956a87b3	[Graph Optimization] Fix IR graph dependency error exposed after enabling SOT by updating the return value of TextImageGatherScatter (#4610) ... * fix TextImageGatherScatter in sot * fix codestyle	2025-10-28 18:31:23 +08:00
RAM	86d5006a57	[Graph Optimization][Speculative Decoding] Update yaml and fix typo (#4612)	2025-10-28 11:43:26 +08:00
Lucas	5c6105f4a2	[XPU] bind some OPs for VL model with pybind (#4522)	2025-10-27 10:50:08 +08:00
xiaozude	f7069b8057	[Metax] adapt DeepSeek (#4498)	2025-10-24 10:14:53 +08:00
Sunny-bot1	8718fa34b2	support static C8 (#4568)	2025-10-23 22:01:03 +08:00
RAM	9dc5c3e370	[Graph Optimization] Support CUDAGraph Padding + MTP (#4545) ... * Support CUDAGraph Padding + MTP * support orther write cache kernel	2025-10-23 20:57:26 +08:00
Sunny-bot1	4ffe41a747	WINT4/WINT8 dense gemm default use Machete (#4451)	2025-10-23 17:57:59 +08:00
Yuanle Liu	3b58310c26	enhance set_stop_value_multi_ends and standardize the registration of some operators (#4525) ... * fix custom_ops * paddleformers>=0.3.1	2025-10-21 22:06:06 +08:00
RAM	775edcc09a	[Executor] Default use CUDAGraph (#3594) ... * add start intercept * Adjustment GraphOptConfig * pre-commit * default use cudagraph * set default value * default use cuda graph * pre-commit * fix test case bug * disable rl * fix moba attention * only support gpu * Temporarily disable PD Disaggregation * set max_num_seqs of test case as 1 * set max_num_seqs and temperature * fix max_num_batched_tokens bug * close cuda graph * success run wint2 * profile run with max_num_batched_tokens * 1.add c++ memchecker 2.success run wint2 * updatee a800 yaml * update docs * 1. delete check 2. fix plas attn test case * default use use_unique_memory_pool * add try-except for warmup * ban mtp, mm, rl * fix test case mock * fix ci bug * fix form_model_get_output_topp0 bug * fix ci bug * refine deepseek ci * refine code * Disable PD * fix sot yaml	2025-10-21 14:25:45 +08:00
Yuanle Liu	cef3164c3b	Optimizing the performance of think length limit using custom operators (#4279) ... * delete impl * delete min_length&max_length * support limit thinking content strategy * fix * fix * fix * update * fix set_value_by_flags_and_idx * fix * fix * fix * fix * update * fix * fix * fix typo * fix ci * fix * fix * support mtp * fix * fix * update * update	2025-10-20 21:09:13 +08:00
GoldPancake	47595a2480	[Feature] support mtp logprob (#4464) ... * support mtp logprob * fix unitest	2025-10-20 15:18:12 +08:00
Haonan Luo	1b9f351d21	Support GPT-OSS-BF16 (#4240) ... * [Feature] AppendAtten support sinks & HEAD_DIM=64 * fix bug * fix bug * fix bug * fix bug * [Feature] support gpt-oss * fix bug * add mask * support-gpt-oss * support-gpt-oss * fix long seq * support wint8 * support wint8 * support wint8 * update test * change sliding windows init pos --------- Co-authored-by: ming1753 <ideaminghp@163.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com>	2025-10-20 14:44:58 +08:00
周周周	817210e47f	[ATTN]delete code and add ffn and moe layer level test (#4440) ... * delete code * delete code * delete code * commit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit	2025-10-19 16:23:11 +08:00
yzwu	4b661512ca	[Iluvatar GPU] Adapt VL model (#4313)	2025-10-17 16:13:38 +08:00
Sunny-bot1	930f7b781c	[Optimization] Put get_block_shape_and_split_kv_block in cuda graph for append attention backend (#4443) ... * get block in cuda graph * fix sot	2025-10-17 10:59:56 +08:00
Ryan	49cea8fb1c	[SOT][Cudagraph] Remove BreakGraph of #3302 && update CustomOp (#3694) ... * rm inplace info && to(gpu) * update append_attention * unpin paddle version * add full_cuda_graph=False * add blank line --------- Co-authored-by: SigureMo <sigure.qaq@gmail.com>	2025-10-17 10:57:55 +08:00
Zhenghai Zhang	6adfbe07ad	【Hackathon 9th No.86】autogen MultiQueryDecoderAttention template_instantiation -part (#4383) ... * split MultiQueryDecoderAttention template_instantiation * update comment * CI	2025-10-16 17:08:19 +08:00
freeliuzc	582aebd48b	[MTP]support mtp chunk_prefill_v1 (#4366) ... * support mtp chunk_prefill_v1 * fix mtp chunkprefill output, fix unit test * fix unit test * fix save_output	2025-10-15 13:21:32 +08:00
Sunny-bot1	6d0cc0dd9c	[Optimization] Optimize split_q_block kernel (#4367)	2025-10-15 11:28:00 +08:00
Sunny-bot1	a751d977bc	[Optimization] Fuse get_max_len and get_kv_max_len (#4369) ... * opt split_q_block * fuse max_lens and max kv_len	2025-10-13 20:35:00 +08:00
freeliuzc	365601ea5a	[MTP]support more branchs in topp kernel (#4352)	2025-10-11 11:33:52 +08:00
AIbin	f7eaca3971	【Bug Fix】mla enables tensorcore by default (#4354) ... * mla tensor-core kernel is enabled by default	2025-10-10 20:45:16 +08:00
lzy	3aa04fbf21	[MTP][Cfp8]supports spec dynamic cfp8 (#4290) ... * supports spec dynamic cfp8 * supports spec dynamic cfp8 --------- Co-authored-by: freeliuzc <lzc842650834@gmail.com>	2025-10-10 16:08:10 +08:00
Zhenghai Zhang	c46d5e48f8	【Hackathon 9th No.86】autogen MultiQueryAppendC8Attention template_instantiation -part (#4330) ... * split MultiQueryAppendC8Attention template_instantiation * update setup_ops.py * fix ci * fix bug	2025-10-10 15:07:48 +08:00
AIbin	c4ebaf8a07	【Inference Optimize】MLA Tensor-Core is enabled by default (#4335)	2025-10-10 10:54:56 +08:00
Nyakku Shigure	5f80862578	Remove redundant inplace outputs for append_attention (#4340)	2025-10-10 10:21:27 +08:00
RAM	aa27b03bc0	[Executor]CUDAGraph support Speculate Decode (#3769) ... * success run ngram * Revert "[Code Simplification] remove cum_offsets (#3410)" This reverts commit 32b39620bc. * success run ngram5 tp4 42bs * success run ngram5 tp4 42bs * mtp draft commit * add decorator for target model * enable draft model in cudagraph v0.5 * revert revrt cum_offset * enable target model in cudagraph v0.9 And clean debug code * Revert "success run ngram" This reverts commit 8351e83993. * add reverted code * enable target model in cudagraph v0.9 * solve comment * fix bid < 0 * Enable Target Model Padding And Draft Model in cudagraph * solve problem * delete rebuild padding debug note * fast compile * Add capture list for mtp * success run 256 tp1 mtp * Enable Lite TP2 Bsz256 * realy enable tp2 bsz 256 * fix problem * Solve problem for Draft model in cudagraph * Solve comment * replace emptytensor as zeros * Solve comments * Revert "fast compile" This reverts commit 834639a7ff. * fix bug * fix merge bug * fix typo * fix bug --------- Co-authored-by: lizexu <2694294196@qq.com> Co-authored-by: littledgg <1658565283@qq.com> Co-authored-by: zeroRains <linjunlu@zerorains.top> Co-authored-by: gongshaotian <gstain5555@outlook.com>	2025-10-09 21:18:29 +08:00
fangfangssj	af3872215e	[BugFix]remove redundant includes (#4312) ... Co-authored-by: Tao Luo <luotao02@baidu.com>	2025-09-30 17:54:19 +08:00
chen	81959c7d88	[NewFeature]custom_allreduce support cudagraph recapture (#4305) ... * custom_allreduce support cudagraph recapture * add shut_down/restart default group	2025-09-29 15:56:54 +08:00
xiaozude	7c919070f7	[Metax] support cutlass moe & optimize flash attention (#4208)	2025-09-29 11:22:43 +08:00
李泳桦	6265f4385f	[feat] support prefix cache clearing when /clear_load_weight is called (#4008) ... * [feat] support clearing prefix cache (cherry-picked from release/2.1) * [fix] fix ipc suffix, use port instead * [fix] fix prefix caching not enabled * [fix] fix key/value_cache_scales indent * [fix] fix ep group all-reduce * [fix] fix clear/update lock not working when workers > 1 * [chore] add preemption triggered info log * [fix] fix code style * [fix] fix max_num_seqs config * [fix] do not force enable_prefix_caching=False in dynamic loading * [fix] fix ci * Revert "[fix] fix ci" This reverts commit 0bc6d55cc8. * [fix] initialize available_gpu_block_num with max_gpu_block_num * [fix] fix config splitwise_role * [fix] fix clearing caches synchronization and add more logs * [chore] print cache_ready_signal in log * [fix] fix scheduler_config.splitwise_role * [fix] fix cache_messager cache_ready_signal create=True * [fix] stop cache messager from launching in mixed deployment	2025-09-28 19:42:53 +08:00
Zhenghai Zhang	aa045aa84f	fix typos (#4274)	2025-09-27 09:25:43 +08:00
chen	7c1fd19f0f	[OPs] MoE support wfp8afp8(channelwise) and improve per_token_quant_fp8 (#4238)	2025-09-24 16:39:51 +08:00
Yohanna	44010cee13	FIX] Fix CUDA error(700): 'cudaErrorIllegalAddress' in CascadeAppendWriteCacheKVQKV cache_kernel(). Continue when batch_id_per_token[token_idx] is default value -1. (#4218)	2025-09-24 14:08:49 +08:00
chen	1a6283424e	Fix noaux_tc cuda Error 700 in CUDAGraph (#4174)	2025-09-23 18:41:33 +08:00
yzwu	504461b6b5	[Iluvatar GPU] Optimize attention performance and fix moe load ckpt error (#3651)	2025-09-22 21:13:59 +08:00
chenjian	918ccdb123	[Feature] Support pd ep deployment with yiyan adapter (#4029) ... * [Feature] Support mixed deployment with yiyan adapter in release2.2 * fix metrics * add unit test * add unit test * add unit test * Support pd ep deployment with yiyan adapter * Support pd ep deployment with yiyan adapter * refactor cache messager * support scheduler v1 in PD * suppport pd v1 + chunk prefill * suppport pd v1 + chunk prefill * add eplb * support eplb * support eplb * support eplb * support v1 * fix * fix * fix bug * remove eplb support * support prefix cache in P * fix bug * fix bug * support one stop in V1 * fix bug * fix ci * fix ci * fix * fix * fix * fix * fix --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-22 16:41:38 +08:00
co63oc	c4830ef24c	fix typos (#4176) ... * fix typos * fix	2025-09-22 14:27:17 +08:00
chen	66a98b44ed	ep support logprob (#4089) (#4151)	2025-09-19 14:07:31 +08:00
gaoziyuan	896e3bb606	[NewFeture]add ep rollout model init and update/clear ep buffer (#4039) ... * fix gid * merge * fix test * fix bug * fix * fix ci	2025-09-17 20:24:53 +08:00
Yuan Xiaolan	de8638b1e9	fix dynamic Cfp8 computing error (#4119) ... Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-16 20:21:49 +08:00
Sunny-bot1	442543cd6b	fix ep wint8 (#4102)	2025-09-16 11:05:33 +08:00
Sunny-bot1	b1a5b756a3	[Optimize] Support WINT8 and group scale for Machete (#3905)	2025-09-15 12:01:34 +08:00
AIbin	a7392a0ff9	【Inference Optimize】DeepSeek-V3-model MLA Optimize (#3886) ... * support MLA chunk_size auto search & cuda_graph	2025-09-11 10:46:09 +08:00
chen	637d96c6ae	[Feature] Support zai-org/GLM-4.5-Air BF16 model (#3928) ... * support glm45_air	2025-09-10 19:36:10 +08:00
freeliuzc	7ee100903f	support rope_3d in spec mode (#4034)	2025-09-10 03:15:05 -07:00
Ayakouji	453487d5b0	[Feat] ernie4_5_vl_moe support CudaGraph (#3226) ... * delete dynamic control flow for decode * coda-style * fix scatter/gather typos and use input stream instead default stream * support 0-Size Tensor * update runner and model * using static mem address as input * fix mem leak * refine code * update mm_buffer * fix typo * fix buffersize * fix unk token * refine code * refine * support other arch * open cudagraph in vlci * fix * update * update * update * fix cmd * update --------- Co-authored-by: aquagull <hongyuh@qq.com> Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-10 13:11:57 +08:00
周周周	dbab579299	clean code (#4020)	2025-09-10 10:56:15 +08:00
lzy	f12159b630	del batch id per token (#3963) ... * Update decoder_write_cache_with_rope_kernel.cu del batch_id_per_token * Update decoder_write_cache_with_rope_impl.cuh * Update test_append_attention.py * Update test_append_attention.py	2025-09-08 21:58:34 +08:00