62 Commits

Author	SHA1	Message	Date
chen	d58c1db8a0	[Feature][OP] Append Attn Support CUDA-PDL (#5072)	2025-11-17 20:47:33 +08:00
周周周	b23e684b67	revert group size 3 (#5079)	2025-11-17 18:54:13 +08:00
Sunny-bot1	8a4ddb29df	Revert "[BugFix] Revert skip capture (#5023)" (#5080)	2025-11-17 16:14:55 +08:00
Sunny-bot1	249feca65a	[BugFix] Revert skip capture (#5023) ... * Revert "[BugFix][Metax] Fix metax compile issue in get_block_shape_and_split_kv_block (#5000)" This reverts commit 05da8e34c0. * Revert "skip DtoH capture (#4988)" This reverts commit 5b24013d46.	2025-11-13 23:52:51 -08:00
周周周	c0a4393d72	[ATTENTION] unitest (#4962)	2025-11-14 13:45:53 +08:00
carryyu	6c3d1da62f	fix conflicts	2025-11-13 20:30:29 +08:00
Sunny-bot1	05da8e34c0	[BugFix][Metax] Fix metax compile issue in get_block_shape_and_split_kv_block (#5000) ... * fix metax compile * fix	2025-11-13 00:55:06 -08:00
Sunny-bot1	5b24013d46	skip DtoH capture (#4988)	2025-11-13 10:57:44 +08:00
周周周	6e01be28e0	format code (#4720)	2025-11-01 19:13:50 +08:00
Zhenghai Zhang	1712e1351b	【Hackathon 9th No.86】autogen MoeFastHardamardImplWrapper template_instantiation (#4592) ... * autogen MoeFastHardamardImplWrapper template_instantiation * fix codestyle * fix codestyle * add impl cu files	2025-10-30 10:28:36 +08:00
RAM	86d5006a57	[Graph Optimization][Speculative Decoding] Update yaml and fix typo (#4612)	2025-10-28 11:43:26 +08:00
xiaozude	f7069b8057	[Metax] adapt DeepSeek (#4498)	2025-10-24 10:14:53 +08:00
Sunny-bot1	8718fa34b2	support static C8 (#4568)	2025-10-23 22:01:03 +08:00
RAM	9dc5c3e370	[Graph Optimization] Support CUDAGraph Padding + MTP (#4545) ... * Support CUDAGraph Padding + MTP * support orther write cache kernel	2025-10-23 20:57:26 +08:00
Haonan Luo	1b9f351d21	Support GPT-OSS-BF16 (#4240) ... * [Feature] AppendAtten support sinks & HEAD_DIM=64 * fix bug * fix bug * fix bug * fix bug * [Feature] support gpt-oss * fix bug * add mask * support-gpt-oss * support-gpt-oss * fix long seq * support wint8 * support wint8 * support wint8 * update test * change sliding windows init pos --------- Co-authored-by: ming1753 <ideaminghp@163.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com>	2025-10-20 14:44:58 +08:00
周周周	817210e47f	[ATTN]delete code and add ffn and moe layer level test (#4440) ... * delete code * delete code * delete code * commit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit * copmmit	2025-10-19 16:23:11 +08:00
Sunny-bot1	930f7b781c	[Optimization] Put get_block_shape_and_split_kv_block in cuda graph for append attention backend (#4443) ... * get block in cuda graph * fix sot	2025-10-17 10:59:56 +08:00
Zhenghai Zhang	6adfbe07ad	【Hackathon 9th No.86】autogen MultiQueryDecoderAttention template_instantiation -part (#4383) ... * split MultiQueryDecoderAttention template_instantiation * update comment * CI	2025-10-16 17:08:19 +08:00
Sunny-bot1	6d0cc0dd9c	[Optimization] Optimize split_q_block kernel (#4367)	2025-10-15 11:28:00 +08:00
Sunny-bot1	a751d977bc	[Optimization] Fuse get_max_len and get_kv_max_len (#4369) ... * opt split_q_block * fuse max_lens and max kv_len	2025-10-13 20:35:00 +08:00
AIbin	f7eaca3971	【Bug Fix】mla enables tensorcore by default (#4354) ... * mla tensor-core kernel is enabled by default	2025-10-10 20:45:16 +08:00
lzy	3aa04fbf21	[MTP][Cfp8]supports spec dynamic cfp8 (#4290) ... * supports spec dynamic cfp8 * supports spec dynamic cfp8 --------- Co-authored-by: freeliuzc <lzc842650834@gmail.com>	2025-10-10 16:08:10 +08:00
Zhenghai Zhang	c46d5e48f8	【Hackathon 9th No.86】autogen MultiQueryAppendC8Attention template_instantiation -part (#4330) ... * split MultiQueryAppendC8Attention template_instantiation * update setup_ops.py * fix ci * fix bug	2025-10-10 15:07:48 +08:00
AIbin	c4ebaf8a07	【Inference Optimize】MLA Tensor-Core is enabled by default (#4335)	2025-10-10 10:54:56 +08:00
RAM	aa27b03bc0	[Executor]CUDAGraph support Speculate Decode (#3769) ... * success run ngram * Revert "[Code Simplification] remove cum_offsets (#3410)" This reverts commit 32b39620bc. * success run ngram5 tp4 42bs * success run ngram5 tp4 42bs * mtp draft commit * add decorator for target model * enable draft model in cudagraph v0.5 * revert revrt cum_offset * enable target model in cudagraph v0.9 And clean debug code * Revert "success run ngram" This reverts commit 8351e83993. * add reverted code * enable target model in cudagraph v0.9 * solve comment * fix bid < 0 * Enable Target Model Padding And Draft Model in cudagraph * solve problem * delete rebuild padding debug note * fast compile * Add capture list for mtp * success run 256 tp1 mtp * Enable Lite TP2 Bsz256 * realy enable tp2 bsz 256 * fix problem * Solve problem for Draft model in cudagraph * Solve comment * replace emptytensor as zeros * Solve comments * Revert "fast compile" This reverts commit 834639a7ff. * fix bug * fix merge bug * fix typo * fix bug --------- Co-authored-by: lizexu <2694294196@qq.com> Co-authored-by: littledgg <1658565283@qq.com> Co-authored-by: zeroRains <linjunlu@zerorains.top> Co-authored-by: gongshaotian <gstain5555@outlook.com>	2025-10-09 21:18:29 +08:00
Zhenghai Zhang	aa045aa84f	fix typos (#4274)	2025-09-27 09:25:43 +08:00
Yohanna	44010cee13	FIX] Fix CUDA error(700): 'cudaErrorIllegalAddress' in CascadeAppendWriteCacheKVQKV cache_kernel(). Continue when batch_id_per_token[token_idx] is default value -1. (#4218)	2025-09-24 14:08:49 +08:00
Yuan Xiaolan	de8638b1e9	fix dynamic Cfp8 computing error (#4119) ... Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-16 20:21:49 +08:00
AIbin	a7392a0ff9	【Inference Optimize】DeepSeek-V3-model MLA Optimize (#3886) ... * support MLA chunk_size auto search & cuda_graph	2025-09-11 10:46:09 +08:00
chen	637d96c6ae	[Feature] Support zai-org/GLM-4.5-Air BF16 model (#3928) ... * support glm45_air	2025-09-10 19:36:10 +08:00
freeliuzc	7ee100903f	support rope_3d in spec mode (#4034)	2025-09-10 03:15:05 -07:00
lzy	f12159b630	del batch id per token (#3963) ... * Update decoder_write_cache_with_rope_kernel.cu del batch_id_per_token * Update decoder_write_cache_with_rope_impl.cuh * Update test_append_attention.py * Update test_append_attention.py	2025-09-08 21:58:34 +08:00
co63oc	aadd6a94d8	fix typos (#3951)	2025-09-08 15:22:41 +08:00
Jundong Liu	3d0aaa5923	[Excutor] Experiment Feature-Support Prefill in cudagraph (#3459) ... * Support prefill in Cudagraph * Refactor GetBlockShapeAndSplitKVBlock Kernel V2 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.1 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.2 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.3 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.4 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.5 * Solve problem about encoder_num_blocks_x_cpu * Add early-exit mechanism for attention kernel * fix test case about append-attention * Update testcode, Add annotations to related tensors * move get_input_length_list * solve test_code * Add annotations about early-exit for attention kernel * Add annotations about early-exit for attention kernel2 * solve comment * solve mtp --------- Co-authored-by: RAM <gstian5555@outlook.com>	2025-09-08 13:12:24 +08:00
lzy	af49b81ffd	supports dynamic Cfp8 (#3767) ... * supports dynamic Cfp8 * add unittest	2025-09-07 20:41:29 -07:00
周周周	f6f726c773	clean code in sttantion (#3917)	2025-09-05 20:49:01 +08:00
xiaoxiaohehe001	f265a26f8b	support mtp rope_3d (#3791) ... * support mtp rope_3d * Update speculate_write_cache_with_rope_kernel.cu	2025-09-04 17:18:05 +08:00
Yuan Xiaolan	fa58a9fa8f	qk norm for speculate decode C16 (#3637)	2025-09-03 14:53:56 +08:00
lzy	7a521bbf62	Modify mask_offset‘s format (#3525) ... * modify mask_offset in decode * modify mask_offset unittest --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-02 03:05:35 -07:00
lizhenyun01	bed09ae8f8	fix mask_offset in append_attn (#3745) ... * fix mask_offset in append_attn * fix test	2025-08-31 15:03:16 +08:00
xiaoxiaohehe001	ad319a87cc	support fa3 rope3d (#3622)	2025-08-27 11:31:29 +08:00
xiaoxiaohehe001	70c75798a7	[NewFeatures] support noex rope3d (#3542) ... * [NewFeatures] support noex rope3d * [NewFeatures] support noex rope3d encoder	2025-08-26 11:44:57 +08:00
freeliuzc	76759108c9	[Feature][SpeculativeDecoding]Support tree-attention (#3514) ... * support tree-attention * fix merge bug * fix unit-test api * fix merge bug	2025-08-22 13:36:41 +08:00
Jundong Liu	ea4a3b479c	[Excutor] Increase buffer size to prevent address corruption; add forward metadata debug tool (#3404) ... * 修复buffer申请不够大，增加打印forwardmetadata的工具 * fix mistake * Make CPU tensor in CPUPlace * Add test about forward_meta_str and Add unitest_requirement --------- Co-authored-by: RAM <gstian5555@outlook.com>	2025-08-18 16:14:09 +08:00
lzy	1e06b9fa6d	make append_attn supports mask_offset (#3138) ... * make append_attn supports mask_offset * add unittest	2025-08-14 03:40:55 -07:00
Jundong Liu	1e4968e810	[Excutor] Fixed the issue of CUDA graph execution failure caused by different branches during decoding (#3223) ... * 彻底解决解码切块问题 * update C8 and C4 kernel * fix problem * fix with pre-commit * retain branch for mtp	2025-08-09 07:37:19 +08:00
gaoziyuan	a799d14df1	[Bugfix] Fix model accuracy in some ops (#3231) ... * fix noaux_tc op * fix * update * fix qk norm * fix linear for prequant loader * test * fix * fix * rm some print * fix noaux_tc op * test * Fix the confused enable_early_stop when only set early_stop_config (#3214) * fix the confused early_stop_config when only set early_stop_config * pre-commit * write a general method * Add ci case for min token and max token (#3229) Co-authored-by: xujing43 <xujing43@baidu.com> * add some evil cases (#3240) * add repitation early stop cases * add repitation early stop cases * add bad cases * add bad cases * add evil cases * qwen3_moe (#3084) * [Feature] support seed parameter (#3161) * support seed * fix * add SamplingMetadata seed test * The next_tokens values are inconsistent! * add air and rejection seed test * fix * add SamplingParams seed test * fix seed=0 * Default to defualt * fix * fix args_utils * fix review * fix review * fix * fix * add xpu,gcu,iluvatar support seed * fix * 【Fix Bug】 修复 fa3 支持集中式bug (#3235) * fix fa3 集中式bug * 增加qknorm参数 * fix qk norm * fix * update * fix linear for prequant loader * fix * fix * rm some print * fix * fix moe init weight&scale * fix moe init weight&scale --------- Co-authored-by: bukejiyu <395822456@qq.com> Co-authored-by: yuanxiaolan <yuanxiaolan01@baidu.com> Co-authored-by: Zero Rains <linjunlu@zerorains.top> Co-authored-by: xjkmfa <108254620+xjkmfa@users.noreply.github.com> Co-authored-by: xujing43 <xujing43@baidu.com> Co-authored-by: Divano <dddivano@outlook.com> Co-authored-by: bukejiyu <52310069+bukejiyu@users.noreply.github.com> Co-authored-by: lizexu123 <39205361+lizexu123@users.noreply.github.com> Co-authored-by: yangjianfengo1 <125249383+yangjianfengo1@users.noreply.github.com> Co-authored-by: qingqing01 <dangqingqing@baidu.com>	2025-08-08 17:30:37 +08:00
Yuan Xiaolan	7ce00e597c	support qk norm (#3145)	2025-08-05 16:46:14 +08:00
RAM	d850660872	[Executor] Refactor GetBlockShapeAndSplitKVBlock Kernel (#2989) ... * reset decoder_block_shape_q buffer * refactor GetBlockShapeAndSplitKVBlock Kernel and cudagraph padding batch * update decode_max_tile_size * fix pre-commit * update block_multihead_attn_backend * update flas attn backend * update MLA Attention * update XPU Attention * update gcu,iluvatar model runner * Update MTP * fix MTP bug	2025-07-31 00:09:31 +08:00
lizhenyun01	238766e403	fix c4 prompt_cache	2025-07-28 14:31:37 +08:00