192 Commits

Author	SHA1	Message	Date
李泳桦	6265f4385f	[feat] support prefix cache clearing when /clear_load_weight is called (#4008) ... * [feat] support clearing prefix cache (cherry-picked from release/2.1) * [fix] fix ipc suffix, use port instead * [fix] fix prefix caching not enabled * [fix] fix key/value_cache_scales indent * [fix] fix ep group all-reduce * [fix] fix clear/update lock not working when workers > 1 * [chore] add preemption triggered info log * [fix] fix code style * [fix] fix max_num_seqs config * [fix] do not force enable_prefix_caching=False in dynamic loading * [fix] fix ci * Revert "[fix] fix ci" This reverts commit 0bc6d55cc8. * [fix] initialize available_gpu_block_num with max_gpu_block_num * [fix] fix config splitwise_role * [fix] fix clearing caches synchronization and add more logs * [chore] print cache_ready_signal in log * [fix] fix scheduler_config.splitwise_role * [fix] fix cache_messager cache_ready_signal create=True * [fix] stop cache messager from launching in mixed deployment	2025-09-28 19:42:53 +08:00
Sunny-bot1	aa1cc09c5b	fix machete pre quant (#4295)	2025-09-28 16:11:09 +08:00
Lucas	87179cb744	[XPU] support XPU VL model inference (#4030) ... * [XPU] support XPU VL model inference * fix image op import and device check * rebase develop * fix perf	2025-09-25 14:34:15 +08:00
chen	7c1fd19f0f	[OPs] MoE support wfp8afp8(channelwise) and improve per_token_quant_fp8 (#4238)	2025-09-24 16:39:51 +08:00
lizexu123	e8318b7477	[BugFix] fix qwen3-embedding model tp>1 (#4223) ... * support qwen3-embedding * fix ci bug * fix * fix ci bug * fix ci bug * fix * fix qwen3-embedding * fix * fix * fix	2025-09-24 14:13:26 +08:00
chen	3161014e49	[BugFix]fix v1 loader moe bf16, and supoort dynamic_load_weight create quant param (#4229) ... * fix v1 loader moe bf16, and supoort dynamic_load_weight create quant param * include_stop_str_in_output=False not return eos text	2025-09-24 14:12:05 +08:00
fmiao2372	f1b5392e20	[Intel HPU] Support intel hpu platform (#4161) ... * [Intel HPU] Support intel hpu platform * fix some issues * apply precommit and move AttentionBackend_HPU * fix format issue * correct ops import * fix ci issue * update code in layers * fix code style issue * remove dense tp moe ep mode * fix enc_dec_block_num * fix rebase issue * rename hpu to gaudi in readme * rename ForwardMeta_HPU to HPUForwardMeta	2025-09-24 12:27:50 +08:00
chen	1a6283424e	Fix noaux_tc cuda Error 700 in CUDAGraph (#4174)	2025-09-23 18:41:33 +08:00
lizexu123	c96a535a5d	[Feature] support qwen3-embedding model load (#4202) ... * support qwen3-embedding * fix ci bug * fix * fix ci bug * fix ci bug * fix	2025-09-23 00:14:35 -07:00
yangjianfengo1	4325b737e7	【FIX】Change the name of sparse attn from moba to plas (#4006) (#4076) ... * 【FIX】Change the name of sparse attn from moba to plas (#4006) * 更新文档 * 【docs】 update readme (#4000) * 更新文档 * update readme * update docs * 【FIX】Change the name of sparse attn from moba to plas (#3845) * 更新文档 * 更新文档 * 更新文档 * 更新文档 * 修改moba为plas * code style * update ci * code style * update ci * code style --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * fix max_num_seqs * fix test load attn --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-23 10:26:40 +08:00
yzwu	504461b6b5	[Iluvatar GPU] Optimize attention performance and fix moe load ckpt error (#3651)	2025-09-22 21:13:59 +08:00
chenjian	918ccdb123	[Feature] Support pd ep deployment with yiyan adapter (#4029) ... * [Feature] Support mixed deployment with yiyan adapter in release2.2 * fix metrics * add unit test * add unit test * add unit test * Support pd ep deployment with yiyan adapter * Support pd ep deployment with yiyan adapter * refactor cache messager * support scheduler v1 in PD * suppport pd v1 + chunk prefill * suppport pd v1 + chunk prefill * add eplb * support eplb * support eplb * support eplb * support v1 * fix * fix * fix bug * remove eplb support * support prefix cache in P * fix bug * fix bug * support one stop in V1 * fix bug * fix ci * fix ci * fix * fix * fix * fix * fix --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-22 16:41:38 +08:00
lizexu123	c86945ef49	[Feature] support pool (#3827) ... * support pool * update pooling * add pooler_config and check * update * support AutoWeightsLoader load weight * fix * update * delete print * update pre-commit * fix * fix xpu * fix ModelRegistry->model_registry * fix Copilot review * fix pooler.py * delete StepPooler * fix abstract * fix default_loader_v1 * fix Pre Commit * support torch qwen3 dense * add test and fix torch-qwen * fix * fix * adapter ci: * fix review * fix pooling_params.py * fix * fix tasks.py 2025 * fix print and logger * Modefy ModelRegistry and delete AutoWeightsLoader * fix logger * fix test_embedding * fix ci bug * ernie4_5 model_registry * fix test * support Qwen3-Embedding-0.6B tp=1 load * fix extra code * fix * delete fix vocab_size * delete prepare_params_dict * fix:	2025-09-22 14:09:09 +08:00
YuanRisheng	24180fba0a	[FDConfig]Remove splitwise_role and engine_worker_queue_port in FDConfig (#4147) ... * remove splitwise_role and engine_worker_queue_port * fix xpu * fix xpu * fix xpu * fix unittest * resolve conflct	2025-09-19 17:01:52 +08:00
Sunny-bot1	c3b8ebeb18	[Optimize] Machete using group scale default (#4121)	2025-09-18 13:51:11 +08:00
gaoziyuan	896e3bb606	[NewFeture]add ep rollout model init and update/clear ep buffer (#4039) ... * fix gid * merge * fix test * fix bug * fix * fix ci	2025-09-17 20:24:53 +08:00
YuanRisheng	2e9e53ff7e	[FDConfig]Remove max_num_batched_tokens/max_num_seqs in parallel config (#4116) ... * remove max_num_batched_tokens in parallel config * remove max_num_seqs * update test case * fix test * fix --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-17 10:43:35 +08:00
bukejiyu	113e330030	fix bf16 and add comments (#4106)	2025-09-15 17:23:07 +08:00
zhupengyang	9409665713	[xpu] support ep (#4067)	2025-09-15 13:53:11 +08:00
bukejiyu	29ed617f0f	[v1 loader]qwen Offline fp8 (#4036) ... * support offline fp8 * update ut * update ut * update ut * fix * update * update	2025-09-15 13:44:11 +08:00
Sunny-bot1	b1a5b756a3	[Optimize] Support WINT8 and group scale for Machete (#3905)	2025-09-15 12:01:34 +08:00
YuanRisheng	88ea565aba	[BugFix]Fix load kv cache quant scale (#4077) ... * fix kv cache * fix kv_cache * fix kv cache	2025-09-12 17:44:03 +08:00
SuperNova	805f29a06c	[Feature] refactor metax_gpu attention and moe and remove some useless code (#3688) ... Co-authored-by: yongqiangma <xing.wo@163.com>	2025-09-12 14:40:25 +08:00
co63oc	8466219ec8	fix typos (#3840) ... * fix typos * ci --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-12 11:04:38 +08:00
chen	4859f40b20	[Feature] GLM-45-AIR Support Mix Quantization(Dense wfp8afp8 and wint8 triton_moe_backend) (#4051)	2025-09-11 20:08:09 +08:00
AIbin	a7392a0ff9	【Inference Optimize】DeepSeek-V3-model MLA Optimize (#3886) ... * support MLA chunk_size auto search & cuda_graph	2025-09-11 10:46:09 +08:00
chen	637d96c6ae	[Feature] Support zai-org/GLM-4.5-Air BF16 model (#3928) ... * support glm45_air	2025-09-10 19:36:10 +08:00
Yuanle Liu	c3b2a60fb8	[BugFix] Fix the abnormal memory usage caused by shape errors in the triton moe backend (#4026) ... * fix device_id to in * fix triton_moe bug	2025-09-09 20:05:54 -07:00
Sunny-bot1	3b1da6e4dd	support v1 loader for machete (#3999)	2025-09-10 10:21:33 +08:00
YuanRisheng	b3fac5bde1	[V1 Loader] Ernie kv cache quant support v1 loader (#3899) ... * support c8 for ernie * add unittest * support vl * fix c8	2025-09-09 05:25:08 -07:00
Jiang-Jia-Jun	c60adf4281	Revert "【FIX】Change the name of sparse attn from moba to plas (#3845)" (#4001) ... This reverts commit e31c8f7336.	2025-09-09 11:08:23 +08:00
yangjianfengo1	e31c8f7336	【FIX】Change the name of sparse attn from moba to plas (#3845) ... * 更新文档 * 更新文档 * 更新文档 * 更新文档 * 修改moba为plas * code style * update ci * code style * update ci	2025-09-09 10:56:50 +08:00
Jundong Liu	3d0aaa5923	[Excutor] Experiment Feature-Support Prefill in cudagraph (#3459) ... * Support prefill in Cudagraph * Refactor GetBlockShapeAndSplitKVBlock Kernel V2 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.1 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.2 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.3 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.4 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.5 * Solve problem about encoder_num_blocks_x_cpu * Add early-exit mechanism for attention kernel * fix test case about append-attention * Update testcode, Add annotations to related tensors * move get_input_length_list * solve test_code * Add annotations about early-exit for attention kernel * Add annotations about early-exit for attention kernel2 * solve comment * solve mtp --------- Co-authored-by: RAM <gstian5555@outlook.com>	2025-09-08 13:12:24 +08:00
lzy	af49b81ffd	supports dynamic Cfp8 (#3767) ... * supports dynamic Cfp8 * add unittest	2025-09-07 20:41:29 -07:00
bukejiyu	e52ce1c4b1	cache feature (#3857)	2025-09-07 18:52:46 +08:00
Yuan Xiaolan	2cf55168ca	load hadamard_block_size from config (#3797)	2025-09-05 17:07:58 +08:00
AIbin	41aee08982	【Inference Optimize】Update MergedReplicatedLinear for DSK qkv_a_proj_with_mqa. (#3673) ... * support MergedReplicatedLinear * update MergedReplicatedLinear to support DSK_wint4 V1_load * update model name * update linear class * fix * fix v0 moe_bias load --------- Co-authored-by: bukejiyu <52310069+bukejiyu@users.noreply.github.com>	2025-09-04 21:16:05 -07:00
gaoziyuan	ab1929f5ff	fix mem boom in ep (#3854)	2025-09-05 11:48:21 +08:00
Ayakouji	31313e0f3d	[Feature] ernie4_5_vl_moe support huggingface safetensor loading (#3750) ... * update * update * update in tp * add todo * update --------- Co-authored-by: aquagull <hongyuh@qq.com>	2025-09-03 02:58:59 -07:00
YuanRisheng	0a1ce612c2	V1 loader support ep (#3801)	2025-09-03 16:05:41 +08:00
co63oc	ce998449e0	fix w8a8.py (#3733)	2025-09-03 10:57:26 +08:00
yangjianfengo1	8e1b35a09b	【Fix bug] w4afp8 的nblock固定为256，并且fa3的append attn 增加mask参数 (#3771) ... * fix w4afp8 * 增加集中式配置 * codestyle * fix fa3 append attn	2025-09-02 19:17:01 +08:00
bukejiyu	b6a4115369	[v1loader]Reduce EB300B model loading time (#3700) ... * speed up eb45 * update	2025-09-02 19:13:57 +08:00
Yuanle Liu	306c024ff3	[BugFix] fix error of import paddle.base.core.Config (#3761) ... * 延迟 import Config * support chunked_prefill * support chunked_prefill	2025-09-02 17:23:27 +08:00
ltd0924	905d89e42f	[Feature] support model weight update in ep (#3765) ... * support model weight update in ep * support model weight update in ep * support model weight update in ep * support model weight update in ep * Update fused_moe_backend_base.py * Update worker_process.py * Update worker_process.py * Update dynamic_weight_manager.py	2025-09-02 17:16:03 +08:00
kevin	1908465542	[Feature] mm and thinking model support structred output (#2749) ... * mm support structured output * update code * update code * update format * update code * update code * add enable_thinking default * update code * add structured_outputs test case * add ci install xgrammar * add ci timeout time * update test for structured_outputs * update code * add error traceback info * update error msg * update structred output code * update code * update code * update config * update torch version --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-02 16:21:09 +08:00
co63oc	d6369b4d51	fix typos (#3684)	2025-09-01 17:50:17 +08:00
lizhenyun01	bed09ae8f8	fix mask_offset in append_attn (#3745) ... * fix mask_offset in append_attn * fix test	2025-08-31 15:03:16 +08:00
Sunny-bot1	fe5d09f9ee	[FIX]Fix Machete compile via ENABLE_MACHETE (#3727) ... * add ENABLE_MACHETE * fix * revert * update * pre_commit * fix * fix --------- Co-authored-by: Ayakouji <yuhongh@qq.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: aquagull <hongyuh@qq.com>	2025-08-30 17:50:17 +08:00
chen	7568b20098	check (#3720)	2025-08-30 16:04:20 +08:00