[LogProbs]Enable prompt logprobs output and modify data transmission method for the online interface. (#5089)

* add prompt logprobs

* Merge prompt_logprobs_tensors and prompt_logprobs

* fix param check

* trigger ci

* fix unitest

* fix logprobs bug
This commit is contained in:
qwes5s5
2025-12-02 13:49:51 +08:00
committed by GitHub
parent af39819fcd
commit 117980dd4e
27 changed files with 4947 additions and 233 deletions

View File

@@ -181,6 +181,7 @@ async def lifespan(app: FastAPI):
port=int(os.environ.get("INFERENCE_MSG_QUEUE_ID", "0")),
fd_config=fd_config,
workers=args.workers,
max_logprobs=args.max_logprobs,
)
await engine_client.connection_manager.initialize()
app.state.dynamic_load_weight = args.dynamic_load_weight