apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-05 08:37:06 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[Serving][backend]serving support multi stream and backend support external stream (#431)

Browse Source 
* serving support multi stream

* pybind add external stream

Co-authored-by: Jason <jiangjiajun@baidu.com>
This commit is contained in:
heliqi
2022-10-26 14:46:13 +08:00
committed by GitHub
parent 718698a32a
commit b064ddf7ed
10 changed files with 32 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
fastdeploy/runtime.h
View File

@@ -102,6 +102,8 @@ struct FASTDEPLOY_DECL RuntimeOption {
/// Use Nvidia GPU to inference
void UseGpu(int gpu_id = 0);
void SetExternalStream(void* external_stream);
/*
* @brief Set number of cpu threads while inference on CPU, by default it will decided by the different backends
*/
@@ -232,6 +234,8 @@ struct FASTDEPLOY_DECL RuntimeOption {
Device device = Device::CPU;
void* external_stream_ = nullptr;
bool enable_pinned_memory = false;
// ======Only for ORT Backend========