apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-13 04:13:58 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[Model] Add Paddle3D smoke model (#1766)

Browse Source 
* add smoke model

* add 3d vis

* update code

* update doc

* mv paddle3d from detection to perception

* update result for velocity

* update code for CI

* add set input data for TRT backend

* add serving support for smoke model

* update code

* update code

* update code

---------

Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
This commit is contained in:
yeliang2258
2023-04-14 16:30:56 +08:00
committed by GitHub
parent 81fbd54c9d
commit a509dd8ec1
53 changed files with 2610 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
examples/vision/perception/paddle3d/smoke/README.md Executable file
View File

@@ -0,0 +1,11 @@
English | [简体中文](README_CN.md)
# Smoke Ready-to-deploy Model
The Smoke deployment model implements the Smoke model from Paddle3D. For more detailed information about the model, please refer to [Smoke Introduction](https://github.com/PaddlePaddle/Paddle3D/tree/develop/docs/models/smoke)
## Detailed Deployment Documents
- [Python Deployment](python)
- [C++ Deployment](cpp)

12
examples/vision/perception/paddle3d/smoke/README_CN.md Executable file
View File

@@ -0,0 +1,12 @@
[English](README.md) | 简体中文
# Smoke 准备部署模型
Smoke 部署模型实现来自 Paddle3D 的 Smoke 模型，模型相关的更多详细信息可以参考[Smoke 介绍](https://github.com/PaddlePaddle/Paddle3D/tree/develop/docs/models/smoke)
## 详细部署文档
- [Python部署](python)
- [C++部署](cpp)

14
examples/vision/perception/paddle3d/smoke/cpp/CMakeLists.txt Executable file
View File

@@ -0,0 +1,14 @@
PROJECT(infer_demo C CXX)
CMAKE_MINIMUM_REQUIRED (VERSION 3.10)
# 指定下载解压后的fastdeploy库路径
option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.")
include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake)
# 添加FastDeploy依赖头文件
include_directories(${FASTDEPLOY_INCS})
add_executable(infer_demo ${PROJECT_SOURCE_DIR}/infer.cc)
# 添加FastDeploy库依赖
target_link_libraries(infer_demo ${FASTDEPLOY_LIBS})

80
examples/vision/perception/paddle3d/smoke/cpp/README.md Executable file
View File

@@ -0,0 +1,80 @@
English | [简体中文](README_CN.md)
# Smoke C++ Deployment Example
This directory provides an example of `infer.cc` to quickly complete the deployment of Smoke on CPU/GPU.
Before deployment, the following two steps need to be confirmed
- 1. The hardware and software environment meets the requirements, refer to [FastDeploy environment requirements](../../../../../docs/en/build_and_install/download_prebuilt_libraries.md)
- 2. According to the development environment, download the precompiled deployment library and samples code, refer to [FastDeploy prebuilt library](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
Taking CPU inference on Linux as an example, execute the following command in this directory to complete the compilation test. To support this model, you need to ensure FastDeploy version 1.0.6 or higher (x.x.x>=1.0.6)
```bash
mkdir build
cd build
# Download the FastDeploy precompiled library, users can choose the appropriate version to use in the `FastDeploy precompiled library` mentioned above
wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz
tar xvf fastdeploy-linux-x64-x.x.x.tgz
cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x
make -j
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
tar -xf smoke.tar.gz
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
# CPU
./infer_demo smoke smoke_test.png 0
# GPU
./infer_demo smoke smoke_test.png 1
```
The visual result after running is shown in the figure below
<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
The above commands are only applicable to Linux or MacOS. For the usage of SDK under Windows, please refer to:
- [How to use FastDeploy C++ SDK in Windows](../../../../../docs/en/faq/use_sdk_on_windows.md)
## Smoke C++ interface
### Class Smoke
```c++
fastdeploy::vision::detection::Smoke(
const string& model_file,
const string& params_file,
const string& config_file,
const RuntimeOption& runtime_option = RuntimeOption(),
const ModelFormat& model_format = ModelFormat::PADDLE)
```
Smoke model loading and initialization.
**parameter**
> * **model_file**(str): model file path
> * **params_file**(str): parameter file path
> * **config_file**(str): configuration file path
> * **runtime_option**(RuntimeOption): Backend reasoning configuration, the default is None, that is, the default configuration is used
> * **model_format**(ModelFormat): model format, the default is Paddle format
#### Predict function
> ```c++
> Smoke::Predict(cv::Mat* im, PerceptionResult* result)
> ```
>
> Model prediction interface, the input image directly outputs the detection result.
>
> **parameters**
>
> > * **im**: input image, note that it must be in HWC, BGR format
> > * **result**: Detection result, including the detection frame, the confidence of each frame, PerceptionResult description reference [visual model prediction results](../../../../../docs/api /vision_results/)
- [Model Introduction](../../)
- [Python deployment](../python)
- [Vision Model Prediction Results](../../../../../docs/api/vision_results/)
- [How to switch model inference backend engine](../../../../../docs/en/faq/how_to_change_backend.md)

80
examples/vision/perception/paddle3d/smoke/cpp/README_CN.md Executable file
View File

@@ -0,0 +1,80 @@
[English](README.md) | 简体中文
# Smoke C++部署示例
本目录下提供 `infer.cc` 快速完成 Smoke 在 CPU/GPU 上部署的示例。
在部署前，需确认以下两个步骤
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
- 2. 根据开发环境，下载预编译部署库和samples代码，参考[FastDeploy预编译库](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
以Linux上 CPU 推理为例，在本目录执行如下命令即可完成编译测试，支持此模型需保证 FastDeploy 版本1.0.6以上(x.x.x>=1.0.6)
```bash
mkdir build
cd build
# 下载FastDeploy预编译库，用户可在上文提到的`FastDeploy预编译库`中自行选择合适的版本使用
wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz
tar xvf fastdeploy-linux-x64-x.x.x.tgz
cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x
make -j
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
tar -xf smoke.tar.gz
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
# CPU推理
./infer_demo smoke smoke_test.png 0
# GPU推理
./infer_demo smoke smoke_test.png 1
```
运行完成可视化结果如下图所示
<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考:
- [如何在Windows中使用FastDeploy C++ SDK](../../../../../docs/cn/faq/use_sdk_on_windows.md)
## Smoke C++ 接口
### Smoke 类
```c++
fastdeploy::vision::detection::Smoke(
const string& model_file,
const string& params_file,
const string& config_file,
const RuntimeOption& runtime_option = RuntimeOption(),
const ModelFormat& model_format = ModelFormat::PADDLE)
```
Smoke模型加载和初始化。
**参数**
> * **model_file**(str): 模型文件路径
> * **params_file**(str): 参数文件路径
> * **config_file**(str): 配置文件路径
> * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置
> * **model_format**(ModelFormat): 模型格式，默认为Paddle格式
#### Predict函数
> ```c++
> Smoke::Predict(cv::Mat* im, PerceptionResult* result)
> ```
>
> 模型预测接口，输入图像直接输出检测结果。
>
> **参数**
>
> > * **im**: 输入图像，注意需为HWC，BGR格式
> > * **result**: 检测结果，包括检测框，各个框的置信度, PerceptionResult 说明参考[视觉模型预测结果](../../../../../docs/api/vision_results/)
- [模型介绍](../../)
- [Python部署](../python)
- [视觉模型预测结果](../../../../../docs/api/vision_results/)
- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)

83
examples/vision/perception/paddle3d/smoke/cpp/infer.cc Normal file
View File

@@ -0,0 +1,83 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "fastdeploy/vision.h"
#ifdef WIN32
const char sep = '\\';
#else
const char sep = '/';
#endif
void InitAndInfer(const std::string& model_dir, const std::string& image_file,
const fastdeploy::RuntimeOption& option) {
auto model_file = model_dir + sep + "smoke.pdmodel";
auto params_file = model_dir + sep + "smoke.pdiparams";
auto config_file = model_dir + sep + "infer_cfg.yml";
fastdeploy::vision::EnableFlyCV();
auto model = fastdeploy::vision::perception::Smoke(
model_file, params_file, config_file, option,
fastdeploy::ModelFormat::PADDLE);
assert(model.Initialized());
auto im = cv::imread(image_file);
fastdeploy::vision::PerceptionResult res;
if (!model.Predict(im, &res)) {
std::cerr << "Failed to predict." << std::endl;
return;
}
std::cout << res.Str() << std::endl;
auto vis_im = fastdeploy::vision::VisPerception(im, res, config_file);
cv::imwrite("vis_result.jpg", vis_im);
std::cout << "Visualized result saved in ./vis_result.jpg" << std::endl;
}
int main(int argc, char* argv[]) {
if (argc < 4) {
std::cout << "Usage: infer_demo path/to/paddle_model"
"path/to/image "
"run_option, "
"e.g ./infer_demo ./smoke ./00000.png 0"
<< std::endl;
std::cout << "The data type of run_option is int, 0: run with cpu; 1: run "
"with gpu; 2: run with paddle-trt"
<< std::endl;
return -1;
}
fastdeploy::RuntimeOption option;
if (std::atoi(argv[3]) == 0) {
option.UseCpu();
} else if (std::atoi(argv[3]) == 1) {
option.UseGpu();
} else if (std::atoi(argv[3]) == 2) {
option.UseGpu();
option.UseTrtBackend();
option.EnablePaddleToTrt();
option.SetTrtInputShape("images", {1, 3, 384, 1280});
option.SetTrtInputShape("down_ratios", {1, 2});
option.SetTrtInputShape("trans_cam_to_img", {1, 3, 3});
option.SetTrtInputData("trans_cam_to_img",
{721.53771973, 0., 609.55932617, 0., 721.53771973,
172.85400391, 0, 0, 1});
option.EnablePaddleTrtCollectShape();
}
option.UsePaddleBackend();
std::string model_dir = argv[1];
std::string test_image = argv[2];
InitAndInfer(model_dir, test_image, option);
return 0;
}

67
examples/vision/perception/paddle3d/smoke/python/README.md Executable file
View File

@@ -0,0 +1,67 @@
English | [简体中文](README_CN.md)
# Smoke Python Deployment Example
Before deployment, the following two steps need to be confirmed
- 1. The hardware and software environment meets the requirements, refer to [FastDeploy environment requirements](../../../../../docs/en/build_and_install/download_prebuilt_libraries.md)
- 2. FastDeploy Python whl package installation, refer to [FastDeploy Python Installation](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
This directory provides an example of `infer.py` to quickly complete the deployment of Smoke on CPU/GPU. Execute the following script to complete
```bash
#Download deployment sample code
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd examples/vision/vision/paddle3d/smoke/python
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
tar -xf smoke.tar.gz
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
# CPU reasoning
python infer.py --model smoke --image smoke_test.png --device cpu
# GPU inference
python infer.py --model smoke --image smoke_test.png --device gpu
```
The visual result after running is shown in the figure below
<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
## Smoke Python interface
```python
fastdeploy.vision.detection.Smoke(model_file, params_file, config_file, runtime_option=None, model_format=ModelFormat.PADDLE)
```
Smoke model loading and initialization.
**parameter**
> * **model_file**(str): model file path
> * **params_file**(str): parameter file path
> * **config_file**(str): configuration file path
> * **runtime_option**(RuntimeOption): Backend reasoning configuration, the default is None, that is, the default configuration is used
> * **model_format**(ModelFormat): model format, the default is Paddle format
### predict function
> ```python
> Smoke. predict(image_data)
> ```
>
> Model prediction interface, the input image directly outputs the detection result.
>
> **parameters**
>
> > * **image_data**(np.ndarray): input data, note that it must be in HWC, BGR format
> **Back**
>
> > Return the `fastdeploy.vision.PerceptionResult` structure, structure description reference document [Vision Model Prediction Results](../../../../../docs/api/vision_results/)
## Other documents
- [Smoke Model Introduction](..)
- [Smoke C++ deployment](../cpp)
- [Description of model prediction results](../../../../../docs/api/vision_results/)
- [How to switch model inference backend engine](../../../../../docs/en/faq/how_to_change_backend.md)

68
examples/vision/perception/paddle3d/smoke/python/README_CN.md Executable file
View File

@@ -0,0 +1,68 @@
[English](README.md) | 简体中文
# Smoke Python 部署示例
在部署前，需确认以下两个步骤
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
- 2. FastDeploy Python whl 包安装，参考[FastDeploy Python安装](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
本目录下提供 `infer.py` 快速完成 Smoke 在 CPU/GPU上部署的示例。执行如下脚本即可完成
```bash
#下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd examples/vision/vision/paddle3d/smoke/python
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
tar -xf smoke.tar.gz
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
# CPU推理
python infer.py --model smoke --image smoke_test.png --device cpu
# GPU推理
python infer.py --model smoke --image smoke_test.png --device gpu
```
运行完成可视化结果如下图所示
<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
## Smoke Python接口
```python
fastdeploy.vision.detection.Smoke(model_file, params_file, config_file, runtime_option=None, model_format=ModelFormat.PADDLE)
```
Smoke模型加载和初始化。
**参数**
> * **model_file**(str): 模型文件路径
> * **params_file**(str): 参数文件路径
> * **config_file**(str): 配置文件路径
> * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置
> * **model_format**(ModelFormat): 模型格式，默认为Paddle格式
### predict 函数
> ```python
> Smoke.predict(image_data)
> ```
>
> 模型预测结口，输入图像直接输出检测结果。
>
> **参数**
>
> > * **image_data**(np.ndarray): 输入数据，注意需为HWC，BGR格式
> **返回**
>
> > 返回`fastdeploy.vision.PerceptionResult`结构体，结构体说明参考文档[视觉模型预测结果](../../../../../docs/api/vision_results/)
## 其它文档
- [Smoke 模型介绍](..)
- [Smoke C++部署](../cpp)
- [模型预测结果说明](../../../../../docs/api/vision_results/)
- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)

50
examples/vision/perception/paddle3d/smoke/python/infer.py Executable file
View File

@@ -0,0 +1,50 @@
import fastdeploy as fd
import cv2
import os
from fastdeploy import ModelFormat
def parse_arguments():
import argparse
import ast
parser = argparse.ArgumentParser()
parser.add_argument(
"--model", required=True, help="Path of smoke paddle model.")
parser.add_argument(
"--image", required=True, help="Path of test image file.")
parser.add_argument(
"--device",
type=str,
default='cpu',
help="Type of inference device, support 'cpu' or 'gpu'.")
return parser.parse_args()
def build_option(args):
option = fd.RuntimeOption()
if args.device.lower() == "gpu":
option.use_gpu(0)
if args.device.lower() == "cpu":
option.use_cpu()
return option
args = parse_arguments()
model_file = os.path.join(args.model, "smoke.pdmodel")
params_file = os.path.join(args.model, "smoke.pdiparams")
config_file = os.path.join(args.model, "infer_cfg.yml")
# 配置runtime，加载模型
runtime_option = build_option(args)
model = fd.vision.perception.Smoke(
model_file, params_file, config_file, runtime_option=runtime_option)
# 预测图片检测结果
im = cv2.imread(args.image)
result = model.predict(im)
print(result)
# 预测结果可视化
vis_im = fd.vision.vis_perception(im, result, config_file)
cv2.imwrite("visualized_result.jpg", vis_im)
print("Visualized result save in ./visualized_result.jpg")

85
examples/vision/perception/paddle3d/smoke/serving/README.md Executable file
View File

@@ -0,0 +1,85 @@
English | [简体中文](README_CN.md)
# Smoke service deployment example
This document introduces the service deployment of the Smoke model in Paddle3D.
For Smoke model export and pre-training model download, please refer to [Smoke Model Deployment](../README.md) document.
Before service deployment, you need to confirm
- 1. Please refer to [FastDeploy Service-based Deployment](../../../../../serving/README_CN.md) for the hardware and software environment requirements of the service image and the image pull command
## Start the service
```bash
#Download deployment sample code
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/perception/paddle3d/smoke/serving
#Download the Smoke model file and test image
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
tar -xf smoke.tar.gz
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
# Put the configuration file into the preprocessing directory
mv smoke/infer_cfg.yml models/preprocess/1/
# Put the model into the models/runtime/1 directory, and rename it to model.pdmodel and model.pdiparams
mv smoke/smoke.pdmodel models/runtime/1/model.pdmodel
mv smoke/smoke.pdiparams models/runtime/1/model.pdiparams
# Pull the fastdeploy image (x.y.z is the image version number, which needs to be replaced with the fastdeploy version number)
# GPU mirroring
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
# CPU mirroring
docker pull paddlepaddle/fastdeploy:z.y.z-cpu-only-21.10
# Run the container. The container name is fd_serving, and mount the current directory as the /serving directory of the container
nvidia-docker run -it --net=host --name fd_serving --shm-size="1g" -v `pwd`/:/serving registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4 -trt8.4-21.10 bash
# Start the service (if you do not set the CUDA_VISIBLE_DEVICES environment variable, you will have the scheduling authority of all GPU cards)
CUDA_VISIBLE_DEVICES=0 fastdeployserver --model-repository=/serving/models
```
>> **NOTE**:
>> For pulling images, please refer to [Serving Deployment Main Document](../../../../../serving/README_CN.md)
>> Execute fastdeployserver to start the service and "Address already in use" appears, please use `--grpc-port` to specify the grpc port number to start the service, and change the request port number in the client example.
>> Other startup parameters can be viewed using fastdeployserver --help
After the service starts successfully, there will be the following output:
```
 …
I0928 04:51:15.784517 206 grpc_server.cc:4117] Started GRPCInferenceService at 0.0.0.0:8001
I0928 04:51:15.785177 206 http_server.cc:2815] Started HTTPService at 0.0.0.0:8000
I0928 04:51:15.826578 206 http_server.cc:167] Started Metrics Service at 0.0.0.0:8002
```
## Client request
Execute the following command on the physical machine, send the grpc request and output the result
```
#Download test image
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
# Install client dependencies
python3 -m pip install tritonclient[all]
# send request
python3 smoke_grpc_client.py
```
After sending the request successfully, the detection result in json format will be returned and printed out:
```
output_name: PERCEPTION_RESULT
, 0.0068892366252839565]
label_ids: [2, 0, 1, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2 , 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 , 0]
```
## Configuration modification
The current default configuration runs the Paddle engine on the CPU, if you want to run it on the GPU or other inference engines. It is necessary to modify the configuration in `models/runtime/config.pbtxt`, for details, please refer to [configuration document](../../../../../serving/docs/zh_CN/model_configuration.md)

85
examples/vision/perception/paddle3d/smoke/serving/README_CN.md Executable file
View File

@@ -0,0 +1,85 @@
[English](README.md) | 简体中文
# Smoke 服务化部署示例
本文档介绍 Paddle3D 中 Smoke 模型的服务化部署。
Smoke 模型导出和预训练模型下载请看[Smoke模型部署](../README.md)文档。
在服务化部署前，需确认
- 1. 服务化镜像的软硬件环境要求和镜像拉取命令请参考[FastDeploy服务化部署](../../../../../serving/README_CN.md)
## 启动服务
```bash
#下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/perception/paddle3d/smoke/serving
#下载 Smoke 模型文件和测试图片
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
tar -xf smoke.tar.gz
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
# 将配置文件放入预处理目录
mv smoke/infer_cfg.yml models/preprocess/1/
# 将模型放入 models/runtime/1 目录下, 并重命名为 model.pdmodel 和 model.pdiparams
mv smoke/smoke.pdmodel models/runtime/1/model.pdmodel
mv smoke/smoke.pdiparams models/runtime/1/model.pdiparams
# 拉取 fastdeploy 镜像(x.y.z 为镜像版本号，需替换成 fastdeploy 版本数字)
# GPU 镜像
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
# CPU 镜像
docker pull paddlepaddle/fastdeploy:z.y.z-cpu-only-21.10
# 运行容器.容器名字为 fd_serving, 并挂载当前目录为容器的 /serving 目录
nvidia-docker run -it --net=host --name fd_serving --shm-size="1g" -v `pwd`/:/serving registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
# 启动服务(不设置 CUDA_VISIBLE_DEVICES 环境变量，会拥有所有 GPU 卡的调度权限)
CUDA_VISIBLE_DEVICES=0 fastdeployserver --model-repository=/serving/models
```
>> **注意**:
>> 拉取镜像请看[服务化部署主文档](../../../../../serving/README_CN.md)
>> 执行 fastdeployserver 启动服务出现 "Address already in use", 请使用 `--grpc-port` 指定 grpc 端口号来启动服务，同时更改客户端示例中的请求端口号.
>> 其他启动参数可以使用 fastdeployserver --help 查看
服务启动成功后， 会有以下输出:
```
......
I0928 04:51:15.784517 206 grpc_server.cc:4117] Started GRPCInferenceService at 0.0.0.0:8001
I0928 04:51:15.785177 206 http_server.cc:2815] Started HTTPService at 0.0.0.0:8000
I0928 04:51:15.826578 206 http_server.cc:167] Started Metrics Service at 0.0.0.0:8002
```
## 客户端请求
在物理机器中执行以下命令，发送 grpc 请求并输出结果
```
#下载测试图片
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
#安装客户端依赖
python3 -m pip install tritonclient[all]
# 发送请求
python3 smoke_grpc_client.py
```
发送请求成功后，会返回 json 格式的检测结果并打印输出:
```
output_name: PERCEPTION_RESULT
scores: [0.8080164790153503, 0.03356542810797691, 0.03165825456380844, 0.020817330107092857, 0.018075695261359215, 0.017861749976873398, 0.016441335901618004, 0.01476177480071783, 0.012927377596497536, 0.012407636269927025, 0.012400650419294834, 0.012216777540743351, 0.01208423636853695, 0.011721019633114338, 0.011697308160364628, 0.011695655062794685, 0.011603309772908688, 0.011140472255647182, 0.010927721858024597, 0.01036786288022995, 0.00984608568251133, 0.009827949106693268, 0.009761993773281574, 0.00959752406924963, 0.009595031850039959, 0.009423951618373394, 0.008946355432271957, 0.008635037578642368, 0.008597995154559612, 0.008552121929824352, 0.00839947909116745, 0.008325068280100822, 0.00830004084855318, 0.00826205126941204, 0.008174785412847996, 0.008085251785814762, 0.008026468567550182, 0.00796759407967329, 0.007873599417507648, 0.007816540077328682, 0.007742374204099178, 0.007734378334134817, 0.0077047450467944145, 0.007684454321861267, 0.007525254040956497, 0.007521109655499458, 0.007519087754189968, 0.007399206515401602, 0.0071790567599236965, 0.0068892366252839565]
label_ids: [2, 0, 1, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
```
## 配置修改
当前默认配置在 CPU 上运行 Paddle 引擎， 如果要在 GPU 或其他推理引擎上运行。 需要修改 `models/runtime/config.pbtxt` 中配置，详情请参考[配置文档](../../../../../serving/docs/zh_CN/model_configuration.md)

110
examples/vision/perception/paddle3d/smoke/serving/models/postprocess/1/model.py Executable file
View File

@@ -0,0 +1,110 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import numpy as np
import time
import fastdeploy as fd
# triton_python_backend_utils is available in every Triton Python model. You
# need to use this module to create inference requests and responses. It also
# contains some utility functions for extracting information from model_config
# and converting Triton input/output types to numpy types.
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
"""Your Python model must use the same class name. Every Python model
that is created must have "TritonPythonModel" as the class name.
"""
def initialize(self, args):
"""`initialize` is called only once when the model is being loaded.
Implementing `initialize` function is optional. This function allows
the model to intialize any state associated with this model.
Parameters
----------
args : dict
Both keys and values are strings. The dictionary keys and values are:
* model_config: A JSON string containing the model configuration
* model_instance_kind: A string containing model instance kind
* model_instance_device_id: A string containing model instance device ID
* model_repository: Model repository path
* model_version: Model version
* model_name: Model name
"""
# You must parse model_config. JSON string is not parsed here
self.model_config = json.loads(args['model_config'])
print("model_config:", self.model_config)
self.input_names = []
for input_config in self.model_config["input"]:
self.input_names.append(input_config["name"])
print("postprocess input names:", self.input_names)
self.output_names = []
self.output_dtype = []
for output_config in self.model_config["output"]:
self.output_names.append(output_config["name"])
dtype = pb_utils.triton_string_to_numpy(output_config["data_type"])
self.output_dtype.append(dtype)
print("postprocess output names:", self.output_names)
self.postprocess_ = fd.vision.perception.SmokePostprocessor()
def execute(self, requests):
"""`execute` must be implemented in every Python model. `execute`
function receives a list of pb_utils.InferenceRequest as the only
argument. This function is called when an inference is requested
for this model. Depending on the batching configuration (e.g. Dynamic
Batching) used, `requests` may contain multiple requests. Every
Python model, must create one pb_utils.InferenceResponse for every
pb_utils.InferenceRequest in `requests`. If there is an error, you can
set the error argument when creating a pb_utils.InferenceResponse.
Parameters
----------
requests : list
A list of pb_utils.InferenceRequest
Returns
-------
list
A list of pb_utils.InferenceResponse. The length of this list must
be the same as `requests`
"""
responses = []
for request in requests:
infer_outputs = []
for name in self.input_names:
infer_output = pb_utils.get_input_tensor_by_name(request, name)
if infer_output:
infer_output = infer_output.as_numpy()
infer_outputs.append(infer_output)
results = self.postprocess_.run(infer_outputs)
r_str = fd.vision.utils.fd_result_to_json(results)
r_np = np.array(r_str, dtype=np.object_)
out_tensor = pb_utils.Tensor(self.output_names[0], r_np)
inference_response = pb_utils.InferenceResponse(
output_tensors=[out_tensor, ])
responses.append(inference_response)
return responses
def finalize(self):
"""`finalize` is called only once when the model is being unloaded.
Implementing `finalize` function is optional. This function allows
the model to perform any necessary clean ups before exit.
"""
print('Cleaning up...')

25
examples/vision/perception/paddle3d/smoke/serving/models/postprocess/config.pbtxt Executable file
View File

@@ -0,0 +1,25 @@
name: "postprocess"
backend: "python"
input [
{
name: "post_input1"
data_type: TYPE_FP32
dims: [ 50, 14 ]
}
]
output [
{
name: "post_output"
data_type: TYPE_STRING
dims: [ -1 ]
}
]
instance_group [
{
count: 1
kind: KIND_CPU
}
]

114
examples/vision/perception/paddle3d/smoke/serving/models/preprocess/1/model.py Executable file
View File

@@ -0,0 +1,114 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import numpy as np
import os
import fastdeploy as fd
# triton_python_backend_utils is available in every Triton Python model. You
# need to use this module to create inference requests and responses. It also
# contains some utility functions for extracting information from model_config
# and converting Triton input/output types to numpy types.
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
"""Your Python model must use the same class name. Every Python model
that is created must have "TritonPythonModel" as the class name.
"""
def initialize(self, args):
"""`initialize` is called only once when the model is being loaded.
Implementing `initialize` function is optional. This function allows
the model to intialize any state associated with this model.
Parameters
----------
args : dict
Both keys and values are strings. The dictionary keys and values are:
* model_config: A JSON string containing the model configuration
* model_instance_kind: A string containing model instance kind
* model_instance_device_id: A string containing model instance device ID
* model_repository: Model repository path
* model_version: Model version
* model_name: Model name
"""
# You must parse model_config. JSON string is not parsed here
self.model_config = json.loads(args['model_config'])
print("model_config:", self.model_config)
self.input_names = []
for input_config in self.model_config["input"]:
self.input_names.append(input_config["name"])
print("preprocess input names:", self.input_names)
self.output_names = []
self.output_dtype = []
for output_config in self.model_config["output"]:
self.output_names.append(output_config["name"])
# dtype = pb_utils.triton_string_to_numpy(output_config["data_type"])
# self.output_dtype.append(dtype)
self.output_dtype.append(output_config["data_type"])
print("preprocess output names:", self.output_names)
# init PaddleClasPreprocess class
yaml_path = os.path.abspath(os.path.dirname(
__file__)) + "/infer_cfg.yml"
self.preprocess_ = fd.vision.perception.SmokePreprocessor(yaml_path)
def execute(self, requests):
"""`execute` must be implemented in every Python model. `execute`
function receives a list of pb_utils.InferenceRequest as the only
argument. This function is called when an inference is requested
for this model. Depending on the batching configuration (e.g. Dynamic
Batching) used, `requests` may contain multiple requests. Every
Python model, must create one pb_utils.InferenceResponse for every
pb_utils.InferenceRequest in `requests`. If there is an error, you can
set the error argument when creating a pb_utils.InferenceResponse.
Parameters
----------
requests : list
A list of pb_utils.InferenceRequest
Returns
-------
list
A list of pb_utils.InferenceResponse. The length of this list must
be the same as `requests`
"""
responses = []
for request in requests:
data = pb_utils.get_input_tensor_by_name(request,
self.input_names[0])
data = data.as_numpy()
outputs = self.preprocess_.run(data)
output_tensors = []
for idx, name in enumerate(self.output_names):
dlpack_tensor = outputs[idx].to_dlpack()
output_tensor = pb_utils.Tensor.from_dlpack(name,
dlpack_tensor)
output_tensors.append(output_tensor)
inference_response = pb_utils.InferenceResponse(
output_tensors=output_tensors)
responses.append(inference_response)
return responses
def finalize(self):
"""`finalize` is called only once when the model is being unloaded.
Implementing `finalize` function is optional. This function allows
the model to perform any necessary clean ups before exit.
"""
print('Cleaning up...')

35
examples/vision/perception/paddle3d/smoke/serving/models/preprocess/config.pbtxt Executable file
View File

@@ -0,0 +1,35 @@
name: "preprocess"
backend: "python"
input [
{
name: "preprocess_input"
data_type: TYPE_UINT8
dims: [ 1, -1, -1, 3 ]
}
]
output [
{
name: "preprocess_output1"
data_type: TYPE_FP32
dims: [ 1, 2 ]
},
{
name: "preprocess_output2"
data_type: TYPE_FP32
dims: [ 1, 3, -1, -1 ]
},
{
name: "preprocess_output3"
data_type: TYPE_FP32
dims: [ 1, 3, 3 ]
}
]
instance_group [
{
count: 1
kind: KIND_CPU
}
]

5
examples/vision/perception/paddle3d/smoke/serving/models/runtime/1/README.md Executable file
View File

@@ -0,0 +1,5 @@
# Runtime Directory
This directory holds the model files.
Paddle models must be model.pdmodel and model.pdiparams files.
ONNX models must be model.onnx files.

57
examples/vision/perception/paddle3d/smoke/serving/models/runtime/config.pbtxt Executable file
View File

@@ -0,0 +1,57 @@
# optional, If name is specified it must match the name of the model repository directory containing the model.
name: "runtime"
backend: "fastdeploy"
# Input configuration of the model
input [
{
# input name
name: "down_ratios"
# input type such as TYPE_FP32、TYPE_UINT8、TYPE_INT8、TYPE_INT16、TYPE_INT32、TYPE_INT64、TYPE_FP16、TYPE_STRING
data_type: TYPE_FP32
dims: [ 1, 2 ]
},
{
# input name
name: "images"
# input type such as TYPE_FP32、TYPE_UINT8、TYPE_INT8、TYPE_INT16、TYPE_INT32、TYPE_INT64、TYPE_FP16、TYPE_STRING
data_type: TYPE_FP32
dims: [ 1, 3, -1, -1 ]
},
{
name: "trans_cam_to_img"
data_type: TYPE_FP32
dims: [ 1, 3, 3 ]
}
]
# The output of the model is configured in the same format as the input
output [
{
name: "concat_13.tmp_0"
data_type: TYPE_FP32
dims: [ 50, 14 ]
}
]
# Number of instances of the model
instance_group [
{
# The number of instances is 1
count: 1
# Use GPU, CPU inference option is:KIND_CPU
kind: KIND_CPU
# The instance is deployed on the 0th GPU card
# gpus: [0]
}
]
optimization {
execution_accelerators {
cpu_execution_accelerator : [ {
# use Paddle engine
name: "paddle",
parameters { key: "use_mkldnn" value: "1" }
}
]
}}

3
examples/vision/perception/paddle3d/smoke/serving/models/smoke/1/README.md Executable file
View File

@@ -0,0 +1,3 @@
# Smoke Pipeline
The pipeline directory does not have model files, but a version number directory needs to be maintained.

72
examples/vision/perception/paddle3d/smoke/serving/models/smoke/config.pbtxt Executable file
View File

@@ -0,0 +1,72 @@
platform: "ensemble"
input [
{
name: "INPUT"
data_type: TYPE_UINT8
dims: [ 1, -1, -1, 3 ]
}
]
output [
{
name: "PERCEPTION_RESULT"
data_type: TYPE_STRING
dims: [ -1 ]
}
]
ensemble_scheduling {
step [
{
model_name: "preprocess"
model_version: 1
input_map {
key: "preprocess_input"
value: "INPUT"
}
output_map {
key: "preprocess_output1"
value: "RUNTIME_INPUT1"
}
output_map {
key: "preprocess_output2"
value: "RUNTIME_INPUT2"
}
output_map {
key: "preprocess_output3"
value: "RUNTIME_INPUT3"
}
},
{
model_name: "runtime"
model_version: 1
input_map {
key: "down_ratios"
value: "RUNTIME_INPUT1"
}
input_map {
key: "images"
value: "RUNTIME_INPUT2"
}
input_map {
key: "trans_cam_to_img"
value: "RUNTIME_INPUT3"
}
output_map {
key: "concat_13.tmp_0"
value: "RUNTIME_OUTPUT1"
}
},
{
model_name: "postprocess"
model_version: 1
input_map {
key: "post_input1"
value: "RUNTIME_OUTPUT1"
}
output_map {
key: "post_output"
value: "PERCEPTION_RESULT"
}
}
]
}

108
examples/vision/perception/paddle3d/smoke/serving/smoke_grpc_client.py Executable file
View File

@@ -0,0 +1,108 @@
import logging
import numpy as np
import time
from typing import Optional
import cv2
import json
from tritonclient import utils as client_utils
from tritonclient.grpc import InferenceServerClient, InferInput, InferRequestedOutput, service_pb2_grpc, service_pb2
LOGGER = logging.getLogger("run_inference_on_triton")
class SyncGRPCTritonRunner:
DEFAULT_MAX_RESP_WAIT_S = 120
def __init__(
self,
server_url: str,
model_name: str,
model_version: str,
*,
verbose=False,
resp_wait_s: Optional[float]=None, ):
self._server_url = server_url
self._model_name = model_name
self._model_version = model_version
self._verbose = verbose
self._response_wait_t = self.DEFAULT_MAX_RESP_WAIT_S if resp_wait_s is None else resp_wait_s
self._client = InferenceServerClient(
self._server_url, verbose=self._verbose)
error = self._verify_triton_state(self._client)
if error:
raise RuntimeError(
f"Could not communicate to Triton Server: {error}")
LOGGER.debug(
f"Triton server {self._server_url} and model {self._model_name}:{self._model_version} "
f"are up and ready!")
model_config = self._client.get_model_config(self._model_name,
self._model_version)
model_metadata = self._client.get_model_metadata(self._model_name,
self._model_version)
LOGGER.info(f"Model config {model_config}")
LOGGER.info(f"Model metadata {model_metadata}")
for tm in model_metadata.inputs:
print("tm:", tm)
self._inputs = {tm.name: tm for tm in model_metadata.inputs}
self._input_names = list(self._inputs)
self._outputs = {tm.name: tm for tm in model_metadata.outputs}
self._output_names = list(self._outputs)
self._outputs_req = [
InferRequestedOutput(name) for name in self._outputs
]
def Run(self, inputs):
"""
Args:
inputs: list, Each value corresponds to an input name of self._input_names
Returns:
results: dict, {name : numpy.array}
"""
infer_inputs = []
for idx, data in enumerate(inputs):
infer_input = InferInput(self._input_names[idx], data.shape,
"UINT8")
infer_input.set_data_from_numpy(data)
infer_inputs.append(infer_input)
results = self._client.infer(
model_name=self._model_name,
model_version=self._model_version,
inputs=infer_inputs,
outputs=self._outputs_req,
client_timeout=self._response_wait_t, )
results = {name: results.as_numpy(name) for name in self._output_names}
return results
def _verify_triton_state(self, triton_client):
if not triton_client.is_server_live():
return f"Triton server {self._server_url} is not live"
elif not triton_client.is_server_ready():
return f"Triton server {self._server_url} is not ready"
elif not triton_client.is_model_ready(self._model_name,
self._model_version):
return f"Model {self._model_name}:{self._model_version} is not ready"
return None
if __name__ == "__main__":
model_name = "smoke"
model_version = "1"
url = "localhost:8001"
runner = SyncGRPCTritonRunner(url, model_name, model_version)
im = cv2.imread("smoke_test.png")
im = np.array([im, ])
for i in range(1):
result = runner.Run([im, ])
for name, values in result.items():
print("output_name:", name)
# values is batch
for value in values:
value = json.loads(value)
print("scores: ", value['scores'])
print("label_ids: ", value['label_ids'])