mirror of
https://github.com/PaddlePaddle/FastDeploy.git
synced 2025-10-06 09:07:10 +08:00
[Other] PPOCR models support model clone function (#1072)
* Refactor PaddleSeg with preprocessor && postprocessor * Fix bugs * Delete redundancy code * Modify by comments * Refactor according to comments * Add batch evaluation * Add single test script * Add ppliteseg single test script && fix eval(raise) error * fix bug * Fix evaluation segmentation.py batch predict * Fix segmentation evaluation bug * Fix evaluation segmentation bugs * Update segmentation result docs * Update old predict api and DisableNormalizeAndPermute * Update resize segmentation label map with cv::INTER_NEAREST * Add Model Clone function for PaddleClas && PaddleDet && PaddleSeg * Add multi thread demo * Add python model clone function * Add multi thread python && C++ example * Fix bug * Update python && cpp multi_thread examples * Add cpp && python directory * Add README.md for examples * Delete redundant code * Create README_CN.md * Rename README_CN.md to README.md * Update README.md * Update README.md * Update VERSION_NUMBER * Update requirements.txt * Update README.md * update version in doc: * [Serving]Update Dockerfile (#1037) Update Dockerfile * Add license notice for RVM onnx model file (#1060) * [Model] Add GPL-3.0 license (#1065) Add GPL-3.0 license * PPOCR model support model clone * Update README.md * Update PPOCRv2 && PPOCRv3 clone code * Update PPOCR python __init__ * Add multi thread ocr example code * Update README.md * Update README.md * Update ResNet50_vd_infer multi process code * Add PPOCR multi process && thread example * Update README.md * Update README.md * Update multi-thread docs Co-authored-by: Jason <jiangjiajun@baidu.com> Co-authored-by: leiqing <54695910+leiqing1@users.noreply.github.com> Co-authored-by: heliqi <1101791222@qq.com> Co-authored-by: WJJ1995 <wjjisloser@163.com>
This commit is contained in:
huangjianhui
51
tutorials/multi_thread/python/single_model/README.md
Normal file
51
tutorials/multi_thread/python/single_model/README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
English | [简体中文](README_CN.md)
|
||||
# Example of PaddleClas models Python multi-thread/multi-process Deployment
|
||||
|
||||
Before deployment, two steps require confirmation
|
||||
|
||||
- 1. Software and hardware should meet the requirements. Please refer to [FastDeploy Environment Requirements](../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
|
||||
- 2. Install the FastDeploy Python whl package. Please refer to [FastDeploy Python Installation](../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
|
||||
|
||||
This directory provides example file `multi_thread_process.py` to fast deploy multi-thread/multi-process ResNet50_vd on CPU/GPU and GPU accelerated by TensorRT. The script is as follows
|
||||
|
||||
|
||||
```bash
|
||||
# Download deployment example code
|
||||
git clone https://github.com/PaddlePaddle/FastDeploy.git
|
||||
cd FastDeploy/tutorials/multi_thread/python
|
||||
|
||||
# Download the ResNet50_vd model file and test images
|
||||
wget https://bj.bcebos.com/paddlehub/fastdeploy/ResNet50_vd_infer.tgz
|
||||
tar -xvf ResNet50_vd_infer.tgz
|
||||
wget https://gitee.com/paddlepaddle/PaddleClas/raw/release/2.4/deploy/images/ImageNet/ILSVRC2012_val_00000010.jpeg
|
||||
|
||||
|
||||
# CPU multi-thread inference
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device cpu --topk 1 --thread_num 1
|
||||
# CPU multi-process inference
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device cpu --topk 1 --use_multi_process True --process_num 1
|
||||
|
||||
# GPU multi-thread inference
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --topk 1 --thread_num 1
|
||||
# GPU multi-process inference
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --topk 1 --use_multi_process True --process_num 1
|
||||
|
||||
# Use TensorRT multi-thread inference on GPU （Attention: It is somewhat time-consuming for the operation of model serialization when running TensorRT inference for the first time. Please be patient.）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --use_trt True --topk 1 --thread_num 1
|
||||
# Use TensorRT multi-process inference on GPU （Attention: It is somewhat time-consuming for the operation of model serialization when running TensorRT inference for the first time. Please be patient.）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --use_trt True --topk 1 --use_multi_process True --process_num 1
|
||||
|
||||
# IPU multi-thread inference（Attention: It is somewhat time-consuming for the operation of model serialization when running IPU inference for the first time. Please be patient.）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device ipu --topk 1 --thread_num 1
|
||||
# IPU multi-process inference（Attention: It is somewhat time-consuming for the operation of model serialization when running IPU inference for the first time. Please be patient.）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device ipu --topk 1 --use_multi_process True --process_num 1
|
||||
```
|
||||
>> **Notice**: `--image_path` can be the path of the pictures folder
|
||||
|
||||
The result returned after running is as follows
|
||||
```bash
|
||||
ClassifyResult(
|
||||
label_ids: 153,
|
||||
scores: 0.686229,
|
||||
)
|
||||
```
|
||||
51
tutorials/multi_thread/python/single_model/README_CN.md
Normal file
51
tutorials/multi_thread/python/single_model/README_CN.md
Normal file
@@ -0,0 +1,51 @@
|
||||
[English](README.md) | 简体中文
|
||||
# PaddleClas模型 Python多线程/进程部署示例
|
||||
|
||||
在部署前，需确认以下两个步骤
|
||||
|
||||
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
|
||||
- 2. FastDeploy Python whl包安装，参考[FastDeploy Python安装](../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
|
||||
|
||||
本目录下提供`multi_thread_process.py`快速完成ResNet50_vd在CPU/GPU，以及GPU上通过TensorRT加速部署的多线程/进程示例。执行如下脚本即可完成
|
||||
|
||||
|
||||
```bash
|
||||
#下载部署示例代码
|
||||
git clone https://github.com/PaddlePaddle/FastDeploy.git
|
||||
cd FastDeploy/tutorials/multi_thread/python
|
||||
|
||||
# 下载ResNet50_vd模型文件和测试图片
|
||||
wget https://bj.bcebos.com/paddlehub/fastdeploy/ResNet50_vd_infer.tgz
|
||||
tar -xvf ResNet50_vd_infer.tgz
|
||||
wget https://gitee.com/paddlepaddle/PaddleClas/raw/release/2.4/deploy/images/ImageNet/ILSVRC2012_val_00000010.jpeg
|
||||
|
||||
|
||||
# CPU多线程推理
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device cpu --topk 1 --thread_num 1
|
||||
# CPU多进程推理
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device cpu --topk 1 --use_multi_process True --process_num 1
|
||||
|
||||
# GPU多线程推理
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --topk 1 --thread_num 1
|
||||
# GPU多进程推理
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --topk 1 --use_multi_process True --process_num 1
|
||||
|
||||
# GPU上使用TensorRT多线程推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --use_trt True --topk 1 --thread_num 1
|
||||
# GPU上使用TensorRT多进程推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device gpu --use_trt True --topk 1 --use_multi_process True --process_num 1
|
||||
|
||||
# IPU多线程推理（注意：IPU推理首次运行会有序列化模型的操作，有一定耗时，需要耐心等待）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device ipu --topk 1 --thread_num 1
|
||||
# IPU多进程推理（注意：IPU推理首次运行会有序列化模型的操作，有一定耗时，需要耐心等待）
|
||||
python multi_thread_process.py --model ResNet50_vd_infer --image_path ILSVRC2012_val_00000010.jpeg --device ipu --topk 1 --use_multi_process True --process_num 1
|
||||
```
|
||||
>> **注意**: `--image_path` 可以输入图片文件夹的路径
|
||||
|
||||
运行完成后返回结果如下所示
|
||||
```bash
|
||||
ClassifyResult(
|
||||
label_ids: 153,
|
||||
scores: 0.686229,
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,176 @@
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from threading import Thread
|
||||
import fastdeploy as fd
|
||||
import cv2
|
||||
import os
|
||||
import psutil
|
||||
from multiprocessing import Pool
|
||||
|
||||
|
||||
def parse_arguments():
|
||||
import argparse
|
||||
import ast
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--model", required=True, help="Path of PaddleClas model.")
|
||||
parser.add_argument(
|
||||
"--image_path",
|
||||
type=str,
|
||||
required=True,
|
||||
help="The directory or path or file list of the images to be predicted."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--topk", type=int, default=1, help="Return topk results.")
|
||||
parser.add_argument(
|
||||
"--device",
|
||||
type=str,
|
||||
default='cpu',
|
||||
help="Type of inference device, support 'cpu' or 'gpu' or 'ipu'.")
|
||||
parser.add_argument(
|
||||
"--use_trt",
|
||||
type=ast.literal_eval,
|
||||
default=False,
|
||||
help="Wether to use tensorrt.")
|
||||
parser.add_argument("--thread_num", type=int, default=1, help="thread num")
|
||||
parser.add_argument(
|
||||
"--use_multi_process",
|
||||
type=ast.literal_eval,
|
||||
default=False,
|
||||
help="Wether to use multi process.")
|
||||
parser.add_argument(
|
||||
"--process_num", type=int, default=1, help="process num")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def get_image_list(image_path):
|
||||
image_list = []
|
||||
if os.path.isfile(image_path):
|
||||
image_list.append(image_path)
|
||||
# load image in a directory
|
||||
elif os.path.isdir(image_path):
|
||||
for root, dirs, files in os.walk(image_path):
|
||||
for f in files:
|
||||
image_list.append(os.path.join(root, f))
|
||||
else:
|
||||
raise FileNotFoundError(
|
||||
'{} is not found. it should be a path of image, or a directory including images.'.
|
||||
format(image_path))
|
||||
|
||||
if len(image_list) == 0:
|
||||
raise RuntimeError(
|
||||
'There are not image file in `--image_path`={}'.format(image_path))
|
||||
|
||||
return image_list
|
||||
|
||||
|
||||
def build_option(args):
|
||||
option = fd.RuntimeOption()
|
||||
|
||||
if args.device.lower() == "gpu":
|
||||
option.use_paddle_backend()
|
||||
option.use_gpu()
|
||||
|
||||
if args.device.lower() == "ipu":
|
||||
option.use_ipu()
|
||||
|
||||
if args.use_trt:
|
||||
option.use_trt_backend()
|
||||
return option
|
||||
|
||||
|
||||
def load_model(args, runtime_option):
|
||||
model_file = os.path.join(args.model, "inference.pdmodel")
|
||||
params_file = os.path.join(args.model, "inference.pdiparams")
|
||||
config_file = os.path.join(args.model, "inference_cls.yaml")
|
||||
global model
|
||||
model = fd.vision.classification.PaddleClasModel(
|
||||
model_file, params_file, config_file, runtime_option=runtime_option)
|
||||
#return model
|
||||
|
||||
|
||||
def predict(model, img_list, topk):
|
||||
result_list = []
|
||||
# predict classification result
|
||||
for image in img_list:
|
||||
im = cv2.imread(image)
|
||||
result = model.predict(im, topk)
|
||||
result_list.append(result)
|
||||
return result_list
|
||||
|
||||
|
||||
def process_predict(image):
|
||||
# predict classification result
|
||||
im = cv2.imread(image)
|
||||
result = model.predict(im, args.topk)
|
||||
print(result)
|
||||
|
||||
|
||||
class WrapperThread(Thread):
|
||||
def __init__(self, func, args):
|
||||
super(WrapperThread, self).__init__()
|
||||
self.func = func
|
||||
self.args = args
|
||||
|
||||
def run(self):
|
||||
self.result = self.func(*self.args)
|
||||
|
||||
def get_result(self):
|
||||
return self.result
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
args = parse_arguments()
|
||||
|
||||
imgs_list = get_image_list(args.image_path)
|
||||
# configure runtime and load model
|
||||
runtime_option = build_option(args)
|
||||
|
||||
if args.use_multi_process:
|
||||
process_num = args.process_num
|
||||
with Pool(
|
||||
process_num,
|
||||
initializer=load_model,
|
||||
initargs=(args, runtime_option)) as pool:
|
||||
pool.map(process_predict, imgs_list)
|
||||
else:
|
||||
load_model(args, runtime_option)
|
||||
threads = []
|
||||
thread_num = args.thread_num
|
||||
image_num_each_thread = int(len(imgs_list) / thread_num)
|
||||
# unless you want independent model in each thread, actually model.clone()
|
||||
# is the same as model when creating thead because of the existence of
|
||||
# GIL(Global Interpreter Lock) in python. In addition, model.clone() will consume
|
||||
# additional memory to store independent member variables
|
||||
for i in range(thread_num):
|
||||
if i == thread_num - 1:
|
||||
t = WrapperThread(
|
||||
predict,
|
||||
args=(model.clone(), imgs_list[i * image_num_each_thread:],
|
||||
args.topk))
|
||||
else:
|
||||
t = WrapperThread(
|
||||
predict,
|
||||
args=(model.clone(), imgs_list[i * image_num_each_thread:(
|
||||
i + 1) * image_num_each_thread - 1], args.topk))
|
||||
threads.append(t)
|
||||
t.start()
|
||||
|
||||
for i in range(thread_num):
|
||||
threads[i].join()
|
||||
|
||||
for i in range(thread_num):
|
||||
for result in threads[i].get_result():
|
||||
print('thread:', i, ', result: ', result)
|
||||
Reference in New Issue
Block a user