apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-06 09:07:10 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

[Model] Modify SR (#674)

Browse Source 
* first commit for yolov7

* pybind for yolov7

* CPP README.md

* CPP README.md

* modified yolov7.cc

* README.md

* python file modify

* delete license in fastdeploy/

* repush the conflict part

* README.md modified

* README.md modified

* file path modified

* file path modified

* file path modified

* file path modified

* file path modified

* README modified

* README modified

* move some helpers to private

* add examples for yolov7

* api.md modified

* api.md modified

* api.md modified

* YOLOv7

* yolov7 release link

* yolov7 release link

* yolov7 release link

* copyright

* change some helpers to private

* change variables to const and fix documents.

* gitignore

* Transfer some funtions to private member of class

* Transfer some funtions to private member of class

* Merge from develop (#9)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <jiangjiajun@baidu.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>

* first commit for yolor

* for merge

* Develop (#11)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <jiangjiajun@baidu.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>

* Yolor (#16)

* Develop (#11) (#12)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <jiangjiajun@baidu.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>

Co-authored-by: Jason <jiangjiajun@baidu.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>

* Develop (#13)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <jiangjiajun@baidu.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* Develop (#14)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>

Co-authored-by: Jason <jiangjiajun@baidu.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>
Co-authored-by: Jason <928090362@qq.com>

* add is_dynamic for YOLO series (#22)

* modify ppmatting backend and docs

* modify ppmatting docs

* fix the PPMatting size problem

* fix LimitShort's log

* retrigger ci

* modify PPMatting docs

* modify the way  for dealing with  LimitShort

* add python comments for external models

* modify resnet c++ comments

* modify C++ comments for external models

* modify python comments and add result class comments

* fix comments compile error

* modify result.h comments

* modify examples doc and code for SR models

* code style

* retrigger ci

* python file code style

* fix examples links

* fix examples links

* fix examples links

Co-authored-by: Jason <jiangjiajun@baidu.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming3.bjyz.baidu.com>
Co-authored-by: DefTruth <31974251+DefTruth@users.noreply.github.com>
Co-authored-by: huangjianhui <852142024@qq.com>
Co-authored-by: Jason <928090362@qq.com>
This commit is contained in:
ziqi-jin
2022-11-25 18:31:22 +08:00
committed by GitHub
parent 86f05e9ac8
commit ad5c9c08b2
24 changed files with 417 additions and 404 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
examples/vision/sr/basicvsr/README.md
View File

@@ -18,7 +18,7 @@
| 模型 | 参数大小 | 精度 | 备注 | | 模型 | 参数大小 | 精度 | 备注 |
|:----------------------------------------------------------------------------|:-------|:----- | :------ | |:----------------------------------------------------------------------------|:-------|:----- | :------ |
| [BasicVSR](https://bj.bcebos.com/paddlehub/fastdeploy/BasicVSR_reds_x4.tgz) | 30.1MB | - | | [BasicVSR](https://bj.bcebos.com/paddlehub/fastdeploy/BasicVSR_reds_x4.tar) | 30.1MB | - |
**注意**：非常不建议在没有独立显卡的设备上运行该模型 **注意**：非常不建议在没有独立显卡的设备上运行该模型

141
examples/vision/sr/basicvsr/cpp/infer.cc
View File

@@ -20,8 +20,8 @@ const char sep = '\\';
const char sep = '/'; const char sep = '/';
#endif #endif
void CpuInfer(const std::string& model_dir, void CpuInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto model = fastdeploy::vision::sr::BasicVSR(model_file, params_file); auto model = fastdeploy::vision::sr::BasicVSR(model_file, params_file);
@@ -32,13 +32,13 @@ void CpuInfer(const std::string& model_dir,
} }
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
@@ -46,20 +46,22 @@ void CpuInfer(const std::string& model_dir,
// it used 4.x version of opencv below // it used 4.x version of opencv below
// notice your opencv version and method of api. // notice your opencv version and method of api.
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -67,27 +69,22 @@ void CpuInfer(const std::string& model_dir,
cv::Mat frame; cv::Mat frame;
int frame_id = 0; int frame_id = 0;
bool reach_end = false; bool reach_end = false;
while (capture.isOpened()) while (capture.isOpened()) {
{
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
for (int i = 0; i < frame_num; i++) for (int i = 0; i < frame_num; i++) {
{
capture.read(frame); capture.read(frame);
if (!frame.empty()) if (!frame.empty()) {
{
imgs.push_back(frame); imgs.push_back(frame);
} else { } else {
reach_end = true; reach_end = true;
} }
} }
if (reach_end) if (reach_end) {
{
break; break;
} }
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
@@ -95,20 +92,21 @@ void CpuInfer(const std::string& model_dir,
frame_id += 1; frame_id += 1;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
void GpuInfer(const std::string& model_dir, void GpuInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto option = fastdeploy::RuntimeOption(); auto option = fastdeploy::RuntimeOption();
option.UseGpu(); option.UseGpu();
auto model = fastdeploy::vision::sr::BasicVSR( auto model =
model_file, params_file, option); fastdeploy::vision::sr::BasicVSR(model_file, params_file, option);
if (!model.Initialized()) { if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl; std::cerr << "Failed to initialize." << std::endl;
@@ -116,32 +114,34 @@ void GpuInfer(const std::string& model_dir,
} }
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
// Get Video info :fps, frame count // Get Video info :fps, frame count
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -149,27 +149,22 @@ void GpuInfer(const std::string& model_dir,
cv::Mat frame; cv::Mat frame;
int frame_id = 0; int frame_id = 0;
bool reach_end = false; bool reach_end = false;
while (capture.isOpened()) while (capture.isOpened()) {
{
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
for (int i = 0; i < frame_num; i++) for (int i = 0; i < frame_num; i++) {
{
capture.read(frame); capture.read(frame);
if (!frame.empty()) if (!frame.empty()) {
{
imgs.push_back(frame); imgs.push_back(frame);
} else { } else {
reach_end = true; reach_end = true;
} }
} }
if (reach_end) if (reach_end) {
{
break; break;
} }
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
@@ -177,22 +172,25 @@ void GpuInfer(const std::string& model_dir,
frame_id += 1; frame_id += 1;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
void TrtInfer(const std::string& model_dir, void TrtInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto option = fastdeploy::RuntimeOption(); auto option = fastdeploy::RuntimeOption();
option.UseGpu(); option.UseGpu();
option.UseTrtBackend();
// use paddle-TRT // use paddle-TRT
option.UseTrtBackend();
option.EnablePaddleTrtCollectShape();
option.SetTrtInputShape("lrs", {1, 2, 3, 180, 320});
option.EnablePaddleToTrt(); option.EnablePaddleToTrt();
auto model = fastdeploy::vision::sr::BasicVSR( auto model =
model_file, params_file, option); fastdeploy::vision::sr::BasicVSR(model_file, params_file, option);
if (!model.Initialized()) { if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl; std::cerr << "Failed to initialize." << std::endl;
@@ -201,35 +199,38 @@ void TrtInfer(const std::string& model_dir,
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
// Get Video info :fps, frame count // Get Video info :fps, frame count
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
//Note that the resolution between the size and the original input is consistent when the model is exported, // Note that the resolution between the size and the original input is
// consistent when the model is exported,
// for example: [1,2,3,180,320], after 4x super separation [1,2,3,720,1080]. // for example: [1,2,3,180,320], after 4x super separation [1,2,3,720,1080].
// Therefore, it is very important to derive the model // Therefore, it is very important to derive the model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -237,27 +238,22 @@ void TrtInfer(const std::string& model_dir,
cv::Mat frame; cv::Mat frame;
int frame_id = 0; int frame_id = 0;
bool reach_end = false; bool reach_end = false;
while (capture.isOpened()) while (capture.isOpened()) {
{
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
for (int i = 0; i < frame_num; i++) for (int i = 0; i < frame_num; i++) {
{
capture.read(frame); capture.read(frame);
if (!frame.empty()) if (!frame.empty()) {
{
imgs.push_back(frame); imgs.push_back(frame);
} else { } else {
reach_end = true; reach_end = true;
} }
} }
if (reach_end) if (reach_end) {
{
break; break;
} }
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
@@ -265,16 +261,17 @@ void TrtInfer(const std::string& model_dir,
frame_id += 1; frame_id += 1;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
int main(int argc, char* argv[]) { int main(int argc, char* argv[]) {
if (argc < 4) { if (argc < 4) {
std::cout std::cout << "Usage: infer_demo path/to/model_dir path/to/video frame "
<< "Usage: infer_demo path/to/model_dir path/to/video frame number run_option, " "number run_option, "
"e.g ./infer_model ./vsr_model_dir ./person.mp4 0 2" "e.g ./infer_model ./vsr_model_dir ./vsr_src.mp4 0 2"
<< std::endl; << std::endl;
std::cout << "The data type of run_option is int, 0: run with cpu; 1: run " std::cout << "The data type of run_option is int, 0: run with cpu; 1: run "
"with gpu; 2: run with gpu and use tensorrt backend." "with gpu; 2: run with gpu and use tensorrt backend."

6
examples/vision/sr/basicvsr/python/README.md
View File

@@ -17,11 +17,11 @@ wget https://bj.bcebos.com/paddlehub/fastdeploy/BasicVSR_reds_x4.tar
tar -xvf BasicVSR_reds_x4.tar tar -xvf BasicVSR_reds_x4.tar
wget https://bj.bcebos.com/paddlehub/fastdeploy/vsr_src.mp4 wget https://bj.bcebos.com/paddlehub/fastdeploy/vsr_src.mp4
# CPU推理 # CPU推理
python infer.py --model BasicVSR_reds_x4 --video person.mp4 --frame_num 2 --device cpu python infer.py --model BasicVSR_reds_x4 --video vsr_src.mp4 --frame_num 2 --device cpu
# GPU推理 # GPU推理
python infer.py --model BasicVSR_reds_x4 --video person.mp4 --frame_num 2 --device gpu python infer.py --model BasicVSR_reds_x4 --video vsr_src.mp4 --frame_num 2 --device gpu
# GPU上使用TensorRT推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待） # GPU上使用TensorRT推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待）
python infer.py --model BasicVSR_reds_x4 --video person.mp4 --frame_num 2 --device gpu --use_trt True python infer.py --model BasicVSR_reds_x4 --video vsr_src.mp4 --frame_num 2 --device gpu --use_trt True
``` ```
## BasicVSR Python接口 ## BasicVSR Python接口

2
examples/vision/sr/basicvsr/python/infer.py
View File

@@ -30,6 +30,8 @@ def build_option(args):
option.use_gpu() option.use_gpu()
if args.use_trt: if args.use_trt:
option.use_trt_backend() option.use_trt_backend()
option.enable_paddle_trt_collect_shape()
option.set_trt_input_shape("lrs", [1, 2, 3, 180, 320])
option.enable_paddle_to_trt() option.enable_paddle_to_trt()
return option return option

2
examples/vision/sr/edvr/README.md
View File

@@ -18,7 +18,7 @@
| 模型 | 参数大小 | 精度 | 备注 | | 模型 | 参数大小 | 精度 | 备注 |
|:--------------------------------------------------------------------------------|:-------|:----- | :------ | |:--------------------------------------------------------------------------------|:-------|:----- | :------ |
| [EDVR](https://bj.bcebos.com/paddlehub/fastdeploy/EDVR_M_wo_tsa_SRx4.tgz) | 14.9MB | - | | [EDVR](https://bj.bcebos.com/paddlehub/fastdeploy/EDVR_M_wo_tsa_SRx4.tar) | 14.9MB | - |
**注意**：非常不建议在没有独立显卡的设备上运行该模型 **注意**：非常不建议在没有独立显卡的设备上运行该模型

6
examples/vision/sr/edvr/cpp/README.md
View File

@@ -25,11 +25,11 @@ wget https://bj.bcebos.com/paddlehub/fastdeploy/vsr_src.mp4
# CPU推理 # CPU推理
./infer_demo EDVR_M_wo_tsa_SRx4 vsr_src.mp4 0 2 ./infer_demo EDVR_M_wo_tsa_SRx4 vsr_src.mp4 0 5
# GPU推理 # GPU推理
./infer_demo EDVR_M_wo_tsa_SRx4 vsr_src.mp4 1 2 ./infer_demo EDVR_M_wo_tsa_SRx4 vsr_src.mp4 1 5
# GPU上TensorRT推理 # GPU上TensorRT推理
./infer_demo EDVR_M_wo_tsa_SRx4 vsr_src.mp4 2 2 ./infer_demo EDVR_M_wo_tsa_SRx4 vsr_src.mp4 2 5
``` ```
以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考: 以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考:

112
examples/vision/sr/edvr/cpp/infer.cc
View File

@@ -20,8 +20,8 @@ const char sep = '\\';
const char sep = '/'; const char sep = '/';
#endif #endif
void CpuInfer(const std::string& model_dir, void CpuInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto model = fastdeploy::vision::sr::EDVR(model_file, params_file); auto model = fastdeploy::vision::sr::EDVR(model_file, params_file);
@@ -32,13 +32,13 @@ void CpuInfer(const std::string& model_dir,
} }
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
@@ -46,20 +46,22 @@ void CpuInfer(const std::string& model_dir,
// it used 4.x version of opencv below // it used 4.x version of opencv below
// notice your opencv version and method of api. // notice your opencv version and method of api.
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -68,8 +70,7 @@ void CpuInfer(const std::string& model_dir,
int frame_id = 0; int frame_id = 0;
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
while (capture.read(frame)) { while (capture.read(frame)) {
if (!frame.empty()) if (!frame.empty()) {
{
if (frame_id < frame_num) { if (frame_id < frame_num) {
imgs.push_back(frame); imgs.push_back(frame);
frame_id++; frame_id++;
@@ -81,28 +82,27 @@ void CpuInfer(const std::string& model_dir,
frame_id++; frame_id++;
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
std::cout << "Processing frame: " << frame_id << std::endl; std::cout << "Processing frame: " << frame_id << std::endl;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
void GpuInfer(const std::string& model_dir, void GpuInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto option = fastdeploy::RuntimeOption(); auto option = fastdeploy::RuntimeOption();
option.UseGpu(); option.UseGpu();
auto model = fastdeploy::vision::sr::EDVR( auto model = fastdeploy::vision::sr::EDVR(model_file, params_file, option);
model_file, params_file, option);
if (!model.Initialized()) { if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl; std::cerr << "Failed to initialize." << std::endl;
@@ -110,32 +110,34 @@ void GpuInfer(const std::string& model_dir,
} }
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
// Get Video info :fps, frame count // Get Video info :fps, frame count
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -144,8 +146,7 @@ void GpuInfer(const std::string& model_dir,
int frame_id = 0; int frame_id = 0;
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
while (capture.read(frame)) { while (capture.read(frame)) {
if (!frame.empty()) if (!frame.empty()) {
{
if (frame_id < frame_num) { if (frame_id < frame_num) {
imgs.push_back(frame); imgs.push_back(frame);
frame_id++; frame_id++;
@@ -157,30 +158,31 @@ void GpuInfer(const std::string& model_dir,
frame_id++; frame_id++;
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
std::cout << "Processing frame: " << frame_id << std::endl; std::cout << "Processing frame: " << frame_id << std::endl;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
void TrtInfer(const std::string& model_dir, void TrtInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto option = fastdeploy::RuntimeOption(); auto option = fastdeploy::RuntimeOption();
option.UseGpu(); option.UseGpu();
option.UseTrtBackend();
// use paddle-TRT // use paddle-TRT
option.UseTrtBackend();
option.EnablePaddleTrtCollectShape();
option.SetTrtInputShape("x", {1, 5, 3, 180, 320});
option.EnablePaddleToTrt(); option.EnablePaddleToTrt();
auto model = fastdeploy::vision::sr::EDVR( auto model = fastdeploy::vision::sr::EDVR(model_file, params_file, option);
model_file, params_file, option);
if (!model.Initialized()) { if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl; std::cerr << "Failed to initialize." << std::endl;
@@ -189,35 +191,38 @@ void TrtInfer(const std::string& model_dir,
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
// Get Video info :fps, frame count // Get Video info :fps, frame count
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
//Note that the resolution between the size and the original input is consistent when the model is exported, // Note that the resolution between the size and the original input is
// consistent when the model is exported,
// for example: [1,2,3,180,320], after 4x super separation [1,2,3,720,1080]. // for example: [1,2,3,180,320], after 4x super separation [1,2,3,720,1080].
// Therefore, it is very important to derive the model // Therefore, it is very important to derive the model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -226,8 +231,7 @@ void TrtInfer(const std::string& model_dir,
int frame_id = 0; int frame_id = 0;
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
while (capture.read(frame)) { while (capture.read(frame)) {
if (!frame.empty()) if (!frame.empty()) {
{
if (frame_id < frame_num) { if (frame_id < frame_num) {
imgs.push_back(frame); imgs.push_back(frame);
frame_id++; frame_id++;
@@ -239,24 +243,24 @@ void TrtInfer(const std::string& model_dir,
frame_id++; frame_id++;
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
std::cout << "Processing frame: " << frame_id << std::endl; std::cout << "Processing frame: " << frame_id << std::endl;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
int main(int argc, char* argv[]) { int main(int argc, char* argv[]) {
if (argc < 4) { if (argc < 4) {
std::cout std::cout << "Usage: infer_demo path/to/model_dir path/to/video frame "
<< "Usage: infer_demo path/to/model_dir path/to/video frame number run_option, " "number run_option, "
"e.g ./infer_model ./vsr_model_dir ./person.mp4 0 2" "e.g ./infer_model ./vsr_model_dir ./vsr_src.mp4 0 5"
<< std::endl; << std::endl;
std::cout << "The data type of run_option is int, 0: run with cpu; 1: run " std::cout << "The data type of run_option is int, 0: run with cpu; 1: run "
"with gpu; 2: run with gpu and use tensorrt backend." "with gpu; 2: run with gpu and use tensorrt backend."

6
examples/vision/sr/edvr/python/README.md
View File

@@ -17,11 +17,11 @@ wget https://bj.bcebos.com/paddlehub/fastdeploy/EDVR_M_wo_tsa_SRx4.tar
tar -xvf EDVR_M_wo_tsa_SRx4.tar tar -xvf EDVR_M_wo_tsa_SRx4.tar
wget https://bj.bcebos.com/paddlehub/fastdeploy/vsr_src.mp4 wget https://bj.bcebos.com/paddlehub/fastdeploy/vsr_src.mp4
# CPU推理 # CPU推理
python infer.py --model EDVR_M_wo_tsa_SRx4 --video person.mp4 --frame_num 2 --device cpu python infer.py --model EDVR_M_wo_tsa_SRx4 --video vsr_src.mp4 --frame_num 5 --device cpu
# GPU推理 # GPU推理
python infer.py --model EDVR_M_wo_tsa_SRx4 --video person.mp4 --frame_num 2 --device gpu python infer.py --model EDVR_M_wo_tsa_SRx4 --video vsr_src.mp4 --frame_num 5 --device gpu
# GPU上使用TensorRT推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待） # GPU上使用TensorRT推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待）
python infer.py --model EDVR_M_wo_tsa_SRx4 --video person.mp4 --frame_num 2 --device gpu --use_trt True python infer.py --model EDVR_M_wo_tsa_SRx4 --video vsr_src.mp4 --frame_num 5 --device gpu --use_trt True
``` ```
## EDVR Python接口 ## EDVR Python接口

2
examples/vision/sr/edvr/python/infer.py
View File

@@ -30,6 +30,8 @@ def build_option(args):
option.use_gpu() option.use_gpu()
if args.use_trt: if args.use_trt:
option.use_trt_backend() option.use_trt_backend()
option.enable_paddle_trt_collect_shape()
option.set_trt_input_shape("x", [1, 5, 3, 180, 320])
option.enable_paddle_to_trt() option.enable_paddle_to_trt()
return option return option

2
examples/vision/sr/ppmsvsr/README.md
View File

@@ -18,7 +18,7 @@
| 模型 | 参数大小 | 精度 | 备注 | | 模型 | 参数大小 | 精度 | 备注 |
|:----------------------------------------------------------------------------|:------|:----- | :------ | |:----------------------------------------------------------------------------|:------|:----- | :------ |
| [PP-MSVSR](https://bj.bcebos.com/paddlehub/fastdeploy/PP-MSVSR_reds_x4.tgz) | 8.8MB | - | | [PP-MSVSR](https://bj.bcebos.com/paddlehub/fastdeploy/PP-MSVSR_reds_x4.tar) | 8.8MB | - |
## 详细部署文档 ## 详细部署文档

140
examples/vision/sr/ppmsvsr/cpp/infer.cc
View File

@@ -20,8 +20,8 @@ const char sep = '\\';
const char sep = '/'; const char sep = '/';
#endif #endif
void CpuInfer(const std::string& model_dir, void CpuInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto model = fastdeploy::vision::sr::PPMSVSR(model_file, params_file); auto model = fastdeploy::vision::sr::PPMSVSR(model_file, params_file);
@@ -32,13 +32,13 @@ void CpuInfer(const std::string& model_dir,
} }
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
@@ -46,20 +46,22 @@ void CpuInfer(const std::string& model_dir,
// it used 4.x version of opencv below // it used 4.x version of opencv below
// notice your opencv version and method of api. // notice your opencv version and method of api.
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -67,27 +69,22 @@ void CpuInfer(const std::string& model_dir,
cv::Mat frame; cv::Mat frame;
int frame_id = 0; int frame_id = 0;
bool reach_end = false; bool reach_end = false;
while (capture.isOpened()) while (capture.isOpened()) {
{
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
for (int i = 0; i < frame_num; i++) for (int i = 0; i < frame_num; i++) {
{
capture.read(frame); capture.read(frame);
if (!frame.empty()) if (!frame.empty()) {
{
imgs.push_back(frame); imgs.push_back(frame);
} else { } else {
reach_end = true; reach_end = true;
} }
} }
if (reach_end) if (reach_end) {
{
break; break;
} }
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
@@ -95,23 +92,21 @@ void CpuInfer(const std::string& model_dir,
frame_id += 1; frame_id += 1;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
void GpuInfer(const std::string& model_dir, void GpuInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto option = fastdeploy::RuntimeOption(); auto option = fastdeploy::RuntimeOption();
// use paddle-TRT // use paddle-TRT
option.UseGpu(); option.UseGpu();
option.UseTrtBackend(); auto model = fastdeploy::vision::sr::PPMSVSR(model_file, params_file, option);
option.EnablePaddleToTrt();
auto model = fastdeploy::vision::sr::PPMSVSR(
model_file, params_file, option);
if (!model.Initialized()) { if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl; std::cerr << "Failed to initialize." << std::endl;
@@ -119,32 +114,34 @@ void GpuInfer(const std::string& model_dir,
} }
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
// Get Video info :fps, frame count // Get Video info :fps, frame count
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -152,27 +149,22 @@ void GpuInfer(const std::string& model_dir,
cv::Mat frame; cv::Mat frame;
int frame_id = 0; int frame_id = 0;
bool reach_end = false; bool reach_end = false;
while (capture.isOpened()) while (capture.isOpened()) {
{
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
for (int i = 0; i < frame_num; i++) for (int i = 0; i < frame_num; i++) {
{
capture.read(frame); capture.read(frame);
if (!frame.empty()) if (!frame.empty()) {
{
imgs.push_back(frame); imgs.push_back(frame);
} else { } else {
reach_end = true; reach_end = true;
} }
} }
if (reach_end) if (reach_end) {
{
break; break;
} }
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
@@ -180,20 +172,23 @@ void GpuInfer(const std::string& model_dir,
frame_id += 1; frame_id += 1;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
void TrtInfer(const std::string& model_dir, void TrtInfer(const std::string& model_dir, const std::string& video_file,
const std::string& video_file, int frame_num) { int frame_num) {
auto model_file = model_dir + sep + "model.pdmodel"; auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams"; auto params_file = model_dir + sep + "model.pdiparams";
auto option = fastdeploy::RuntimeOption(); auto option = fastdeploy::RuntimeOption();
option.UseGpu(); option.UseGpu();
option.UseTrtBackend(); option.UseTrtBackend();
auto model = fastdeploy::vision::sr::PPMSVSR( option.EnablePaddleTrtCollectShape();
model_file, params_file, option); option.SetTrtInputShape("lqs", {1, 2, 3, 180, 320});
option.EnablePaddleToTrt();
auto model = fastdeploy::vision::sr::PPMSVSR(model_file, params_file, option);
if (!model.Initialized()) { if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl; std::cerr << "Failed to initialize." << std::endl;
@@ -202,35 +197,38 @@ void TrtInfer(const std::string& model_dir,
// note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default)) // note: input/output shape is [b, n, c, h, w] (n = frame_nums; b=1(default))
// b and n is dependent on export model shape // b and n is dependent on export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
cv::VideoCapture capture; cv::VideoCapture capture;
// change your save video path // change your save video path
std::string video_out_name = "output.mp4"; std::string video_out_name = "output.mp4";
capture.open(video_file); capture.open(video_file);
if (!capture.isOpened()) if (!capture.isOpened()) {
{
std::cout << "can not open video " << std::endl; std::cout << "can not open video " << std::endl;
return; return;
} }
// Get Video info :fps, frame count // Get Video info :fps, frame count
int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS)); int video_fps = static_cast<int>(capture.get(cv::CAP_PROP_FPS));
int video_frame_count = static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT)); int video_frame_count =
static_cast<int>(capture.get(cv::CAP_PROP_FRAME_COUNT));
// Set fixed size for output frame, only for msvsr model // Set fixed size for output frame, only for msvsr model
//Note that the resolution between the size and the original input is consistent when the model is exported, // Note that the resolution between the size and the original input is
// consistent when the model is exported,
// for example: [1,2,3,180,320], after 4x super separation [1,2,3,720,1080]. // for example: [1,2,3,180,320], after 4x super separation [1,2,3,720,1080].
// Therefore, it is very important to derive the model // Therefore, it is very important to derive the model
int out_width = 1280; int out_width = 1280;
int out_height = 720; int out_height = 720;
std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count << std::endl; std::cout << "fps: " << video_fps << "\tframe_count: " << video_frame_count
<< std::endl;
// Create VideoWriter for output // Create VideoWriter for output
cv::VideoWriter video_out; cv::VideoWriter video_out;
std::string video_out_path("./"); std::string video_out_path("./");
video_out_path += video_out_name; video_out_path += video_out_name;
int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v'); int fcc = cv::VideoWriter::fourcc('m', 'p', '4', 'v');
video_out.open(video_out_path, fcc, video_fps, cv::Size(out_width, out_height), true); video_out.open(video_out_path, fcc, video_fps,
if (!video_out.isOpened()) cv::Size(out_width, out_height), true);
{ if (!video_out.isOpened()) {
std::cout << "create video writer failed!" << std::endl; std::cout << "create video writer failed!" << std::endl;
return; return;
} }
@@ -238,27 +236,22 @@ void TrtInfer(const std::string& model_dir,
cv::Mat frame; cv::Mat frame;
int frame_id = 0; int frame_id = 0;
bool reach_end = false; bool reach_end = false;
while (capture.isOpened()) while (capture.isOpened()) {
{
std::vector<cv::Mat> imgs; std::vector<cv::Mat> imgs;
for (int i = 0; i < frame_num; i++) for (int i = 0; i < frame_num; i++) {
{
capture.read(frame); capture.read(frame);
if (!frame.empty()) if (!frame.empty()) {
{
imgs.push_back(frame); imgs.push_back(frame);
} else { } else {
reach_end = true; reach_end = true;
} }
} }
if (reach_end) if (reach_end) {
{
break; break;
} }
std::vector<cv::Mat> results; std::vector<cv::Mat> results;
model.Predict(imgs, results); model.Predict(imgs, results);
for (auto &item : results) for (auto& item : results) {
{
// cv::imshow("13",item); // cv::imshow("13",item);
// cv::waitKey(30); // cv::waitKey(30);
video_out.write(item); video_out.write(item);
@@ -266,16 +259,17 @@ void TrtInfer(const std::string& model_dir,
frame_id += 1; frame_id += 1;
} }
} }
std::cout << "inference finished, output video saved at " << video_out_path << std::endl; std::cout << "inference finished, output video saved at " << video_out_path
<< std::endl;
capture.release(); capture.release();
video_out.release(); video_out.release();
} }
int main(int argc, char* argv[]) { int main(int argc, char* argv[]) {
if (argc < 4) { if (argc < 4) {
std::cout std::cout << "Usage: infer_demo path/to/model_dir path/to/video frame "
<< "Usage: infer_demo path/to/model_dir path/to/video frame number run_option, " "number run_option, "
"e.g ./infer_model ./vsr_model_dir ./person.mp4 0 2" "e.g ./infer_model ./vsr_model_dir ./vsr_src.mp4 0 2"
<< std::endl; << std::endl;
std::cout << "The data type of run_option is int, 0: run with cpu; 1: run " std::cout << "The data type of run_option is int, 0: run with cpu; 1: run "
"with gpu; 2: run with gpu and use tensorrt backend." "with gpu; 2: run with gpu and use tensorrt backend."

6
examples/vision/sr/ppmsvsr/python/README.md
View File

@@ -17,11 +17,11 @@ wget https://bj.bcebos.com/paddlehub/fastdeploy/PP-MSVSR_reds_x4.tar
tar -xvf PP-MSVSR_reds_x4.tar tar -xvf PP-MSVSR_reds_x4.tar
wget https://bj.bcebos.com/paddlehub/fastdeploy/vsr_src.mp4 wget https://bj.bcebos.com/paddlehub/fastdeploy/vsr_src.mp4
# CPU推理 # CPU推理
python infer.py --model PP-MSVSR_reds_x4 --video person.mp4 --frame_num 2 --device cpu python infer.py --model PP-MSVSR_reds_x4 --video vsr_src.mp4 --frame_num 2 --device cpu
# GPU推理 # GPU推理
python infer.py --model PP-MSVSR_reds_x4 --video person.mp4 --frame_num 2 --device gpu python infer.py --model PP-MSVSR_reds_x4 --video vsr_src.mp4 --frame_num 2 --device gpu
# GPU上使用TensorRT推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待） # GPU上使用TensorRT推理 （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待）
python infer.py --model PP-MSVSR_reds_x4 --video person.mp4 --frame_num 2 --device gpu --use_trt True python infer.py --model PP-MSVSR_reds_x4 --video vsr_src.mp4 --frame_num 2 --device gpu --use_trt True
``` ```
## VSR Python接口 ## VSR Python接口

2
examples/vision/sr/ppmsvsr/python/infer.py
View File

@@ -30,6 +30,8 @@ def build_option(args):
option.use_gpu() option.use_gpu()
if args.use_trt: if args.use_trt:
option.use_trt_backend() option.use_trt_backend()
option.enable_paddle_trt_collect_shape()
option.set_trt_input_shape("lqs", [1, 2, 3, 180, 320])
option.enable_paddle_to_trt() option.enable_paddle_to_trt()
return option return option

15
fastdeploy/vision.h Executable file → Normal file
View File

@@ -15,9 +15,9 @@
#include "fastdeploy/core/config.h" #include "fastdeploy/core/config.h"
#ifdef ENABLE_VISION #ifdef ENABLE_VISION
#include "fastdeploy/vision/classification/contrib/resnet.h"
#include "fastdeploy/vision/classification/contrib/yolov5cls.h" #include "fastdeploy/vision/classification/contrib/yolov5cls.h"
#include "fastdeploy/vision/classification/ppcls/model.h" #include "fastdeploy/vision/classification/ppcls/model.h"
#include "fastdeploy/vision/classification/contrib/resnet.h"
#include "fastdeploy/vision/detection/contrib/nanodet_plus.h" #include "fastdeploy/vision/detection/contrib/nanodet_plus.h"
#include "fastdeploy/vision/detection/contrib/scaledyolov4.h" #include "fastdeploy/vision/detection/contrib/scaledyolov4.h"
#include "fastdeploy/vision/detection/contrib/yolor.h" #include "fastdeploy/vision/detection/contrib/yolor.h"
@@ -29,33 +29,34 @@
#include "fastdeploy/vision/detection/contrib/yolov7end2end_trt.h" #include "fastdeploy/vision/detection/contrib/yolov7end2end_trt.h"
#include "fastdeploy/vision/detection/contrib/yolox.h" #include "fastdeploy/vision/detection/contrib/yolox.h"
#include "fastdeploy/vision/detection/ppdet/model.h" #include "fastdeploy/vision/detection/ppdet/model.h"
#include "fastdeploy/vision/facealign/contrib/face_landmark_1000.h"
#include "fastdeploy/vision/facealign/contrib/pfld.h"
#include "fastdeploy/vision/facealign/contrib/pipnet.h"
#include "fastdeploy/vision/facedet/contrib/retinaface.h" #include "fastdeploy/vision/facedet/contrib/retinaface.h"
#include "fastdeploy/vision/facedet/contrib/scrfd.h" #include "fastdeploy/vision/facedet/contrib/scrfd.h"
#include "fastdeploy/vision/facedet/contrib/ultraface.h" #include "fastdeploy/vision/facedet/contrib/ultraface.h"
#include "fastdeploy/vision/facedet/contrib/yolov5face.h" #include "fastdeploy/vision/facedet/contrib/yolov5face.h"
#include "fastdeploy/vision/facealign/contrib/pfld.h"
#include "fastdeploy/vision/facealign/contrib/face_landmark_1000.h"
#include "fastdeploy/vision/facealign/contrib/pipnet.h"
#include "fastdeploy/vision/faceid/contrib/adaface.h" #include "fastdeploy/vision/faceid/contrib/adaface.h"
#include "fastdeploy/vision/faceid/contrib/arcface.h" #include "fastdeploy/vision/faceid/contrib/arcface.h"
#include "fastdeploy/vision/faceid/contrib/cosface.h" #include "fastdeploy/vision/faceid/contrib/cosface.h"
#include "fastdeploy/vision/faceid/contrib/insightface_rec.h" #include "fastdeploy/vision/faceid/contrib/insightface_rec.h"
#include "fastdeploy/vision/faceid/contrib/partial_fc.h" #include "fastdeploy/vision/faceid/contrib/partial_fc.h"
#include "fastdeploy/vision/faceid/contrib/vpl.h" #include "fastdeploy/vision/faceid/contrib/vpl.h"
#include "fastdeploy/vision/headpose/contrib/fsanet.h"
#include "fastdeploy/vision/keypointdet/pptinypose/pptinypose.h" #include "fastdeploy/vision/keypointdet/pptinypose/pptinypose.h"
#include "fastdeploy/vision/matting/contrib/modnet.h" #include "fastdeploy/vision/matting/contrib/modnet.h"
#include "fastdeploy/vision/matting/contrib/rvm.h" #include "fastdeploy/vision/matting/contrib/rvm.h"
#include "fastdeploy/vision/matting/ppmatting/ppmatting.h" #include "fastdeploy/vision/matting/ppmatting/ppmatting.h"
#include "fastdeploy/vision/ocr/ppocr/classifier.h" #include "fastdeploy/vision/ocr/ppocr/classifier.h"
#include "fastdeploy/vision/ocr/ppocr/dbdetector.h" #include "fastdeploy/vision/ocr/ppocr/dbdetector.h"
#include "fastdeploy/vision/ocr/ppocr/utils/ocr_utils.h"
#include "fastdeploy/vision/ocr/ppocr/ppocr_v2.h" #include "fastdeploy/vision/ocr/ppocr/ppocr_v2.h"
#include "fastdeploy/vision/ocr/ppocr/ppocr_v3.h" #include "fastdeploy/vision/ocr/ppocr/ppocr_v3.h"
#include "fastdeploy/vision/ocr/ppocr/recognizer.h" #include "fastdeploy/vision/ocr/ppocr/recognizer.h"
#include "fastdeploy/vision/ocr/ppocr/utils/ocr_utils.h"
#include "fastdeploy/vision/segmentation/ppseg/model.h" #include "fastdeploy/vision/segmentation/ppseg/model.h"
#include "fastdeploy/vision/tracking/pptracking/model.h"
#include "fastdeploy/vision/headpose/contrib/fsanet.h"
#include "fastdeploy/vision/sr/ppsr/model.h" #include "fastdeploy/vision/sr/ppsr/model.h"
#include "fastdeploy/vision/tracking/pptracking/model.h"
#endif #endif
#include "fastdeploy/vision/visualize/visualize.h" #include "fastdeploy/vision/visualize/visualize.h"

4
fastdeploy/vision/sr/ppsr/basicvsr.cc
View File

@@ -23,8 +23,8 @@ BasicVSR::BasicVSR(const std::string& model_file,
const RuntimeOption& custom_option, const RuntimeOption& custom_option,
const ModelFormat& model_format) { const ModelFormat& model_format) {
// unsupported ORT backend // unsupported ORT backend
valid_cpu_backends = {Backend::PDINFER}; valid_cpu_backends = {Backend::PDINFER, Backend::ORT, Backend::OPENVINO};
valid_gpu_backends = {Backend::PDINFER}; valid_gpu_backends = {Backend::PDINFER, Backend::TRT, Backend::ORT};
runtime_option = custom_option; runtime_option = custom_option;
runtime_option.model_format = model_format; runtime_option.model_format = model_format;

3
fastdeploy/vision/sr/ppsr/basicvsr.h
View File

@@ -28,8 +28,7 @@ class FASTDEPLOY_DECL BasicVSR : public PPMSVSR{
* @param[in] custom_option RuntimeOption for inference, the default will use cpu, and choose the backend defined in `valid_cpu_backends` * @param[in] custom_option RuntimeOption for inference, the default will use cpu, and choose the backend defined in `valid_cpu_backends`
* @param[in] model_format Model format of the loaded model, default is Paddle format * @param[in] model_format Model format of the loaded model, default is Paddle format
*/ */
BasicVSR(const std::string& model_file, BasicVSR(const std::string& model_file, const std::string& params_file,
const std::string& params_file,
const RuntimeOption& custom_option = RuntimeOption(), const RuntimeOption& custom_option = RuntimeOption(),
const ModelFormat& model_format = ModelFormat::PADDLE); const ModelFormat& model_format = ModelFormat::PADDLE);
/// model name contained BasicVSR /// model name contained BasicVSR

16
fastdeploy/vision/sr/ppsr/edvr.cc
View File

@@ -18,13 +18,12 @@ namespace fastdeploy {
namespace vision { namespace vision {
namespace sr { namespace sr {
EDVR::EDVR(const std::string& model_file, EDVR::EDVR(const std::string& model_file, const std::string& params_file,
const std::string& params_file,
const RuntimeOption& custom_option, const RuntimeOption& custom_option,
const ModelFormat& model_format) { const ModelFormat& model_format) {
// unsupported ORT backend // unsupported ORT backend
valid_cpu_backends = {Backend::PDINFER}; valid_cpu_backends = {Backend::PDINFER, Backend::ORT, Backend::OPENVINO};
valid_gpu_backends = {Backend::PDINFER}; valid_gpu_backends = {Backend::PDINFER, Backend::TRT, Backend::ORT};
runtime_option = custom_option; runtime_option = custom_option;
runtime_option.model_format = model_format; runtime_option.model_format = model_format;
@@ -34,11 +33,13 @@ EDVR::EDVR(const std::string& model_file,
initialized = Initialize(); initialized = Initialize();
} }
bool EDVR::Postprocess(std::vector<FDTensor>& infer_results, std::vector<cv::Mat>& results){ bool EDVR::Postprocess(std::vector<FDTensor>& infer_results,
std::vector<cv::Mat>& results) {
// group to image // group to image
// output_shape is [b, n, c, h, w] n = frame_nums b=1(default) // output_shape is [b, n, c, h, w] n = frame_nums b=1(default)
// b and n is dependence export model shape // b and n is dependence export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
auto output_shape = infer_results[0].shape; auto output_shape = infer_results[0].shape;
// EDVR // EDVR
int h_ = output_shape[2]; int h_ = output_shape[2];
@@ -53,7 +54,8 @@ bool EDVR::Postprocess(std::vector<FDTensor>& infer_results, std::vector<cv::Mat
int index = 0; int index = 0;
for (int h = 0; h < h_; ++h) { for (int h = 0; h < h_; ++h) {
for (int w = 0; w < w_; ++w) { for (int w = 0; w < w_; ++w) {
temp.at<cv::Vec3f>(h, w) = {out_data[2 * pix_num + index + frame_pix_num * frame], temp.at<cv::Vec3f>(h, w) = {
out_data[2 * pix_num + index + frame_pix_num * frame],
out_data[pix_num + index + frame_pix_num * frame], out_data[pix_num + index + frame_pix_num * frame],
out_data[index + frame_pix_num * frame]}; out_data[index + frame_pix_num * frame]};
index += 1; index += 1;

4
fastdeploy/vision/sr/ppsr/edvr.h
View File

@@ -19,7 +19,6 @@ namespace fastdeploy {
namespace vision { namespace vision {
namespace sr { namespace sr {
class FASTDEPLOY_DECL EDVR : public PPMSVSR { class FASTDEPLOY_DECL EDVR : public PPMSVSR {
public: public:
/** /**
@@ -29,8 +28,7 @@ class FASTDEPLOY_DECL EDVR : public PPMSVSR{
* @param[in] custom_option RuntimeOption for inference, the default will use cpu, and choose the backend defined in `valid_cpu_backends` * @param[in] custom_option RuntimeOption for inference, the default will use cpu, and choose the backend defined in `valid_cpu_backends`
* @param[in] model_format Model format of the loaded model, default is Paddle format * @param[in] model_format Model format of the loaded model, default is Paddle format
*/ */
EDVR(const std::string& model_file, EDVR(const std::string& model_file, const std::string& params_file,
const std::string& params_file,
const RuntimeOption& custom_option = RuntimeOption(), const RuntimeOption& custom_option = RuntimeOption(),
const ModelFormat& model_format = ModelFormat::PADDLE); const ModelFormat& model_format = ModelFormat::PADDLE);
/// model name contained EDVR /// model name contained EDVR

4
fastdeploy/vision/sr/ppsr/model.h
View File

@@ -13,6 +13,6 @@
// limitations under the License. // limitations under the License.
#pragma once #pragma once
#include "fastdeploy/vision/sr/ppsr/ppmsvsr.h"
#include "fastdeploy/vision/sr/ppsr/edvr.h"
#include "fastdeploy/vision/sr/ppsr/basicvsr.h" #include "fastdeploy/vision/sr/ppsr/basicvsr.h"
#include "fastdeploy/vision/sr/ppsr/edvr.h"
#include "fastdeploy/vision/sr/ppsr/ppmsvsr.h"

32
fastdeploy/vision/sr/ppsr/ppmsvsr.cc
View File

@@ -18,13 +18,12 @@ namespace fastdeploy {
namespace vision { namespace vision {
namespace sr { namespace sr {
PPMSVSR::PPMSVSR(const std::string& model_file, PPMSVSR::PPMSVSR(const std::string& model_file, const std::string& params_file,
const std::string& params_file,
const RuntimeOption& custom_option, const RuntimeOption& custom_option,
const ModelFormat& model_format) { const ModelFormat& model_format) {
// unsupported ORT backend // unsupported ORT backend
valid_cpu_backends = {Backend::PDINFER}; valid_cpu_backends = {Backend::PDINFER, Backend::ORT, Backend::OPENVINO};
valid_gpu_backends = {Backend::PDINFER}; valid_gpu_backends = {Backend::PDINFER, Backend::TRT, Backend::ORT};
runtime_option = custom_option; runtime_option = custom_option;
runtime_option.model_format = model_format; runtime_option.model_format = model_format;
@@ -45,7 +44,6 @@ bool PPMSVSR::Initialize(){
} }
bool PPMSVSR::Preprocess(Mat* mat, std::vector<float>& output) { bool PPMSVSR::Preprocess(Mat* mat, std::vector<float>& output) {
BGR2RGB::Run(mat); BGR2RGB::Run(mat);
Normalize::Run(mat, mean_, scale_, true); Normalize::Run(mat, mean_, scale_, true);
HWC2CHW::Run(mat); HWC2CHW::Run(mat);
@@ -56,10 +54,10 @@ bool PPMSVSR::Preprocess(Mat* mat, std::vector<float>& output) {
return true; return true;
} }
bool PPMSVSR::Predict(std::vector<cv::Mat>& imgs, std::vector<cv::Mat>& results) { bool PPMSVSR::Predict(std::vector<cv::Mat>& imgs,
std::vector<cv::Mat>& results) {
// Theoretically, the more frame nums there are, the better the result will be, // Theoretically, the more frame nums there are, the better the result will
// but it will lead to a significant increase in memory // be, but it will lead to a significant increase in memory
int frame_num = imgs.size(); int frame_num = imgs.size();
int rows = imgs[0].rows; int rows = imgs[0].rows;
int cols = imgs[0].cols; int cols = imgs[0].cols;
@@ -71,11 +69,12 @@ bool PPMSVSR::Predict(std::vector<cv::Mat>& imgs, std::vector<cv::Mat>& results)
Mat mat(imgs[i]); Mat mat(imgs[i]);
std::vector<float> data_temp; std::vector<float> data_temp;
Preprocess(&mat, data_temp); Preprocess(&mat, data_temp);
all_data_temp.insert(all_data_temp.end(), data_temp.begin(), data_temp.end()); all_data_temp.insert(all_data_temp.end(), data_temp.begin(),
data_temp.end());
} }
// share memory in order to avoid memory copy, data type must be float32 // share memory in order to avoid memory copy, data type must be float32
input_tensors[0].SetExternalData({1 ,frame_num , channels, rows, cols}, FDDataType::FP32, input_tensors[0].SetExternalData({1, frame_num, channels, rows, cols},
all_data_temp.data()); FDDataType::FP32, all_data_temp.data());
input_tensors[0].shape = {1, frame_num, channels, rows, cols}; input_tensors[0].shape = {1, frame_num, channels, rows, cols};
input_tensors[0].name = InputInfoOfRuntime(0).name; input_tensors[0].name = InputInfoOfRuntime(0).name;
std::vector<FDTensor> output_tensors; std::vector<FDTensor> output_tensors;
@@ -90,11 +89,13 @@ bool PPMSVSR::Predict(std::vector<cv::Mat>& imgs, std::vector<cv::Mat>& results)
return true; return true;
} }
bool PPMSVSR::Postprocess(std::vector<FDTensor>& infer_results, std::vector<cv::Mat>& results){ bool PPMSVSR::Postprocess(std::vector<FDTensor>& infer_results,
std::vector<cv::Mat>& results) {
// group to image // group to image
// output_shape is [b, n, c, h, w] n = frame_nums b=1(default) // output_shape is [b, n, c, h, w] n = frame_nums b=1(default)
// b and n is dependence export model shape // b and n is dependence export model shape
// see https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md // see
// https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md
auto output_shape = infer_results[0].shape; auto output_shape = infer_results[0].shape;
// PP-MSVSR // PP-MSVSR
int h_ = output_shape[3]; int h_ = output_shape[3];
@@ -110,7 +111,8 @@ bool PPMSVSR::Postprocess(std::vector<FDTensor>& infer_results, std::vector<cv::
int index = 0; int index = 0;
for (int h = 0; h < h_; ++h) { for (int h = 0; h < h_; ++h) {
for (int w = 0; w < w_; ++w) { for (int w = 0; w < w_; ++w) {
temp.at<cv::Vec3f>(h, w) = {out_data[2 * pix_num + index + frame_pix_num * frame], temp.at<cv::Vec3f>(h, w) = {
out_data[2 * pix_num + index + frame_pix_num * frame],
out_data[pix_num + index + frame_pix_num * frame], out_data[pix_num + index + frame_pix_num * frame],
out_data[index + frame_pix_num * frame]}; out_data[index + frame_pix_num * frame]};
index += 1; index += 1;

6
fastdeploy/vision/sr/ppsr/ppmsvsr.h
View File

@@ -12,14 +12,13 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
#pragma once #pragma once
#include "fastdeploy/vision/common/processors/transform.h"
#include "fastdeploy/fastdeploy_model.h" #include "fastdeploy/fastdeploy_model.h"
#include "fastdeploy/vision/common/processors/transform.h"
namespace fastdeploy { namespace fastdeploy {
namespace vision { namespace vision {
namespace sr { namespace sr {
class FASTDEPLOY_DECL PPMSVSR : public FastDeployModel { class FASTDEPLOY_DECL PPMSVSR : public FastDeployModel {
public: public:
/** /**
@@ -29,8 +28,7 @@ class FASTDEPLOY_DECL PPMSVSR:public FastDeployModel{
* @param[in] custom_option RuntimeOption for inference, the default will use cpu, and choose the backend defined in `valid_cpu_backends` * @param[in] custom_option RuntimeOption for inference, the default will use cpu, and choose the backend defined in `valid_cpu_backends`
* @param[in] model_format Model format of the loaded model, default is Paddle format * @param[in] model_format Model format of the loaded model, default is Paddle format
*/ */
PPMSVSR(const std::string& model_file, PPMSVSR(const std::string& model_file, const std::string& params_file,
const std::string& params_file,
const RuntimeOption& custom_option = RuntimeOption(), const RuntimeOption& custom_option = RuntimeOption(),
const ModelFormat& model_format = ModelFormat::PADDLE); const ModelFormat& model_format = ModelFormat::PADDLE);
/// model name contained PP-MSVSR。 /// model name contained PP-MSVSR。

27
fastdeploy/vision/sr/ppsr/ppsr_pybind.cc
View File

@@ -16,8 +16,10 @@
namespace fastdeploy { namespace fastdeploy {
void BindPPSR(pybind11::module& m) { void BindPPSR(pybind11::module& m) {
pybind11::class_<vision::sr::PPMSVSR, FastDeployModel>(m, "PPMSVSR") pybind11::class_<vision::sr::PPMSVSR, FastDeployModel>(m, "PPMSVSR")
.def(pybind11::init<std::string, std::string, RuntimeOption, ModelFormat>()) .def(pybind11::init<std::string, std::string, RuntimeOption,
.def("predict", [](vision::sr::PPMSVSR& self, std::vector<pybind11::array>& datas){ ModelFormat>())
.def("predict",
[](vision::sr::PPMSVSR& self, std::vector<pybind11::array>& datas) {
std::vector<cv::Mat> inputs; std::vector<cv::Mat> inputs;
for (auto& data : datas) { for (auto& data : datas) {
auto mat = PyArrayToCvMat(data); auto mat = PyArrayToCvMat(data);
@@ -27,14 +29,17 @@ void BindPPSR(pybind11::module &m) {
std::vector<pybind11::array> res_pyarray; std::vector<pybind11::array> res_pyarray;
self.Predict(inputs, res); self.Predict(inputs, res);
for (auto& img : res) { for (auto& img : res) {
auto ret = pybind11::array_t<unsigned char>({img.rows, img.cols, img.channels()}, img.data); auto ret = pybind11::array_t<unsigned char>(
{img.rows, img.cols, img.channels()}, img.data);
res_pyarray.push_back(ret); res_pyarray.push_back(ret);
} }
return res_pyarray; return res_pyarray;
}); });
pybind11::class_<vision::sr::EDVR, FastDeployModel>(m, "EDVR") pybind11::class_<vision::sr::EDVR, FastDeployModel>(m, "EDVR")
.def(pybind11::init<std::string, std::string, RuntimeOption, ModelFormat>()) .def(pybind11::init<std::string, std::string, RuntimeOption,
.def("predict", [](vision::sr::EDVR& self, std::vector<pybind11::array>& datas){ ModelFormat>())
.def("predict",
[](vision::sr::EDVR& self, std::vector<pybind11::array>& datas) {
std::vector<cv::Mat> inputs; std::vector<cv::Mat> inputs;
for (auto& data : datas) { for (auto& data : datas) {
auto mat = PyArrayToCvMat(data); auto mat = PyArrayToCvMat(data);
@@ -44,14 +49,17 @@ void BindPPSR(pybind11::module &m) {
std::vector<pybind11::array> res_pyarray; std::vector<pybind11::array> res_pyarray;
self.Predict(inputs, res); self.Predict(inputs, res);
for (auto& img : res) { for (auto& img : res) {
auto ret = pybind11::array_t<unsigned char>({img.rows, img.cols, img.channels()}, img.data); auto ret = pybind11::array_t<unsigned char>(
{img.rows, img.cols, img.channels()}, img.data);
res_pyarray.push_back(ret); res_pyarray.push_back(ret);
} }
return res_pyarray; return res_pyarray;
}); });
pybind11::class_<vision::sr::BasicVSR, FastDeployModel>(m, "BasicVSR") pybind11::class_<vision::sr::BasicVSR, FastDeployModel>(m, "BasicVSR")
.def(pybind11::init<std::string, std::string, RuntimeOption, ModelFormat>()) .def(pybind11::init<std::string, std::string, RuntimeOption,
.def("predict", [](vision::sr::BasicVSR& self, std::vector<pybind11::array>& datas){ ModelFormat>())
.def("predict",
[](vision::sr::BasicVSR& self, std::vector<pybind11::array>& datas) {
std::vector<cv::Mat> inputs; std::vector<cv::Mat> inputs;
for (auto& data : datas) { for (auto& data : datas) {
auto mat = PyArrayToCvMat(data); auto mat = PyArrayToCvMat(data);
@@ -61,7 +69,8 @@ void BindPPSR(pybind11::module &m) {
std::vector<pybind11::array> res_pyarray; std::vector<pybind11::array> res_pyarray;
self.Predict(inputs, res); self.Predict(inputs, res);
for (auto& img : res) { for (auto& img : res) {
auto ret = pybind11::array_t<unsigned char>({img.rows, img.cols, img.channels()}, img.data); auto ret = pybind11::array_t<unsigned char>(
{img.rows, img.cols, img.channels()}, img.data);
res_pyarray.push_back(ret); res_pyarray.push_back(ret);
} }
return res_pyarray; return res_pyarray;

9
fastdeploy/vision/vision_pybind.cc Executable file → Normal file
View File

@@ -115,7 +115,8 @@ void BindVision(pybind11::module& m) {
.def("__repr__", &vision::MattingResult::Str) .def("__repr__", &vision::MattingResult::Str)
.def("__str__", &vision::MattingResult::Str); .def("__str__", &vision::MattingResult::Str);
pybind11::class_<vision::KeyPointDetectionResult>(m, "KeyPointDetectionResult") pybind11::class_<vision::KeyPointDetectionResult>(m,
"KeyPointDetectionResult")
.def(pybind11::init()) .def(pybind11::init())
.def_readwrite("keypoints", &vision::KeyPointDetectionResult::keypoints) .def_readwrite("keypoints", &vision::KeyPointDetectionResult::keypoints)
.def_readwrite("scores", &vision::KeyPointDetectionResult::scores) .def_readwrite("scores", &vision::KeyPointDetectionResult::scores)
@@ -129,8 +130,10 @@ void BindVision(pybind11::module& m) {
.def("__repr__", &vision::HeadPoseResult::Str) .def("__repr__", &vision::HeadPoseResult::Str)
.def("__str__", &vision::HeadPoseResult::Str); .def("__str__", &vision::HeadPoseResult::Str);
m.def("enable_flycv", &vision::EnableFlyCV, "Enable image preprocessing by FlyCV."); m.def("enable_flycv", &vision::EnableFlyCV,
m.def("disable_flycv", &vision::DisableFlyCV, "Disable image preprocessing by FlyCV, change to use OpenCV."); "Enable image preprocessing by FlyCV.");
m.def("disable_flycv", &vision::DisableFlyCV,
"Disable image preprocessing by FlyCV, change to use OpenCV.");
BindDetection(m); BindDetection(m);
BindClassification(m); BindClassification(m);