[Backend] add sophgo backend (#1015)

* Add Sophgo Device add sophgo backend in fastdeploy add resnet50, yolov5s, liteseg examples. * replace sophgo lib with download links; fix model.cc bug * modify CodeStyle * remove unuseful files;change the names of sophgo device and sophgo backend * sophgo support python and add python examples * remove unuseful rows in cmake according pr Co-authored-by: Zilong Xing <zilong.xing@sophgo.com>
2025-10-05 00:33:03 +08:00 · 2023-01-04 15:49:17 +08:00
parent 0c292c0766
commit 34bea7649d
41 changed files with 1583 additions and 9 deletions
--- a/examples/vision/detection/yolov5/sophgo/README.md
+++ b/examples/vision/detection/yolov5/sophgo/README.md
@@ -0,0 +1,75 @@
+# YOLOv5 SOPHGO部署示例
+
+## 支持模型列表
+
+YOLOv5 v6.0部署模型实现来自[YOLOv5](https://github.com/ultralytics/yolov5/tree/v6.0),和[基于COCO的预训练模型](https://github.com/ultralytics/yolov5/releases/tag/v6.0)
+
+## 准备YOLOv5部署模型以及转换模型
+
+SOPHGO-TPU部署模型前需要将Paddle模型转换成bmodel模型，具体步骤如下:
+- 下载预训练ONNX模型，请参考[YOLOv5准备部署模型](https://github.com/PaddlePaddle/FastDeploy/tree/develop/examples/vision/detection/yolov5)
+- ONNX模型转换bmodel模型的过程，请参考[TPU-MLIR](https://github.com/sophgo/tpu-mlir)
+
+## 模型转换example
+
+下面以YOLOv5s为例子,教大家如何转换ONNX模型到SOPHGO-TPU模型
+
+## 下载YOLOv5s模型
+
+### 下载ONNX YOLOv5s静态图模型
+```shell
+wget https://bj.bcebos.com/paddlehub/fastdeploy/yolov5s.onnx
+
+```
+### 导出bmodel模型
+
+以转化BM1684x的bmodel模型为例子，我们需要下载[TPU-MLIR](https://github.com/sophgo/tpu-mlir)工程，安装过程具体参见[TPU-MLIR文档](https://github.com/sophgo/tpu-mlir/blob/master/README.md)。
+### 1.	安装
+``` shell
+docker pull sophgo/tpuc_dev:latest
+
+# myname1234是一个示例，也可以设置其他名字
+docker run --privileged --name myname1234 -v $PWD:/workspace -it sophgo/tpuc_dev:latest
+
+source ./envsetup.sh
+./build.sh
+```
+
+### 2.	ONNX模型转换为bmodel模型
+``` shell
+mkdir YOLOv5s && cd YOLOv5s
+
+# 在该文件中放入测试图片，同时将上一步下载的yolov5s.onnx放入该文件夹中
+cp -rf ${REGRESSION_PATH}/dataset/COCO2017 .
+cp -rf ${REGRESSION_PATH}/image .
+# 放入onnx模型文件yolov5s.onnx
+
+mkdir workspace && cd workspace
+
+# 将ONNX模型转换为mlir模型，其中参数--output_names可以通过NETRON查看
+model_transform.py \
+    --model_name yolov5s \
+    --model_def ../yolov5s.onnx \
+    --input_shapes [[1,3,640,640]] \
+    --mean 0.0,0.0,0.0 \
+    --scale 0.0039216,0.0039216,0.0039216 \
+    --keep_aspect_ratio \
+    --pixel_format rgb \
+    --output_names output,350,498,646 \
+    --test_input ../image/dog.jpg \
+    --test_result yolov5s_top_outputs.npz \
+    --mlir yolov5s.mlir
+
+# 将mlir模型转换为BM1684x的F32 bmodel模型
+model_deploy.py \
+  --mlir yolov5s.mlir \
+  --quantize F32 \
+  --chip bm1684x \
+  --test_input yolov5s_in_f32.npz \
+  --test_reference yolov5s_top_outputs.npz \
+  --model yolov5s_1684x_f32.bmodel
+```
+最终获得可以在BM1684x上能够运行的bmodel模型yolov5s_1684x_f32.bmodel。如果需要进一步对模型进行加速，可以将ONNX模型转换为INT8 bmodel，具体步骤参见[TPU-MLIR文档](https://github.com/sophgo/tpu-mlir/blob/master/README.md)。
+
+## 其他链接
+- [Cpp部署](./cpp)
--- a/examples/vision/detection/yolov5/sophgo/cpp/CMakeLists.txt
+++ b/examples/vision/detection/yolov5/sophgo/cpp/CMakeLists.txt
@@ -0,0 +1,17 @@
+PROJECT(infer_demo C CXX)
+CMAKE_MINIMUM_REQUIRED (VERSION 3.10)
+# 指定下载解压后的fastdeploy库路径
+option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.")
+
+set(ENABLE_LITE_BACKEND OFF)
+#set(FDLIB ${FASTDEPLOY_INSTALL_DIR})
+
+include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake)
+
+# 添加FastDeploy依赖头文件
+include_directories(${FASTDEPLOY_INCS})
+include_directories(${FastDeploy_INCLUDE_DIRS})
+
+add_executable(infer_demo ${PROJECT_SOURCE_DIR}/infer.cc)
+# 添加FastDeploy库依赖
+target_link_libraries(infer_demo ${FASTDEPLOY_LIBS})
--- a/examples/vision/detection/yolov5/sophgo/cpp/README.md
+++ b/examples/vision/detection/yolov5/sophgo/cpp/README.md
@@ -0,0 +1,56 @@
+# YOLOv5 C++部署示例
+
+本目录下提供`infer.cc`快速完成yolov5s模型在SOPHGO BM1684x板子上加速部署的示例。
+
+在部署前，需确认以下两个步骤:
+
+1. 软硬件环境满足要求
+2. 根据开发环境，从头编译FastDeploy仓库
+
+以上步骤请参考[SOPHGO部署库编译](../../../../../../docs/cn/build_and_install/sophgo.md)实现
+
+## 生成基本目录文件
+
+该例程由以下几个部分组成
+```text
+.
+├── CMakeLists.txt
+├── build  # 编译文件夹
+├── image  # 存放图片的文件夹
+├── infer.cc
+└── model  # 存放模型文件的文件夹
+```
+
+## 编译
+
+### 编译并拷贝SDK到thirdpartys文件夹
+
+请参考[SOPHGO部署库编译](../../../../../../docs/cn/build_and_install/sophgo.md)仓库编译SDK，编译完成后，将在build目录下生成fastdeploy-0.0.3目录.
+
+### 拷贝模型文件，以及配置文件至model文件夹
+将Paddle模型转换为SOPHGO bmodel模型，转换步骤参考[文档](../README.md)  
+将转换后的SOPHGO bmodel模型文件拷贝至model中
+
+### 准备测试图片至image文件夹
+```bash
+wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg
+cp 000000014439.jpg ./images
+```
+
+### 编译example
+
+```bash
+cd build
+cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-0.0.3
+make
+```
+
+## 运行例程
+
+```bash
+./infer_demo model images/000000014439.jpg
+```
+
+
+- [模型介绍](../../)
+- [模型转换](../)
--- a/examples/vision/detection/yolov5/sophgo/cpp/infer.cc
+++ b/examples/vision/detection/yolov5/sophgo/cpp/infer.cc
@@ -0,0 +1,61 @@
+// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+#include <string>
+#include "fastdeploy/vision.h"
+#ifdef WIN32
+const char sep = '\\';
+#else
+const char sep = '/';
+#endif
+
+void InitAndInfer(const std::string& model_dir, const std::string& image_file) {
+  auto model_file = model_dir + sep + "yolov5s_1684x_f32.bmodel";
+  auto params_file = model_dir + sep + "";
+  
+  fastdeploy::RuntimeOption option;
+  option.UseSophgo();
+  auto model_format = fastdeploy::ModelFormat::SOPHGO;
+
+  auto model = fastdeploy::vision::detection::YOLOv5(
+      model_file, params_file, option, model_format);
+
+  assert(model.Initialized());
+
+  auto im = cv::imread(image_file);
+
+  fastdeploy::vision::DetectionResult res;
+  if (!model.Predict(im, &res)) {
+    std::cerr << "Failed to predict." << std::endl;
+    return;
+  }
+
+  std::cout << res.Str() << std::endl;
+
+}
+
+int main(int argc, char* argv[]) {
+  if (argc < 3) {
+    std::cout << "Usage: infer_demo path/to/model "
+                 "path/to/image "
+                 "run_option, "
+                 "e.g ./infer_demo ./model ./test.jpeg"
+              << std::endl;
+    return -1;
+  }
+
+  std::string model_dir = argv[1];
+  std::string test_image = argv[2];
+  InitAndInfer(model_dir, test_image);
+  return 0;
+}
--- a/examples/vision/detection/yolov5/sophgo/python/README.md
+++ b/examples/vision/detection/yolov5/sophgo/python/README.md
@@ -0,0 +1,46 @@
+# YOLOv5 Python部署示例
+
+在部署前，需确认以下两个步骤
+
+- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../../docs/cn/build_and_install/sophgo.md)
+
+本目录下提供`infer.py`快速完成 YOLOv5 在SOPHGO TPU上部署的示例。执行如下脚本即可完成
+
+```bash
+# 下载部署示例代码
+git clone https://github.com/PaddlePaddle/FastDeploy.git
+cd FastDeploy/examples/vision/detection/yolov5/sophgo/python
+
+# 下载图片
+wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg
+
+# 推理
+python3 infer.py --model_file ./bmodel/yolov5s_1684x_f32.bmodel --image 000000014439.jpg
+
+# 运行完成后返回结果如下所示
+DetectionResult: [xmin, ymin, xmax, ymax, score, label_id]
+268.480255,81.053055, 298.694794, 169.439026, 0.896569, 0
+104.731163,45.661972, 127.583824, 93.449387, 0.869531, 0
+378.909363,39.750137, 395.608643, 84.243454, 0.868430, 0
+158.552979,80.361511, 199.185760, 168.181915, 0.842988, 0
+414.375305,90.948090, 506.321899, 280.405182, 0.835842, 0
+364.003448,56.608932, 381.978607, 115.968216, 0.815136, 0
+351.725128,42.635330, 366.910309, 98.048386, 0.808936, 0
+505.888306,114.366791, 593.124878, 275.995270, 0.801361, 0
+327.708618,38.363693, 346.849915, 80.893021, 0.794725, 0
+583.493408,114.532883, 612.354614, 175.873535, 0.760649, 0
+186.470657,44.941360, 199.664505, 61.037643, 0.632591, 0
+169.615891,48.014603, 178.141556, 60.888596, 0.613938, 0
+25.810200,117.199692, 59.888783, 152.850128, 0.590614, 0
+352.145294,46.712723, 381.946075, 106.752151, 0.505329, 0
+1.875000,150.734375, 37.968750, 173.781250, 0.404573, 24
+464.657288,15.901413, 472.512939, 34.116409, 0.346033, 0
+64.625000,135.171875, 84.500000, 154.406250, 0.332831, 24
+57.812500,151.234375, 103.000000, 174.156250, 0.332566, 24
+165.906250,88.609375, 527.906250, 339.953125, 0.259424, 33
+101.406250,152.562500, 118.890625, 169.140625, 0.253891, 24
+```
+
+## 其它文档
+- [YOLOv5 C++部署](../cpp)
+- [转换YOLOv5 SOPHGO模型文档](../README.md)
--- a/examples/vision/detection/yolov5/sophgo/python/infer.py
+++ b/examples/vision/detection/yolov5/sophgo/python/infer.py
@@ -0,0 +1,40 @@
+import fastdeploy as fd
+import cv2
+import os
+
+
+def parse_arguments():
+    import argparse
+    import ast
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", required=True, help="Path of model.")
+    parser.add_argument(
+        "--image", type=str, required=True, help="Path of test image file.")
+
+    return parser.parse_args()
+
+
+args = parse_arguments()
+
+# 配置runtime，加载模型
+runtime_option = fd.RuntimeOption()
+runtime_option.use_sophgo()
+
+model_file = args.model
+params_file = ""
+
+model = fd.vision.detection.YOLOv5(
+    model_file,
+    params_file,
+    runtime_option=runtime_option,
+    model_format=fd.ModelFormat.SOPHGO)
+
+# 预测图片分类结果
+im = cv2.imread(args.image)
+result = model.predict(im)
+print(result)
+
+# 预测结果可视化
+vis_im = fd.vision.vis_detection(im, result)
+cv2.imwrite("sophgo_result.jpg", vis_im)
+print("Visualized result save in ./sophgo_result.jpg")