apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-18 22:44:39 +08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy into huawei

Browse Source
This commit is contained in:
yunyaoXYY
2022-12-28 12:57:13 +00:00
parent b6903b0aa4 02e2a5365b
commit 3ed19b1f03
84 changed files with 2505 additions and 671 deletions
Download Patch File Download Diff File Expand all files Collapse all files

160
README_CN.md
View File

@@ -186,76 +186,75 @@ int main(int argc, char* argv[]) {
<img src="https://user-images.githubusercontent.com/54695910/198619323-c9b1cbce-1c1c-4f92-9737-4805c7c0ff2f.png" />
</div>
| 任务场景 | 模型 | Linux | Linux | Win | Win | Mac | Mac | Linux | Linux | Linux | Linux | Linux | Linux |
|:----------------------:|:--------------------------------------------------------------------------------------------:|:------------------------------------------------:|:----------:|:-------:|:----------:|:-------:|:-------:|:-----------:|:---------------:|:-------------:|:-------------:|:-------:|:-------:|
| --- | --- | X86 CPU | NVIDIA GPU | X86 CPU | NVIDIA GPU | X86 CPU | Arm CPU | AArch64 CPU | 飞腾D2000 aarch64 | NVIDIA Jetson | Graphcore IPU | KunlunXin XPU | Serving |
| Classification | [PaddleClas/ResNet50](./examples/vision/classification/paddleclas) | [✅](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [TorchVison/ResNet](examples/vision/classification/resnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Classification | [ltralytics/YOLOv5Cls](examples/vision/classification/yolov5cls) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Classification | [PaddleClas/PP-LCNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/PP-LCNetv2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/EfficientNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/GhostNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/MobileNetV1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/MobileNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/MobileNetV3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/ShuffleNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/SqueeezeNetV1.1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/Inceptionv3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Classification | [PaddleClas/PP-HGNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOE](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/PicoDet](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/YOLOX](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/YOLOv3](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLO](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOv2](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/Faster-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/Mask-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [Megvii-BaseDetection/YOLOX](./examples/vision/detection/yolox) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7](./examples/vision/detection/yolov7) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_trt](./examples/vision/detection/yolov7end2end_trt) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_ort_](./examples/vision/detection/yolov7end2end_ort) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Detection | [meituan/YOLOv6](./examples/vision/detection/yolov6) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [ultralytics/YOLOv5](./examples/vision/detection/yolov5) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [WongKinYiu/YOLOR](./examples/vision/detection/yolor) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Detection | [WongKinYiu/ScaledYOLOv4](./examples/vision/detection/scaledyolov4) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Detection | [ppogg/YOLOv5Lite](./examples/vision/detection/yolov5lite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ? | ❔ |❔ |
| Detection | [RangiLyu/NanoDetPlus](./examples/vision/detection/nanodet_plus) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| KeyPoint | [PaddleDetection/TinyPose](./examples/vision/keypointdetection/tiny_pose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| KeyPoint | [PaddleDetection/PicoDet + TinyPose](./examples/vision/keypointdetection/det_keypoint_unite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| HeadPose | [omasaht/headpose](examples/vision/headpose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Tracking | [PaddleDetection/PP-Tracking](examples/vision/tracking/pptracking) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| OCR | [PaddleOCR/PP-OCRv2](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| OCR | [PaddleOCR/PP-OCRv3](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Segmentation | [PaddleSeg/PP-LiteSeg](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/PP-HumanSegLite](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/HRNet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/PP-HumanSegServer](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/Unet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ | ❔ |
| Segmentation | [PaddleSeg/Deeplabv3](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| FaceDetection | [biubug6/RetinaFace](./examples/vision/facedet/retinaface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceDetection | [Linzaer/UltraFace](./examples/vision/facedet/ultraface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceDetection | [deepcam-cn/YOLOv5Face](./examples/vision/facedet/yolov5face) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceDetection | [insightface/SCRFD](./examples/vision/facedet/scrfd) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceAlign | [Hsintao/PFLD](examples/vision/facealign/pfld) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ |
| FaceAlign | [Single430/FaceLandmark1000](./examples/vision/facealign/face_landmark_1000) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| FaceAlign | [jhb86253817/PIPNet](./examples/vision/facealign) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ |
| FaceRecognition | [insightface/ArcFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/CosFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/PartialFC](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/VPL](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Matting | [ZHKKKe/MODNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ |
| Matting | [PeterL1n/RobustVideoMatting]() | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Matting | [PaddleSeg/PP-Matting](./examples/vision/matting/ppmatting) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Matting | [PaddleSeg/PP-HumanMatting](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |❔ |
| Matting | [PaddleSeg/ModNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/BasicVSR](./) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/EDVR](./examples/vision/sr/edvr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/PP-MSVSR](./examples/vision/sr/ppmsvsr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Information Extraction | [PaddleNLP/UIE](./examples/text/uie) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | |
| NLP | [PaddleNLP/ERNIE-3.0](./examples/text/ernie-3.0) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ✅ | ✅ |
| Speech | [PaddleSpeech/PP-TTS](./examples/audio/pp-tts) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | -- |❔ | ✅ |
| 任务场景 | 模型 | Linux | Linux | Win | Win | Mac | Mac | Linux | Linux | Linux | Linux | Linux | Linux | Linux |
|:----------------------:|:--------------------------------------------------------------------------------------------:|:------------------------------------------------:|:----------:|:-------:|:----------:|:-------:|:-------:|:-----------:|:---------------:|:-------------:|:-------------:|:-------:|:-------:|:-------:|
| --- | --- | X86 CPU | NVIDIA GPU | X86 CPU | NVIDIA GPU | X86 CPU | Arm CPU | AArch64 CPU | 飞腾D2000 aarch64 | [NVIDIA Jetson](./docs/cn/build_and_install/jetson.md) | [Graphcore IPU](./docs/cn/build_and_install/ipu.md) | [昆仑芯 XPU](./docs/cn/build_and_install/kunlunxin.md) |[华为 昇腾](./docs/cn/build_and_install/huawei_ascend.md) | [Serving](./serving) |
| Classification | [PaddleClas/ResNet50](./examples/vision/classification/paddleclas) | [✅](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ |
| Classification | [TorchVison/ResNet](examples/vision/classification/resnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Classification | [ltralytics/YOLOv5Cls](examples/vision/classification/yolov5cls) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |❔ |
| Classification | [PaddleClas/PP-LCNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/PP-LCNetv2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/EfficientNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/GhostNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/MobileNetV1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/MobileNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/MobileNetV3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/ShuffleNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/SqueeezeNetV1.1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/Inceptionv3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/PP-HGNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOE](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ✅ |
| Detection | [PaddleDetection/PicoDet](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |
| Detection | [PaddleDetection/YOLOX](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/YOLOv3](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLO](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOv2](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/Faster-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |❔ | ✅ |
| Detection | [PaddleDetection/Mask-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |❔ | ✅ |
| Detection | [Megvii-BaseDetection/YOLOX](./examples/vision/detection/yolox) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7](./examples/vision/detection/yolov7) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_trt](./examples/vision/detection/yolov7end2end_trt) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_ort_](./examples/vision/detection/yolov7end2end_ort) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| Detection | [meituan/YOLOv6](./examples/vision/detection/yolov6) | ✅ | ✅ | ✅ |✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [ultralytics/YOLOv5](./examples/vision/detection/yolov5) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |✅ |
| Detection | [WongKinYiu/YOLOR](./examples/vision/detection/yolor) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ✅ | ❔ |
| Detection | [WongKinYiu/ScaledYOLOv4](./examples/vision/detection/scaledyolov4) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| Detection | [ppogg/YOLOv5Lite](./examples/vision/detection/yolov5lite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ? | ❔ |❔ |❔ |
| Detection | [RangiLyu/NanoDetPlus](./examples/vision/detection/nanodet_plus) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| KeyPoint | [PaddleDetection/TinyPose](./examples/vision/keypointdetection/tiny_pose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |❔ | ❔ |
| KeyPoint | [PaddleDetection/PicoDet + TinyPose](./examples/vision/keypointdetection/det_keypoint_unite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |❔ |
| HeadPose | [omasaht/headpose](examples/vision/headpose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| Tracking | [PaddleDetection/PP-Tracking](examples/vision/tracking/pptracking) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| OCR | [PaddleOCR/PP-OCRv2](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |✅ | ❔ |
| OCR | [PaddleOCR/PP-OCRv3](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ✅ |
| Segmentation | [PaddleSeg/PP-LiteSeg](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |❔ | ❔ |
| Segmentation | [PaddleSeg/PP-HumanSegLite](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |✅ | ❔ |
| Segmentation | [PaddleSeg/HRNet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ✅ |❔ |
| Segmentation | [PaddleSeg/PP-HumanSegServer](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ✅ |❔ |
| Segmentation | [PaddleSeg/Unet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ | ✅ |❔ |
| Segmentation | [PaddleSeg/Deeplabv3](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ✅ |❔ |
| FaceDetection | [biubug6/RetinaFace](./examples/vision/facedet/retinaface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ |
| FaceDetection | [Linzaer/UltraFace](./examples/vision/facedet/ultraface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceDetection | [deepcam-cn/YOLOv5Face](./examples/vision/facedet/yolov5face) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceDetection | [insightface/SCRFD](./examples/vision/facedet/scrfd) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceAlign | [Hsintao/PFLD](examples/vision/facealign/pfld) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceAlign | [Single430/FaceLandmark1000](./examples/vision/facealign/face_landmark_1000) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ | ❔ |
| FaceAlign | [jhb86253817/PIPNet](./examples/vision/facealign) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceRecognition | [insightface/ArcFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceRecognition | [insightface/CosFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceRecognition | [insightface/PartialFC](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/VPL](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ |
| Matting | [ZHKKKe/MODNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| Matting | [PeterL1n/RobustVideoMatting]() | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ | ❔ |
| Matting | [PaddleSeg/PP-Matting](./examples/vision/matting/ppmatting) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Matting | [PaddleSeg/PP-HumanMatting](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ |❔ |
| Matting | [PaddleSeg/ModNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/BasicVSR](./) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| Video Super-Resolution | [PaddleGAN/EDVR](./examples/vision/sr/edvr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/PP-MSVSR](./examples/vision/sr/ppmsvsr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | ❔ |
| Information Extraction | [PaddleNLP/UIE](./examples/text/uie) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | |
| NLP | [PaddleNLP/ERNIE-3.0](./examples/text/ernie-3.0) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ✅ |❔ | ✅ |
| Speech | [PaddleSpeech/PP-TTS](./examples/audio/pp-tts) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | -- |❔ |❔ | ✅ |
</div></details>
@@ -275,8 +274,8 @@ int main(int argc, char* argv[]) {
| 任务场景 | 模型 | 大小(MB) | Linux | Android | Linux | Linux | Linux | Linux | Linux | 更新中... |
|:------------------:|:-----------------------------------------------------------------------------------------:|:--------:|:-------:|:-------:|:-------:|:-----------------------:|:------------------------------:|:---------------------------:|:--------------------------------:|:-------:|
| --- | --- | --- | ARM CPU | ARM CPU | 瑞芯微NPU<br>RK3568/RK3588 | 瑞芯微NPU<br>RV1109/RV1126/RK1808 | 晶晨NPU <br>A311D/S905D/C308X | 恩智浦NPU<br>i.MX&nbsp;8M&nbsp;Plus | 更新中...｜ |
| Classification | [PaddleClas/ResNet50](examples/vision/classification/paddleclas) | 98 | ✅ | ✅ | ❔ | ✅ | | | |
| --- | --- | --- | ARM CPU | [ARM CPU](./java/android) | [瑞芯微NPU<br>RK3568/RK3588](./docs/cn/build_and_install/rknpu2.md) | [瑞芯微NPU<br>RV1109/RV1126/RK1808](./docs/cn/build_and_install/rv1126.md) | [晶晨NPU <br>A311D/S905D/C308X](./docs/cn/build_and_install/a311d.md) | 恩智浦NPU<br>i.MX&nbsp;8M&nbsp;Plus | 更新中...｜ |
| Classification | [PaddleClas/ResNet50](examples/vision/classification/paddleclas) | 98 | ✅ | ✅ | [✅](./examples/vision/classification/paddleclas/rknpu2) | ✅ | | | |
| Classification | [PaddleClas/PP-LCNet](examples/vision/classification/paddleclas) | 11.9 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/PP-LCNetv2](examples/vision/classification/paddleclas) | 26.6 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/EfficientNet](examples/vision/classification/paddleclas) | 31.4 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
@@ -288,15 +287,16 @@ int main(int argc, char* argv[]) {
| Classification | [PaddleClas/SqueezeNetV1.1](examples/vision/classification/paddleclas) | 5 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/Inceptionv3](examples/vision/classification/paddleclas) | 95.5 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/PP-HGNet](examples/vision/classification/paddleclas) | 59 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Detection | [PaddleDetection/PicoDet_s](examples/vision/detection/paddledetection) | 4.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -- |
| Face Detection | [deepinsight/SCRFD](./examples/vision/facedet/scrfd) | 2.5 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Detection | [PaddleDetection/PicoDet_s](examples/vision/detection/paddledetection) | 4.9 | ✅ | ✅ | [✅](./examples/vision/detection/paddledetection/rknpu2) | ✅ | ✅ | ✅ | -- |
| Detection | [YOLOv5](./examples/vision/detection/rkyolo) | | ❔ | ❔ | [✅](./examples/vision/detection/rkyolo) | ❔ | ❔ | ❔ | -- |
| Face Detection | [deepinsight/SCRFD](./examples/vision/facedet/scrfd) | 2.5 | ✅ | ✅ | [✅](./examples/vision/facedet/scrfd/rknpu2) | -- | -- | -- | -- |
| Keypoint Detection | [PaddleDetection/PP-TinyPose](examples/vision/keypointdetection/tiny_pose) | 5.5 | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ | -- |
| Segmentation | [PaddleSeg/PP-LiteSeg(STDC1)](examples/vision/segmentation/paddleseg) | 32.2 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg-Lite](examples/vision/segmentation/paddleseg) | 0.556 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/HRNet-w18](examples/vision/segmentation/paddleseg) | 38.7 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg](examples/vision/segmentation/paddleseg) | 107.2 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Unet](examples/vision/segmentation/paddleseg) | 53.7 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Deeplabv3](examples/vision/segmentation/paddleseg) | 150 | ❔ | ✅ | ✅ | | | | |
| Segmentation | [PaddleSeg/PP-LiteSeg(STDC1)](examples/vision/segmentation/paddleseg) | 32.2 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg-Lite](examples/vision/segmentation/paddleseg) | 0.556 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/HRNet-w18](examples/vision/segmentation/paddleseg) | 38.7 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg](examples/vision/segmentation/paddleseg) | 107.2 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Unet](examples/vision/segmentation/paddleseg) | 53.7 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Deeplabv3](examples/vision/segmentation/paddleseg) | 150 | ❔ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | | | | |
| OCR | [PaddleOCR/PP-OCRv2](examples/vision/ocr/PP-OCRv2) | 2.3+4.4 | ✅ | ✅ | ❔ | -- | -- | -- | -- |
| OCR | [PaddleOCR/PP-OCRv3](examples/vision/ocr/PP-OCRv3) | 2.4+10.6 | ✅ | ❔ | ❔ | ❔ | ❔ | ❔ | -- |

160
README_EN.md
View File

@@ -186,75 +186,75 @@ Notes: ✅: already supported; ❔: to be supported in the future; N/A: Not Ava
<img src="https://user-images.githubusercontent.com/54695910/198620704-741523c1-dec7-44e5-9f2b-29ddd9997344.png" />
</div>
| Task | Model | Linux | Linux | Win | Win | Mac | Mac | Linux | Linux | Linux | Linux | Linux | Linux |
|:----------------------:|:--------------------------------------------------------------------------------------------:|:------------------------------------------------:|:------------------------:|:------------------------:|:------------------------:|:-----------------------:|:---------------------:|:--------------------------:| :----------------: |:---------------------------:|:---------------------------:|:-------:|:-------:|
| --- | --- | <font size=2> X86 CPU | <font size=2> NVIDIA GPU | <font size=2> Intel CPU | <font size=2> NVIDIA GPU | <font size=2> Intel CPU | <font size=2> Arm CPU | <font size=2> AArch64 CPU | Phytium D2000CPU | <font size=2> NVIDIA Jetson | <font size=2> Graphcore IPU | KunlunXin XPU |Serving |
| Classification | [PaddleClas/ResNet50](./examples/vision/classification/paddleclas) | [✅](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [TorchVison/ResNet](examples/vision/classification/resnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Classification | [ltralytics/YOLOv5Cls](examples/vision/classification/yolov5cls) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Classification | [PaddleClas/PP-LCNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/PP-LCNetv2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/EfficientNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/GhostNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/MobileNetV1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/MobileNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/MobileNetV3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/ShuffleNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/SqueeezeNetV1.1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Classification | [PaddleClas/Inceptionv3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Classification | [PaddleClas/PP-HGNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOE](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/PicoDet](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/YOLOX](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/YOLOv3](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLO](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOv2](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/Faster-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [PaddleDetection/Mask-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [Megvii-BaseDetection/YOLOX](./examples/vision/detection/yolox) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7](./examples/vision/detection/yolov7) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_trt](./examples/vision/detection/yolov7end2end_trt) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_ort_](./examples/vision/detection/yolov7end2end_ort) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Detection | [meituan/YOLOv6](./examples/vision/detection/yolov6) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [ultralytics/YOLOv5](./examples/vision/detection/yolov5) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Detection | [WongKinYiu/YOLOR](./examples/vision/detection/yolor) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Detection | [WongKinYiu/ScaledYOLOv4](./examples/vision/detection/scaledyolov4) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Detection | [ppogg/YOLOv5Lite](./examples/vision/detection/yolov5lite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Detection | [RangiLyu/NanoDetPlus](./examples/vision/detection/nanodet_plus) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| KeyPoint | [PaddleDetection/TinyPose](./examples/vision/keypointdetection/tiny_pose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| KeyPoint | [PaddleDetection/PicoDet + TinyPose](./examples/vision/keypointdetection/det_keypoint_unite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| HeadPose | [omasaht/headpose](examples/vision/headpose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Tracking | [PaddleDetection/PP-Tracking](examples/vision/tracking/pptracking) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| OCR | [PaddleOCR/PP-OCRv2](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| OCR | [PaddleOCR/PP-OCRv3](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |
| Segmentation | [PaddleSeg/PP-LiteSeg](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/PP-HumanSegLite](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/HRNet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/PP-HumanSegServer](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/Unet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| Segmentation | [PaddleSeg/Deeplabv3](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ❔ |
| FaceDetection | [biubug6/RetinaFace](./examples/vision/facedet/retinaface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceDetection | [Linzaer/UltraFace](./examples/vision/facedet/ultraface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceDetection | [deepcam-cn/YOLOv5Face](./examples/vision/facedet/yolov5face) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceDetection | [insightface/SCRFD](./examples/vision/facedet/scrfd) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceAlign | [Hsintao/PFLD](examples/vision/facealign/pfld) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceAlign | [Single430FaceLandmark1000](./examples/vision/facealign/face_landmark_1000) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| FaceAlign | [jhb86253817/PIPNet](./examples/vision/facealign) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/ArcFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/CosFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/PartialFC](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/VPL](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Matting | [ZHKKKe/MODNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Matting | [PeterL1n/RobustVideoMatting]() | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Matting | [PaddleSeg/PP-Matting](./examples/vision/matting/ppmatting) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Matting | [PaddleSeg/PP-HumanMatting](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Matting | [PaddleSeg/ModNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/BasicVSR](./) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/EDVR](./examples/vision/sr/edvr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/PP-MSVSR](./examples/vision/sr/ppmsvsr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |
| Information Extraction | [PaddleNLP/UIE](./examples/text/uie) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | |
| NLP | [PaddleNLP/ERNIE-3.0](./examples/text/ernie-3.0) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ✅ | ✅ |
| Speech | [PaddleSpeech/PP-TTS](./examples/audio/pp-tts) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | -- | ❔ | ✅ |
| Task | Model | Linux | Linux | Win | Win | Mac | Mac | Linux | Linux | Linux | Linux | Linux | Linux |Linux |
|:----------------------:|:--------------------------------------------------------------------------------------------:|:------------------------------------------------:|:----------:|:-------:|:----------:|:-------:|:-------:|:-----------:|:---------------:|:-------------:|:-------------:|:-------:|:-------:|:-------:|
| --- | --- | X86 CPU | NVIDIA GPU | X86 CPU | NVIDIA GPU | X86 CPU | Arm CPU | AArch64 CPU | Phytium D2000CPU | [NVIDIA Jetson](./docs/cn/build_and_install/jetson.md) | [Graphcore IPU](./docs/cn/build_and_install/ipu.md) | [KunlunXin XPU](./docs/cn/build_and_install/kunlunxin.md) |[Huawei Ascend](./docs/en/build_and_install/huawei_ascend.md) | [Serving](./serving) |
| Classification | [PaddleClas/ResNet50](./examples/vision/classification/paddleclas) | [✅](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ |
| Classification | [TorchVison/ResNet](examples/vision/classification/resnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Classification | [ltralytics/YOLOv5Cls](examples/vision/classification/yolov5cls) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |❔ |
| Classification | [PaddleClas/PP-LCNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/PP-LCNetv2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/EfficientNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/GhostNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/MobileNetV1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/MobileNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/MobileNetV3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/ShuffleNetV2](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/SqueeezeNetV1.1](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/Inceptionv3](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ✅ |
| Classification | [PaddleClas/PP-HGNet](./examples/vision/classification/paddleclas) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOE](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ✅ |
| Detection | [PaddleDetection/PicoDet](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |
| Detection | [PaddleDetection/YOLOX](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/YOLOv3](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLO](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/PP-YOLOv2](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ |
| Detection | [PaddleDetection/Faster-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |❔ | ✅ |
| Detection | [PaddleDetection/Mask-RCNN](./examples/vision/detection/paddledetection) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |❔ | ✅ |
| Detection | [Megvii-BaseDetection/YOLOX](./examples/vision/detection/yolox) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7](./examples/vision/detection/yolov7) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_trt](./examples/vision/detection/yolov7end2end_trt) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | ❔ |
| Detection | [WongKinYiu/YOLOv7end2end_ort_](./examples/vision/detection/yolov7end2end_ort) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| Detection | [meituan/YOLOv6](./examples/vision/detection/yolov6) | ✅ | ✅ | ✅ |✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |
| Detection | [ultralytics/YOLOv5](./examples/vision/detection/yolov5) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ |✅ |
| Detection | [WongKinYiu/YOLOR](./examples/vision/detection/yolor) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ✅ | ❔ |
| Detection | [WongKinYiu/ScaledYOLOv4](./examples/vision/detection/scaledyolov4) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| Detection | [ppogg/YOLOv5Lite](./examples/vision/detection/yolov5lite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ? | ❔ |❔ |❔ |
| Detection | [RangiLyu/NanoDetPlus](./examples/vision/detection/nanodet_plus) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| KeyPoint | [PaddleDetection/TinyPose](./examples/vision/keypointdetection/tiny_pose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |❔ | ❔ |
| KeyPoint | [PaddleDetection/PicoDet + TinyPose](./examples/vision/keypointdetection/det_keypoint_unite) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ |❔ |
| HeadPose | [omasaht/headpose](examples/vision/headpose) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| Tracking | [PaddleDetection/PP-Tracking](examples/vision/tracking/pptracking) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| OCR | [PaddleOCR/PP-OCRv2](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |✅ | ❔ |
| OCR | [PaddleOCR/PP-OCRv3](./examples/vision/ocr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ✅ |
| Segmentation | [PaddleSeg/PP-LiteSeg](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |❔ | ❔ |
| Segmentation | [PaddleSeg/PP-HumanSegLite](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ |✅ | ❔ |
| Segmentation | [PaddleSeg/HRNet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ✅ |❔ |
| Segmentation | [PaddleSeg/PP-HumanSegServer](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ✅ |❔ |
| Segmentation | [PaddleSeg/Unet](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ✅ | ✅ | ✅ |❔ |
| Segmentation | [PaddleSeg/Deeplabv3](./examples/vision/segmentation/paddleseg) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ✅ | ✅ |❔ |
| FaceDetection | [biubug6/RetinaFace](./examples/vision/facedet/retinaface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ |
| FaceDetection | [Linzaer/UltraFace](./examples/vision/facedet/ultraface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceDetection | [deepcam-cn/YOLOv5Face](./examples/vision/facedet/yolov5face) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceDetection | [insightface/SCRFD](./examples/vision/facedet/scrfd) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceAlign | [Hsintao/PFLD](examples/vision/facealign/pfld) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceAlign | [Single430/FaceLandmark1000](./examples/vision/facealign/face_landmark_1000) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ | ❔ |
| FaceAlign | [jhb86253817/PIPNet](./examples/vision/facealign) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceRecognition | [insightface/ArcFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceRecognition | [insightface/CosFace](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ |❔ |
| FaceRecognition | [insightface/PartialFC](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ |
| FaceRecognition | [insightface/VPL](./examples/vision/faceid/insightface) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ |
| Matting | [ZHKKKe/MODNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| Matting | [PeterL1n/RobustVideoMatting]() | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ | ❔ |
| Matting | [PaddleSeg/PP-Matting](./examples/vision/matting/ppmatting) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ | ❔ |
| Matting | [PaddleSeg/PP-HumanMatting](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |✅ |❔ |
| Matting | [PaddleSeg/ModNet](./examples/vision/matting/modnet) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ |❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/BasicVSR](./) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ | ❔ |❔ |
| Video Super-Resolution | [PaddleGAN/EDVR](./examples/vision/sr/edvr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | ❔ |
| Video Super-Resolution | [PaddleGAN/PP-MSVSR](./examples/vision/sr/ppmsvsr) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | ❔ |
| Information Extraction | [PaddleNLP/UIE](./examples/text/uie) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | ❔ | ❔ |❔ | |
| NLP | [PaddleNLP/ERNIE-3.0](./examples/text/ernie-3.0) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | ❔ | ✅ |❔ | ✅ |
| Speech | [PaddleSpeech/PP-TTS](./examples/audio/pp-tts) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❔ | -- |❔ |❔ | ✅ |
</div></details>
@@ -274,8 +274,8 @@ Notes: ✅: already supported; ❔: to be supported in the future; N/A: Not Ava
| Task | Model | Size (MB) | Linux | Android | Linux | Linux | Linux | Linux | TBD... |
|:------------------:|:----------------------------------------------------------------------------------------:|:---------:|:-------:|:-------:|:-----------------------------:|:------------------------------------:|:---------------------------------:|:---------------------------------:|:-------:|
| --- | --- | --- | ARM CPU | ARM CPU | Rockchip-NPU<br>RK3568/RK3588 | Rockchip-NPU<br>RV1109/RV1126/RK1808 | Amlogic-NPU <br>A311D/S905D/C308X | NXP-NPU<br>i.MX&nbsp;8M&nbsp;Plus | TBD...｜ |
| Classification | [PaddleClas/ResNet50](examples/vision/classification/paddleclas) | 98 | ✅ | ✅ | | ✅ | | | |
| --- | --- | --- | ARM CPU | [ARM CPU](./java/android) | [Rockchip-NPU<br>RK3568/RK3588](./docs/en/build_and_install/a311d.md) | [Rockchip-NPU<br>RV1109/RV1126/RK1808](./docs/en/build_and_install/rv1126.md) | [Amlogic-NPU <br>A311D/S905D/C308X](./docs/en/build_and_install/a311d.md) | NXP-NPU<br>i.MX&nbsp;8M&nbsp;Plus | TBD...｜ |
| Classification | [PaddleClas/ResNet50](examples/vision/classification/paddleclas) | 98 | ✅ | ✅ | [✅](./examples/vision/classification/paddleclas/rknpu2) | ✅ | | | |
| Classification | [PaddleClas/PP-LCNet](examples/vision/classification/paddleclas) | 11.9 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/PP-LCNetv2](examples/vision/classification/paddleclas) | 26.6 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/EfficientNet](examples/vision/classification/paddleclas) | 31.4 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
@@ -287,16 +287,16 @@ Notes: ✅: already supported; ❔: to be supported in the future; N/A: Not Ava
| Classification | [PaddleClas/SqueezeNetV1.1](examples/vision/classification/paddleclas) | 5 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/Inceptionv3](examples/vision/classification/paddleclas) | 95.5 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/PP-HGNet](examples/vision/classification/paddleclas) | 59 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Classification | [PaddleClas/SwinTransformer_224_win7](examples/vision/classification/paddleclas) | 352.7 | ✅ | ✅ | ❔ | ✅ | -- | -- | -- |
| Detection | [PaddleDetection/PicoDet_s](examples/vision/detection/paddledetection) | 4.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -- |
| Face Detection | [deepinsight/SCRFD](./examples/vision/facedet/scrfd) | 2.5 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Detection | [PaddleDetection/PicoDet_s](examples/vision/detection/paddledetection) | 4.9 | ✅ | ✅ | [✅](./examples/vision/detection/paddledetection/rknpu2) | ✅ | ✅ | ✅ | -- |
| Detection | [YOLOv5](./examples/vision/detection/rkyolo) | | ❔ | ❔ | [✅](./examples/vision/detection/rkyolo) | ❔ | ❔ | ❔ | -- |
| Face Detection | [deepinsight/SCRFD](./examples/vision/facedet/scrfd) | 2.5 | ✅ | ✅ | [✅](./examples/vision/facedet/scrfd/rknpu2) | -- | -- | -- | -- |
| Keypoint Detection | [PaddleDetection/PP-TinyPose](examples/vision/keypointdetection/tiny_pose) | 5.5 | ✅ | ✅ | ❔ | ❔ | ❔ | ❔ | -- |
| Segmentation | [PaddleSeg/PP-LiteSeg(STDC1)](examples/vision/segmentation/paddleseg) | 32.2 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg-Lite](examples/vision/segmentation/paddleseg) | 0.556 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/HRNet-w18](examples/vision/segmentation/paddleseg) | 38.7 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg](examples/vision/segmentation/paddleseg) | 107.2 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Unet](examples/vision/segmentation/paddleseg) | 53.7 | ✅ | ✅ | ✅ | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Deeplabv3](examples/vision/segmentation/paddleseg) | 150 | ❔ | ✅ | ✅ | | | | |
| Segmentation | [PaddleSeg/PP-LiteSeg(STDC1)](examples/vision/segmentation/paddleseg) | 32.2 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg-Lite](examples/vision/segmentation/paddleseg) | 0.556 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/HRNet-w18](examples/vision/segmentation/paddleseg) | 38.7 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/PP-HumanSeg](examples/vision/segmentation/paddleseg) | 107.2 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Unet](examples/vision/segmentation/paddleseg) | 53.7 | ✅ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | -- | -- | -- | -- |
| Segmentation | [PaddleSeg/Deeplabv3](examples/vision/segmentation/paddleseg) | 150 | ❔ | ✅ | [✅](./examples/vision/segmentation/paddleseg/rknpu2) | | | | |
| OCR | [PaddleOCR/PP-OCRv2](examples/vision/ocr/PP-OCRv2) | 2.3+4.4 | ✅ | ✅ | ❔ | -- | -- | -- | -- |
| OCR | [PaddleOCR/PP-OCRv3](examples/vision/ocr/PP-OCRv3) | 2.4+10.6 | ✅ | ❔ | ❔ | ❔ | ❔ | ❔ | -- |

2
cmake/flycv.cmake
View File

@@ -84,6 +84,8 @@ else()
else()
if(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64")
set(FLYCV_FILE "flycv-linux-aarch64-${FLYCV_VERSION}.tgz")
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "arm")
set(FLYCV_FILE "flycv-linux-armhf-${FLYCV_VERSION}.tgz")
else()
set(FLYCV_FILE "flycv-linux-x64-${FLYCV_VERSION}.tgz")
endif()

109
docs/cn/build_and_install/rknpu2.md
View File

@@ -1,105 +1,14 @@
[English](../../en/build_and_install/rknpu2.md) | 简体中文
# RK2代NPU部署库编译
# FastDeploy RKNPU2资源导航
## 写在前面
FastDeploy已经初步支持RKNPU2的部署。使用的过程中，如果出现Bug请提Issues反馈。
RKNPU2指的是Rockchip推出的RK356X以及RK3588系列芯片的NPU。
目前FastDeploy已经初步支持使用RKNPU2来部署模型。
如果您在使用的过程中出现问题，请附带上您的运行环境，在Issues中反馈。
## 简介
FastDeploy当前在RK平台上支持后端引擎如下:
## 导航目录
| 后端 | 平台 | 支持模型格式 | 说明 |
|:------------------|:---------------------|:-------|:-------------------------------------------|
| ONNX&nbsp;Runtime | RK356X <br> RK3588 | ONNX | 编译开关`ENABLE_ORT_BACKEND`为ON或OFF控制，默认OFF |
| RKNPU2 | RK356X <br> RK3588 | RKNN | 编译开关`ENABLE_RKNPU2_BACKEND`为ON或OFF控制，默认OFF |
## C++ SDK编译安装
RKNPU2仅支持linux下进行编译,以下教程均在linux环境下完成。
### 更新驱动和安装编译时需要的环境
在运行代码之前，我们需要安装以下最新的RKNPU驱动，目前驱动更新至1.4.0。为了简化安装我编写了快速安装脚本，一键即可进行安装。
**方法1: 通过脚本安装**
```bash
# 下载解压rknpu2_device_install_1.4.0
wget https://bj.bcebos.com/fastdeploy/third_libs/rknpu2_device_install_1.4.0.zip
unzip rknpu2_device_install_1.4.0.zip
cd rknpu2_device_install_1.4.0
# RK3588运行以下代码
sudo rknn_install_rk3588.sh
# RK356X运行以下代码
sudo rknn_install_rk356X.sh
```
**方法2: 通过gitee安装**
```bash
# 安装必备的包
sudo apt update -y
sudo apt install -y python3
sudo apt install -y python3-dev
sudo apt install -y python3-pip
sudo apt install -y gcc
sudo apt install -y python3-opencv
sudo apt install -y python3-numpy
sudo apt install -y cmake
# 下载rknpu2
# RK3588运行以下代码
git clone https://gitee.com/mirrors_rockchip-linux/rknpu2.git
sudo cp ./rknpu2/runtime/RK3588/Linux/librknn_api/aarch64/* /usr/lib
sudo cp ./rknpu2/runtime/RK3588/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/
# RK356X运行以下代码
git clone https://gitee.com/mirrors_rockchip-linux/rknpu2.git
sudo cp ./rknpu2/runtime/RK356X/Linux/librknn_api/aarch64/* /usr/lib
sudo cp ./rknpu2/runtime/RK356X/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/
```
### 编译C++ SDK
```bash
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy
mkdir build && cd build
# 编译配置详情见README文件，这里只介绍关键的几个配置
# -DENABLE_ORT_BACKEND: 是否开启ONNX模型，默认关闭
# -DENABLE_RKNPU2_BACKEND: 是否开启RKNPU模型，默认关闭
# -RKNN2_TARGET_SOC: 编译SDK的板子型号，只能输入RK356X或者RK3588，注意区分大小写
cmake .. -DENABLE_ORT_BACKEND=ON \
-DENABLE_RKNPU2_BACKEND=ON \
-DENABLE_VISION=ON \
-DRKNN2_TARGET_SOC=RK3588 \
-DCMAKE_INSTALL_PREFIX=${PWD}/fastdeploy-0.0.3
make -j8
make install
```
### 编译Python SDK
Python打包依赖`wheel`，编译前请先执行`pip install wheel`
```bash
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy
cd python
export ENABLE_ORT_BACKEND=ON
export ENABLE_RKNPU2_BACKEND=ON
export ENABLE_VISION=ON
export RKNN2_TARGET_SOC=RK3588
python3 setup.py build
python3 setup.py bdist_wheel
cd dist
pip3 install fastdeploy_python-0.0.0-cp39-cp39-linux_aarch64.whl
```
## 部署模型
请查看[RKNPU2部署模型教程](../faq/rknpu2/rknpu2.md)
* [RKNPU2开发环境搭建](../faq/rknpu2/environment.md)
* [编译FastDeploy](../faq/rknpu2/build.md)
* [RKNN模型导出建议](../faq/rknpu2/export.md)
* [RKNPU2模型部署demo](../faq/rknpu2/rknpu2.md)

1
docs/cn/build_and_install/rv1126.md
View File

@@ -61,6 +61,7 @@ mkdir build && cd build
cmake -DCMAKE_TOOLCHAIN_FILE=./../cmake/toolchain.cmake \
-DWITH_TIMVX=ON \
-DTARGET_ABI=armhf \
-DENABLE_FLYCV=ON \ # 是否开启 FlyCV 优化前后处理，可以选择开启
-DCMAKE_INSTALL_PREFIX=fastdeploy-timvx \
-DENABLE_VISION=ON \ # 是否编译集成视觉模型的部署模块，可选择开启
-Wno-dev ..

54
docs/cn/faq/rknpu2/build.md Normal file
View File

@@ -0,0 +1,54 @@
# FastDeploy RKNPU2引擎编译
## FastDeploy后端支持详情
FastDeploy当前在RK平台上支持后端引擎如下:
| 后端 | 平台 | 支持模型格式 | 说明 |
|:------------------|:---------------------|:-------|:-------------------------------------------|
| ONNX&nbsp;Runtime | RK356X <br> RK3588 | ONNX | 编译开关`ENABLE_ORT_BACKEND`为ON或OFF控制，默认OFF |
| RKNPU2 | RK356X <br> RK3588 | RKNN | 编译开关`ENABLE_RKNPU2_BACKEND`为ON或OFF控制，默认OFF |
## 板端编译
### 编译FastDeploy C++ SDK
RKNPU2暂时仅支持linux系统, 以下教程在RK3568(debian 10)、RK3588(debian 11) 环境下完成。
```bash
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy
mkdir build && cd build
# 编译配置详情见README文件，这里只介绍关键的几个配置
# -DENABLE_ORT_BACKEND: 是否开启ONNX模型，默认关闭
# -DENABLE_RKNPU2_BACKEND: 是否开启RKNPU模型，默认关闭
# -RKNN2_TARGET_SOC: 编译SDK的板子型号，只能输入RK356X或者RK3588，注意区分大小写
cmake .. -DENABLE_ORT_BACKEND=ON \
-DENABLE_RKNPU2_BACKEND=ON \
-DENABLE_VISION=ON \
-DRKNN2_TARGET_SOC=RK3588 \
-DCMAKE_INSTALL_PREFIX=${PWD}/fastdeploy-0.0.3
make -j8
make install
```
### 编译Python SDK
RKNPU2暂时仅支持linux系统, 以下教程在RK3568(debian 10)、RK3588(debian 11) 环境下完成。Python打包依赖`wheel`，编译前请先执行`pip install wheel`
```bash
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy
cd python
export ENABLE_ORT_BACKEND=ON
export ENABLE_RKNPU2_BACKEND=ON
export ENABLE_VISION=ON
export RKNN2_TARGET_SOC=RK3588
python3 setup.py build
python3 setup.py bdist_wheel
cd dist
pip3 install fastdeploy_python-0.0.0-cp39-cp39-linux_aarch64.whl
```

90
docs/cn/faq/rknpu2/environment.md Normal file
View File

@@ -0,0 +1,90 @@
# FastDeploy RKNPU2推理环境搭建
## 简介
在FastDeploy上部署模型前我们需要搭建一下开发环境。FastDeploy将环境搭建分成板端推理环境搭建和PC端模型转换环境搭建两个部分。
## 板端推理环境搭建
根据开发者的反馈，我们提供了一键安装脚本和命令行安装开发版驱动两种方式来安装板端的推理环境。
### 通过脚本安装
多数开发者不喜欢使用复杂的命令行来进行安装，FastDeploy贴心的为开发者提供了一键安装稳定版RKNN的方式。参考以下命令，即可一键安装板端编译环境
```bash
# 下载解压rknpu2_device_install_1.4.0
wget https://bj.bcebos.com/fastdeploy/third_libs/rknpu2_device_install_1.4.0.zip
unzip rknpu2_device_install_1.4.0.zip
cd rknpu2_device_install_1.4.0
# RK3588运行以下代码
sudo rknn_install_rk3588.sh
# RK356X运行以下代码
sudo rknn_install_rk356X.sh
```
### 通过命令行安装
在开发的过程中，有的开发者希望能够体验到最新的RK驱动，我们也提供了对应的安装方式，使用以下下命令行即可从零开始安装RKNN的驱动。
```bash
# 安装必备的包
sudo apt update -y
sudo apt install -y python3
sudo apt install -y python3-dev
sudo apt install -y python3-pip
sudo apt install -y gcc
sudo apt install -y python3-opencv
sudo apt install -y python3-numpy
sudo apt install -y cmake
# 下载rknpu2
# RK3588运行以下代码
git clone https://gitee.com/mirrors_rockchip-linux/rknpu2.git
sudo cp ./rknpu2/runtime/RK3588/Linux/librknn_api/aarch64/* /usr/lib
sudo cp ./rknpu2/runtime/RK3588/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/
# RK356X运行以下代码
git clone https://gitee.com/mirrors_rockchip-linux/rknpu2.git
sudo cp ./rknpu2/runtime/RK356X/Linux/librknn_api/aarch64/* /usr/lib
sudo cp ./rknpu2/runtime/RK356X/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/
```
## 安装rknn_toolkit2
安装rknn_toolkit2中会存在依赖问题，这里介绍以下如何安装。 rknn_toolkit2依赖一些特定的包，因此建议使用conda新建一个虚拟环境进行安装。
安装conda的方法百度有很多，这里跳过，直接介绍如何安装rknn_toolkit2。
### 下载rknn_toolkit2
一般可以通过git直接下载rknn_toolkit2
```bash
git clone https://github.com/rockchip-linux/rknn-toolkit2.git
```
### 下载安装需要的软件包
```bash
sudo apt-get install libxslt1-dev zlib1g zlib1g-dev libglib2.0-0 \
libsm6 libgl1-mesa-glx libprotobuf-dev gcc g++
```
### 安装rknn_toolkit2环境
```bash
# 创建虚拟环境
conda create -n rknn2 python=3.6
conda activate rknn2
# rknn_toolkit2对numpy存在特定依赖,因此需要先安装numpy==1.16.6
pip install numpy==1.16.6
# 安装rknn_toolkit2-1.3.0_11912b58-cp38-cp38-linux_x86_64.whl
cd ~/下载/rknn-toolkit2-master/packages
pip install rknn_toolkit2-1.3.0_11912b58-cp38-cp38-linux_x86_64.whl
```
## 资源链接
* [RKNPU2、rknntoolkit2开发板下载地址 密码：rknn](https://eyun.baidu.com/s/3eTDMk6Y)
## 其他文档
- [RKNN 模型转换文档](./export.md)

50
docs/cn/faq/rknpu2/install_rknn_toolkit2.md
View File

@@ -1,50 +0,0 @@
[English](../../../en/faq/rknpu2/install_rknn_toolkit2.md) | 中文
# 安装rknn_toolkit2仓库
## 下载rknn_toolkit2
rknn_toolkit2的下载一般有两种方式，以下将一一介绍:
* github仓库下载
github仓库中提供了稳定版本的rknn_toolkit2下载
```bash
git clone https://github.com/rockchip-linux/rknn-toolkit2.git
```
* 百度网盘下载
在有些时候，如果稳定版本的rknn_toolkit2存在bug，不满足模型部署的要求，我们也可以使用百度网盘下载beta版本的rknn_toolkit2使用。其安装方式与
稳定版本一致
```text
链接：https://eyun.baidu.com/s/3eTDMk6Y 密码：rknn
```
## 安装rknn_toolkit2
安装rknn_toolkit2中会存在依赖问题，这里介绍以下如何安装。首先，因为rknn_toolkit2依赖一些特定的包，因此建议使用conda新建一个虚拟环境进行安装。
安装conda的方法百度有很多，这里跳过，直接介绍如何安装rknn_toolkit2。
### 下载安装需要的软件包
```bash
sudo apt-get install libxslt1-dev zlib1g zlib1g-dev libglib2.0-0 \
libsm6 libgl1-mesa-glx libprotobuf-dev gcc g++
```
### 安装rknn_toolkit2环境
```bash
# 创建虚拟环境
conda create -n rknn2 python=3.6
conda activate rknn2
# rknn_toolkit2对numpy存在特定依赖,因此需要先安装numpy==1.16.6
pip install numpy==1.16.6
# 安装rknn_toolkit2-1.3.0_11912b58-cp38-cp38-linux_x86_64.whl
cd ~/下载/rknn-toolkit2-master/packages
pip install rknn_toolkit2-1.3.0_11912b58-cp38-cp38-linux_x86_64.whl
```
## 其他文档
- [onnx转换rknn文档](./export.md)

72
docs/cn/faq/rknpu2/rknpu2.md
View File

@@ -1,8 +1,8 @@
[English](../../../en/faq/rknpu2/rknpu2.md) | 中文
# RKNPU2模型部署
# RKNPU2概述
## 安装环境
RKNPU2模型导出只支持在x86Linux平台上进行导出，安装流程请参考[RKNPU2模型导出环境配置文档](./install_rknn_toolkit2.md)
RKNPU2模型导出只支持在x86Linux平台上进行导出，安装流程请参考[RKNPU2模型导出环境配置文档](./environment.md)
## ONNX模型转换为RKNN模型
ONNX模型不能直接调用RK芯片中的NPU进行运算，需要把ONNX模型转换为RKNN模型，具体流程请查看[RKNPU2转换文档](./export.md)
@@ -13,60 +13,14 @@ ONNX模型不能直接调用RK芯片中的NPU进行运算，需要把ONNX模型
* ARM CPU使用ONNX框架进行测试
* NPU均使用单核进行测试
| 任务场景 | 模型 | 模型版本(表示已经测试的版本) | ARM CPU/RKNN速度(ms) |
|------------------|-------------------|-------------------------------|--------------------|
| Detection | Picodet | Picodet-s | 162/112 |
| Detection | RKYOLOV5 | YOLOV5-S-Relu(int8) | -/57 |
| Detection | RKYOLOX | - | -/- |
| Detection | RKYOLOV7 | - | -/- |
| Segmentation | Unet | Unet-cityscapes | -/- |
| Segmentation | PP-HumanSegV2Lite | portrait | 133/43 |
| Segmentation | PP-HumanSegV2Lite | human | 133/43 |
| Face Detection | SCRFD | SCRFD-2.5G-kps-640 | 108/42 |
## RKNPU2 Backend推理使用教程
这里以Scrfd模型为例子教你如何使用RKNPU2 Backend推理模型。以下注释中的改动，是对比onnx cpu的改动。
```c++
int infer_scrfd_npu() {
char model_path[] = "./model/scrfd_2.5g_bnkps_shape640x640.rknn";
char image_file[] = "./image/test_lite_face_detector_3.jpg";
auto option = fastdeploy::RuntimeOption();
// 改动1: option需要调用UseRKNPU2
option.UseRKNPU2();
// 改动2: 模型加载时需要传递fastdeploy::ModelFormat::RKNN参数
auto *model = new fastdeploy::vision::facedet::SCRFD(model_path,"",option,fastdeploy::ModelFormat::RKNN);
if (!model->Initialized()) {
std::cerr << "Failed to initialize." << std::endl;
return 0;
}
// 改动3(可选): RKNPU2支持使用NPU进行normalize操作，并且输入格式为nhwc格式。
// DisableNormalizeAndPermute操作将屏蔽预处理时的nor操作和hwc转chw操作。
// 如果你使用的是已经支持的模型列表，请在Predict前调用该方法。
model->DisableNormalizeAndPermute();
auto im = cv::imread(image_file);
auto im_bak = im.clone();
fastdeploy::vision::FaceDetectionResult res;
clock_t start = clock();
if (!model->Predict(&im, &res, 0.8, 0.8)) {
std::cerr << "Failed to predict." << std::endl;
return 0;
}
clock_t end = clock();
double dur = (double) (end - start);
printf("infer_scrfd_npu use time:%f\n", (dur / CLOCKS_PER_SEC));
auto vis_im = fastdeploy::vision::Visualize::VisFaceDetection(im_bak, res);
cv::imwrite("scrfd_rknn_vis_result.jpg", vis_im);
std::cout << "Visualized result saved in ./scrfd_rknn_vis_result.jpg" << std::endl;
return 0;
}
```
## 其他关联文档
- [rknpu2板端环境安装配置](../../build_and_install/rknpu2.md)
- [rknn_toolkit2安装文档](./install_rknn_toolkit2.md)
- [onnx转换rknn文档](./export.md)
| 任务场景 | 模型 | 模型版本(表示已经测试的版本) | ARM CPU/RKNN速度(ms) |
|----------------|------------------------------------------------------------------------------------------|--------------------------|--------------------|
| Detection | [Picodet](../../../../examples/vision/detection/paddledetection/rknpu2/README.md) | Picodet-s | 162/112 |
| Detection | [RKYOLOV5](../../../../examples/vision/detection/rkyolo/README.md) | YOLOV5-S-Relu(int8) | -/57 |
| Detection | [RKYOLOX](../../../../examples/vision/detection/rkyolo/README.md) | - | -/- |
| Detection | [RKYOLOV7](../../../../examples/vision/detection/rkyolo/README.md) | - | -/- |
| Segmentation | [Unet](../../../../examples/vision/segmentation/paddleseg/rknpu2/README.md) | Unet-cityscapes | -/- |
| Segmentation | [PP-HumanSegV2Lite](../../../../examples/vision/segmentation/paddleseg/rknpu2/README.md) | portrait(int8) | 133/43 |
| Segmentation | [PP-HumanSegV2Lite](../../../../examples/vision/segmentation/paddleseg/rknpu2/README.md) | human(int8) | 133/43 |
| Face Detection | [SCRFD](../../../../examples/vision/facedet/scrfd/rknpu2/README.md) | SCRFD-2.5G-kps-640(int8) | 108/42 |
| Classification | [ResNet](../../../../examples/vision/classification/paddleclas/rknpu2/README.md) | ResNet50_vd | -/92 |

1
docs/en/build_and_install/rv1126.md
View File

@@ -60,6 +60,7 @@ mkdir build && cd build
cmake -DCMAKE_TOOLCHAIN_FILE=./../cmake/toolchain.cmake \
-DWITH_TIMVX=ON \
-DTARGET_ABI=armhf \
-DENABLE_FLYCV=ON \ # Whether to enable FlyCV optimization
-DCMAKE_INSTALL_PREFIX=fastdeploy-timvx \
-DENABLE_VISION=ON \ # Whether to compile the vision module
-Wno-dev ..

8
examples/text/ernie-3.0/serving/README.md
View File

@@ -51,12 +51,12 @@ models
```bash
# x.y.z represent image versions. Please refer to the serving document to replace them with numbers
# GPU Image
docker pull paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
# CPU Image
docker pull paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
# Running
docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10 bash
docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10 bash
```
## Deployment Models
@@ -70,7 +70,7 @@ token_cls_rpc_client.py # 序列标注任务发送pipeline预测请求的脚
*Attention*:Attention: When starting the service, each python backend process of Server requests 64M memory by default, and the docker started by default cannot start more than one python backend node. There are two solutions:
- 1.Set the `shm-size` parameter when starting the container, for example, `docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash`
- 1.Set the `shm-size` parameter when starting the container, for example, `docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash`
- 2.Set the `shm-default-byte-size` parameter of python backend when starting the service. Set the default memory of python backend to 10M： `tritonserver --model-repository=/models --backend-config=python,shm-default-byte-size=10485760`
### Classification Task

8
examples/text/ernie-3.0/serving/README_CN.md
View File

@@ -51,12 +51,12 @@ models
```bash
# x.y.z为镜像版本号，需参照serving文档替换为数字
# GPU镜像
docker pull paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
# CPU镜像
docker pull paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
# 运行
docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10 bash
docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10 bash
```
## 部署模型
@@ -69,7 +69,7 @@ token_cls_rpc_client.py # 序列标注任务发送pipeline预测请求的脚
```
*注意*:启动服务时，Server的每个python后端进程默认申请`64M`内存，默认启动的docker无法启动多个python后端节点。有两个解决方案：
- 1.启动容器时设置`shm-size`参数, 比如:`docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash`
- 1.启动容器时设置`shm-size`参数, 比如:`docker run -it --net=host --name fastdeploy_server --shm-size="1g" -v /path/serving/models:/models registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash`
- 2.启动服务时设置python后端的`shm-default-byte-size`参数, 设置python后端的默认内存为10M： `tritonserver --model-repository=/models --backend-config=python,shm-default-byte-size=10485760`
### 分类任务

28
examples/text/uie/cpp/infer.cc
View File

@@ -49,17 +49,17 @@ int main(int argc, char* argv[]) {
backend_type = std::atoi(argv[3]);
}
switch (backend_type) {
case 0:
option.UsePaddleInferBackend();
break;
case 1:
option.UseOrtBackend();
break;
case 2:
option.UseOpenVINOBackend();
break;
default:
break;
case 0:
option.UsePaddleInferBackend();
break;
case 1:
option.UseOrtBackend();
break;
case 2:
option.UseOpenVINOBackend();
break;
default:
break;
}
std::string model_dir(argv[1]);
std::string model_path = model_dir + sep + "inference.pdmodel";
@@ -68,9 +68,9 @@ int main(int argc, char* argv[]) {
using fastdeploy::text::SchemaNode;
using fastdeploy::text::UIEResult;
auto predictor =
fastdeploy::text::UIEModel(model_path, param_path, vocab_path, 0.5, 128,
{"时间", "选手", "赛事名称"}, option);
auto predictor = fastdeploy::text::UIEModel(
model_path, param_path, vocab_path, 0.5, 128,
{"时间", "选手", "赛事名称"}, /* batch_size = */ 1, option);
std::cout << "After init predictor" << std::endl;
std::vector<std::unordered_map<std::string, std::vector<UIEResult>>> results;
// Named Entity Recognition

4
examples/text/uie/python/infer.py
View File

@@ -129,6 +129,7 @@ if __name__ == "__main__":
position_prob=0.5,
max_length=args.max_length,
schema=schema,
batch_size=args.batch_size,
runtime_option=runtime_option,
schema_language=SchemaLanguage.ZH)
@@ -181,7 +182,8 @@ if __name__ == "__main__":
schema = {"评价维度": ["观点词", "情感倾向[正向，负向]"]}
print(f"The extraction schema: {schema}")
uie.set_schema(schema)
results = uie.predict(["店面干净，很清静"], return_dict=True)
results = uie.predict(
["店面干净，很清静，服务员服务热情，性价比很高，发现收银台有排队"], return_dict=True)
pprint(results)
print()

57
examples/vision/classification/paddleclas/rknpu2/README.md Normal file
View File

@@ -0,0 +1,57 @@
# PaddleClas 模型RKNPU2部署
## 转换模型
下面以 ResNet50_vd为例子，教大家如何转换分类模型到RKNN模型。
```bash
# 安装 paddle2onnx
pip install paddle2onnx
# 下载ResNet50_vd模型文件和测试图片
wget https://bj.bcebos.com/paddlehub/fastdeploy/ResNet50_vd_infer.tgz
tar -xvf ResNet50_vd_infer.tgz
# 静态图转ONNX模型，注意，这里的save_file请和压缩包名对齐
paddle2onnx --model_dir ResNet50_vd_infer \
--model_filename inference.pdmodel \
--params_filename inference.pdiparams \
--save_file ResNet50_vd_infer/ResNet50_vd_infer.onnx \
--enable_dev_version True \
--opset_version 12 \
--enable_onnx_checker True
# 固定shape，注意这里的inputs得对应netron.app展示的 inputs 的 name，有可能是image 或者 x
python -m paddle2onnx.optimize --input_model ResNet50_vd_infer/ResNet50_vd_infer.onnx \
--output_model ResNet50_vd_infer/ResNet50_vd_infer.onnx \
--input_shape_dict "{'inputs':[1,3,224,224]}"
```
### 编写模型导出配置文件
以转化RK3588的RKNN模型为例子，我们需要编辑tools/rknpu2/config/ResNet50_vd_infer_rknn.yaml，来转换ONNX模型到RKNN模型。
默认的 mean=0, std=1是在内存做normalize，如果你需要在NPU上执行normalize操作，请根据你的模型配置normalize参数，例如:
```yaml
model_path: ./ResNet50_vd_infer.onnx
output_folder: ./
target_platform: RK3588
normalize:
mean: [[0.485,0.456,0.406]]
std: [[0.229,0.224,0.225]]
outputs: []
outputs_nodes: []
do_quantization: False
dataset:
```
# ONNX模型转RKNN模型
```shell
python tools/rknpu2/export.py \
--config_path tools/rknpu2/config/ResNet50_vd_infer_rknn.yaml \
--target_platform rk3588
```
## 其他链接
- [Cpp部署](./cpp)
- [Python部署](./python)
- [视觉模型预测结果](../../../../../docs/api/vision_results/)

37
examples/vision/classification/paddleclas/rknpu2/cpp/CMakeLists.txt Normal file
View File

@@ -0,0 +1,37 @@
CMAKE_MINIMUM_REQUIRED(VERSION 3.10)
project(rknpu_test)
set(CMAKE_CXX_STANDARD 14)
# 指定下载解压后的fastdeploy库路径
set(FASTDEPLOY_INSTALL_DIR "thirdpartys/fastdeploy-0.0.3")
include(${FASTDEPLOY_INSTALL_DIR}/FastDeployConfig.cmake)
include_directories(${FastDeploy_INCLUDE_DIRS})
add_executable(rknpu_test infer.cc)
target_link_libraries(rknpu_test
${FastDeploy_LIBS}
)
set(CMAKE_INSTALL_PREFIX ${CMAKE_SOURCE_DIR}/build/install)
install(TARGETS rknpu_test DESTINATION ./)
install(DIRECTORY ppclas_model_dir DESTINATION ./)
install(DIRECTORY images DESTINATION ./)
file(GLOB FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/*)
message("${FASTDEPLOY_LIBS}")
install(PROGRAMS ${FASTDEPLOY_LIBS} DESTINATION lib)
file(GLOB ONNXRUNTIME_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/onnxruntime/lib/*)
install(PROGRAMS ${ONNXRUNTIME_LIBS} DESTINATION lib)
install(DIRECTORY ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/opencv/lib DESTINATION ./)
file(GLOB PADDLETOONNX_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddle2onnx/lib/*)
install(PROGRAMS ${PADDLETOONNX_LIBS} DESTINATION lib)
file(GLOB RKNPU2_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/rknpu2_runtime/RK3588/lib/*)
install(PROGRAMS ${RKNPU2_LIBS} DESTINATION lib)

78
examples/vision/classification/paddleclas/rknpu2/cpp/README.md Normal file
View File

@@ -0,0 +1,78 @@
# PaddleClas C++部署示例
本目录下用于展示 ResNet50_vd 模型在RKNPU2上的部署，以下的部署过程以 ResNet50_vd 为例子。
在部署前，需确认以下两个步骤:
1. 软硬件环境满足要求
2. 根据开发环境，下载预编译部署库或者从头编译FastDeploy仓库
以上步骤请参考[RK2代NPU部署库编译](../../../../../../docs/cn/build_and_install/rknpu2.md)实现
## 生成基本目录文件
该例程由以下几个部分组成
```text
.
├── CMakeLists.txt
├── build # 编译文件夹
├── images # 存放图片的文件夹
├── infer.cc
├── ppclas_model_dir # 存放模型文件的文件夹
└── thirdpartys # 存放sdk的文件夹
```
首先需要先生成目录结构
```bash
mkdir build
mkdir images
mkdir ppclas_model_dir
mkdir thirdpartys
```
## 编译
### 编译并拷贝SDK到thirdpartys文件夹
请参考[RK2代NPU部署库编译](../../../../../../docs/cn/build_and_install/rknpu2.md)仓库编译SDK，编译完成后，将在build目录下生成
fastdeploy-0.0.3目录，请移动它至thirdpartys目录下.
### 拷贝模型文件，以及配置文件至model文件夹
在Paddle动态图模型 -> Paddle静态图模型 -> ONNX模型的过程中，将生成ONNX文件以及对应的yaml配置文件，请将配置文件存放到model文件夹内。
转换为RKNN后的模型文件也需要拷贝至model，转换方案: ([ResNet50_vd RKNN模型](../README.md))。
### 准备测试图片至image文件夹
```bash
wget https://gitee.com/paddlepaddle/PaddleClas/raw/release/2.4/deploy/images/ImageNet/ILSVRC2012_val_00000010.jpeg
```
### 编译example
```bash
cd build
cmake ..
make -j8
make install
```
## 运行例程
```bash
cd ./build/install
./rknpu_test ./ppclas_model_dir ./images/ILSVRC2012_val_00000010.jpeg
```
## 运行结果展示
ClassifyResult(
label_ids: 153,
scores: 0.684570,
)
## 注意事项
RKNPU上对模型的输入要求是使用NHWC格式，且图片归一化操作会在转RKNN模型时，内嵌到模型中，因此我们在使用FastDeploy部署时，
DisablePermute(C++)或`disable_permute(Python)，在预处理阶段禁用数据格式的转换。
## 其它文档
- [ResNet50_vd Python 部署](../python)
- [模型预测结果说明](../../../../../../docs/api/vision_results/)
- [转换ResNet50_vd RKNN模型文档](../README.md)

58
examples/vision/classification/paddleclas/rknpu2/cpp/infer.cc Executable file
View File

@@ -0,0 +1,58 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "fastdeploy/vision.h"
void RKNPU2Infer(const std::string& model_dir, const std::string& image_file) {
auto model_file = model_dir + "/ResNet50_vd_infer_rk3588.rknn";
auto params_file = "";
auto config_file = model_dir + "/inference_cls.yaml";
auto option = fastdeploy::RuntimeOption();
option.UseRKNPU2();
auto format = fastdeploy::ModelFormat::RKNN;
auto model = fastdeploy::vision::classification::PaddleClasModel(
model_file, params_file, config_file,option,format);
if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl;
return;
}
model.GetPreprocessor().DisablePermute();
fastdeploy::TimeCounter tc;
tc.Start();
auto im = cv::imread(image_file);
fastdeploy::vision::ClassifyResult res;
if (!model.Predict(im, &res)) {
std::cerr << "Failed to predict." << std::endl;
return;
}
// print res
std::cout << res.Str() << std::endl;
tc.End();
tc.PrintInfo("PPClas in RKNPU2");
}
int main(int argc, char* argv[]) {
if (argc < 3) {
std::cout
<< "Usage: rknpu_test path/to/model_dir path/to/image run_option, "
"e.g ./rknpu_test ./ppclas_model_dir ./images/ILSVRC2012_val_00000010.jpeg"
<< std::endl;
return -1;
}
RKNPU2Infer(argv[1], argv[2]);
return 0;
}

35
examples/vision/classification/paddleclas/rknpu2/python/README.md Normal file
View File

@@ -0,0 +1,35 @@
# PaddleClas Python部署示例
在部署前，需确认以下两个步骤
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../../docs/cn/build_and_install/rknpu2.md)
本目录下提供`infer.py`快速完成 ResNet50_vd 在RKNPU上部署的示例。执行如下脚本即可完成
```bash
# 下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/classification/paddleclas/rknpu2/python
# 下载图片
wget https://gitee.com/paddlepaddle/PaddleClas/raw/release/2.4/deploy/images/ImageNet/ILSVRC2012_val_00000010.jpeg
# 推理
python3 infer.py --model_file ./ResNet50_vd_infer/ResNet50_vd_infer_rk3588.rknn --config_file ResNet50_vd_infer/inference_cls.yaml --image ILSVRC2012_val_00000010.jpeg
# 运行完成后返回结果如下所示
ClassifyResult(
label_ids: 153,
scores: 0.684570,
)
```
## 注意事项
RKNPU上对模型的输入要求是使用NHWC格式，且图片归一化操作会在转RKNN模型时，内嵌到模型中，因此我们在使用FastDeploy部署时，
DisablePermute(C++)或`disable_permute(Python)，在预处理阶段禁用数据格式的转换。
## 其它文档
- [ResNet50_vd C++部署](../cpp)
- [模型预测结果说明](../../../../../../docs/api/vision_results/)
- [转换ResNet50_vd RKNN模型文档](../README.md)

50
examples/vision/classification/paddleclas/rknpu2/python/infer.py Normal file
View File

@@ -0,0 +1,50 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import fastdeploy as fd
import cv2
import os
def parse_arguments():
import argparse
import ast
parser = argparse.ArgumentParser()
parser.add_argument(
"--model_file", required=True, help="Path of rknn model.")
parser.add_argument("--config_file", required=True, help="Path of config.")
parser.add_argument(
"--image", type=str, required=True, help="Path of test image file.")
return parser.parse_args()
if __name__ == "__main__":
args = parse_arguments()
model_file = args.model_file
params_file = ""
config_file = args.config_file
# 配置runtime，加载模型
runtime_option = fd.RuntimeOption()
runtime_option.use_rknpu2()
model = fd.vision.classification.ResNet50vd(
model_file,
params_file,
config_file,
runtime_option=runtime_option,
model_format=fd.ModelFormat.RKNN)
# 禁用通道转换
model.preprocessor.disable_permute()
im = cv2.imread(args.image)
result = model.predict(im, topk=1)
print(result)

16
examples/vision/classification/paddleclas/rv1126/cpp/CMakeLists.txt
View File

@@ -20,19 +20,11 @@ install(TARGETS infer_demo DESTINATION ./)
install(DIRECTORY models DESTINATION ./)
install(DIRECTORY images DESTINATION ./)
# install(DIRECTORY run_with_adb.sh DESTINATION ./)
file(GLOB FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/*)
install(PROGRAMS ${FASTDEPLOY_LIBS} DESTINATION lib)
file(GLOB OPENCV_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/opencv/lib/lib*)
install(PROGRAMS ${OPENCV_LIBS} DESTINATION lib)
file(GLOB PADDLELITE_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/lib*)
install(PROGRAMS ${PADDLELITE_LIBS} DESTINATION lib)
file(GLOB TIMVX_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/verisilicon_timvx/*)
install(PROGRAMS ${TIMVX_LIBS} DESTINATION lib)
file(GLOB_RECURSE FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/lib*.so*)
file(GLOB_RECURSE ALL_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/lib*.so*)
list(APPEND ALL_LIBS ${FASTDEPLOY_LIBS})
install(PROGRAMS ${ALL_LIBS} DESTINATION lib)
file(GLOB ADB_TOOLS run_with_adb.sh)
install(PROGRAMS ${ADB_TOOLS} DESTINATION ./)

2
examples/vision/classification/paddleclas/rv1126/cpp/infer.cc
View File

@@ -23,7 +23,7 @@ void InitAndInfer(const std::string& model_dir, const std::string& image_file) {
auto model_file = model_dir + sep + "inference.pdmodel";
auto params_file = model_dir + sep + "inference.pdiparams";
auto config_file = model_dir + sep + "inference_cls.yaml";
fastdeploy::vision::EnableFlyCV();
fastdeploy::RuntimeOption option;
option.UseTimVX();

6
examples/vision/classification/paddleclas/serving/README.md
View File

@@ -26,12 +26,12 @@ mv ResNet50_vd_infer/inference.pdiparams models/runtime/1/model.pdiparams
# 拉取fastdeploy镜像(x.y.z为镜像版本号，需参照serving文档替换为数字)
# GPU镜像
docker pull paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
# CPU镜像
docker pull paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
# 运行容器.容器名字为 fd_serving, 并挂载当前目录为容器的 /serving 目录
nvidia-docker run -it --net=host --name fd_serving -v `pwd`/:/serving paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
nvidia-docker run -it --net=host --name fd_serving -v `pwd`/:/serving registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
# 启动服务(不设置CUDA_VISIBLE_DEVICES环境变量，会拥有所有GPU卡的调度权限)
CUDA_VISIBLE_DEVICES=0 fastdeployserver --model-repository=/serving/models --backend-config=python,shm-default-byte-size=10485760

14
examples/vision/detection/fastestdet/cpp/CMakeLists.txt Normal file
View File

@@ -0,0 +1,14 @@
PROJECT(infer_demo C CXX)
CMAKE_MINIMUM_REQUIRED (VERSION 3.10)
# Specifies the path to the fastdeploy library after you have downloaded it
option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.")
include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake)
# Include the FastDeploy dependency header file
include_directories(${FASTDEPLOY_INCS})
add_executable(infer_demo ${PROJECT_SOURCE_DIR}/infer.cc)
# Add the FastDeploy library dependency
target_link_libraries(infer_demo ${FASTDEPLOY_LIBS})

87
examples/vision/detection/fastestdet/cpp/README.md Normal file
View File

@@ -0,0 +1,87 @@
# FastestDet C++部署示例
本目录下提供`infer.cc`快速完成FastestDet在CPU/GPU，以及GPU上通过TensorRT加速部署的示例。
在部署前，需确认以下两个步骤
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
- 2. 根据开发环境，下载预编译部署库和samples代码，参考[FastDeploy预编译库](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
以Linux上CPU推理为例，在本目录执行如下命令即可完成编译测试
```bash
mkdir build
cd build
wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-1.0.3.tgz
tar xvf fastdeploy-linux-x64-1.0.3.tgz
cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-1.0.3
make -j
#下载官方转换好的FastestDet模型文件和测试图片
wget https://bj.bcebos.com/paddlehub/fastdeploy/FastestDet.onnx
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg
# CPU推理
./infer_demo FastestDet.onnx 000000014439.jpg 0
# GPU推理
./infer_demo FastestDet.onnx 000000014439.jpg 1
# GPU上TensorRT推理
./infer_demo FastestDet.onnx 000000014439.jpg 2
```
运行完成可视化结果如下图所示
<img width="640" src="https://user-images.githubusercontent.com/44280887/206176291-61eb118b-391b-4431-b79e-a393b9452138.jpg">
以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考:
- [如何在Windows中使用FastDeploy C++ SDK](../../../../../docs/cn/faq/use_sdk_on_windows.md)
## FastestDet C++接口
### FastestDet类
```c++
fastdeploy::vision::detection::FastestDet(
const string& model_file,
const string& params_file = "",
const RuntimeOption& runtime_option = RuntimeOption(),
const ModelFormat& model_format = ModelFormat::ONNX)
```
FastestDet模型加载和初始化，其中model_file为导出的ONNX模型格式。
**参数**
> * **model_file**(str): 模型文件路径
> * **params_file**(str): 参数文件路径，当模型格式为ONNX时，此参数传入空字符串即可
> * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置
> * **model_format**(ModelFormat): 模型格式，默认为ONNX格式
#### Predict函数
> ```c++
> FastestDet::Predict(cv::Mat* im, DetectionResult* result,
> float conf_threshold = 0.65,
> float nms_iou_threshold = 0.45)
> ```
>
> 模型预测接口，输入图像直接输出检测结果。
>
> **参数**
>
> > * **im**: 输入图像，注意需为HWC，BGR格式
> > * **result**: 检测结果，包括检测框，各个框的置信度, DetectionResult说明参考[视觉模型预测结果](../../../../../docs/api/vision_results/)
> > * **conf_threshold**: 检测框置信度过滤阈值
> > * **nms_iou_threshold**: NMS处理过程中iou阈值
### 类成员变量
#### 预处理参数
用户可按照自己的实际需求，修改下列预处理参数，从而影响最终的推理和部署效果
> > * **size**(vector&lt;int&gt;): 通过此参数修改预处理过程中resize的大小，包含两个整型元素，表示[width, height], 默认值为[352, 352]
- [模型介绍](../../)
- [Python部署](../python)
- [视觉模型预测结果](../../../../../docs/api/vision_results/)
- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)

105
examples/vision/detection/fastestdet/cpp/infer.cc Normal file
View File

@@ -0,0 +1,105 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "fastdeploy/vision.h"
void CpuInfer(const std::string& model_file, const std::string& image_file) {
auto model = fastdeploy::vision::detection::FastestDet(model_file);
if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl;
return;
}
auto im = cv::imread(image_file);
fastdeploy::vision::DetectionResult res;
if (!model.Predict(im, &res)) {
std::cerr << "Failed to predict." << std::endl;
return;
}
std::cout << res.Str() << std::endl;
auto vis_im = fastdeploy::vision::VisDetection(im, res);
cv::imwrite("vis_result.jpg", vis_im);
std::cout << "Visualized result saved in ./vis_result.jpg" << std::endl;
}
void GpuInfer(const std::string& model_file, const std::string& image_file) {
auto option = fastdeploy::RuntimeOption();
option.UseGpu();
auto model = fastdeploy::vision::detection::FastestDet(model_file, "", option);
if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl;
return;
}
auto im = cv::imread(image_file);
fastdeploy::vision::DetectionResult res;
if (!model.Predict(im, &res)) {
std::cerr << "Failed to predict." << std::endl;
return;
}
std::cout << res.Str() << std::endl;
auto vis_im = fastdeploy::vision::VisDetection(im, res);
cv::imwrite("vis_result.jpg", vis_im);
std::cout << "Visualized result saved in ./vis_result.jpg" << std::endl;
}
void TrtInfer(const std::string& model_file, const std::string& image_file) {
auto option = fastdeploy::RuntimeOption();
option.UseGpu();
option.UseTrtBackend();
option.SetTrtInputShape("images", {1, 3, 352, 352});
auto model = fastdeploy::vision::detection::FastestDet(model_file, "", option);
if (!model.Initialized()) {
std::cerr << "Failed to initialize." << std::endl;
return;
}
auto im = cv::imread(image_file);
fastdeploy::vision::DetectionResult res;
if (!model.Predict(im, &res)) {
std::cerr << "Failed to predict." << std::endl;
return;
}
std::cout << res.Str() << std::endl;
auto vis_im = fastdeploy::vision::VisDetection(im, res);
cv::imwrite("vis_result.jpg", vis_im);
std::cout << "Visualized result saved in ./vis_result.jpg" << std::endl;
}
int main(int argc, char* argv[]) {
if (argc < 4) {
std::cout << "Usage: infer_demo path/to/model path/to/image run_option, "
"e.g ./infer_model ./FastestDet.onnx ./test.jpeg 0"
<< std::endl;
std::cout << "The data type of run_option is int, 0: run with cpu; 1: run "
"with gpu; 2: run with gpu and use tensorrt backend."
<< std::endl;
return -1;
}
if (std::atoi(argv[3]) == 0) {
CpuInfer(argv[1], argv[2]);
} else if (std::atoi(argv[3]) == 1) {
GpuInfer(argv[1], argv[2]);
} else if (std::atoi(argv[3]) == 2) {
TrtInfer(argv[1], argv[2]);
}
return 0;
}

74
examples/vision/detection/fastestdet/python/README.md Normal file
View File

@@ -0,0 +1,74 @@
# FastestDet Python部署示例
在部署前，需确认以下两个步骤
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
- 2. FastDeploy Python whl包安装，参考[FastDeploy Python安装](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
本目录下提供`infer.py`快速完成FastestDet在CPU/GPU，以及GPU上通过TensorRT加速部署的示例。执行如下脚本即可完成
```bash
#下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd examples/vision/detection/fastestdet/python/
#下载fastestdet模型文件和测试图片
wget https://bj.bcebos.com/paddlehub/fastdeploy/FastestDet.onnx
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg
# CPU推理
python infer.py --model FastestDet.onnx --image 000000014439.jpg --device cpu
# GPU推理
python infer.py --model FastestDet.onnx --image 000000014439.jpg --device gpu
# GPU上使用TensorRT推理
python infer.py --model FastestDet.onnx --image 000000014439.jpg --device gpu --use_trt True
```
运行完成可视化结果如下图所示
<img width="640" src="https://user-images.githubusercontent.com/44280887/206176291-61eb118b-391b-4431-b79e-a393b9452138.jpg">
## FastestDet Python接口
```python
fastdeploy.vision.detection.FastestDet(model_file, params_file=None, runtime_option=None, model_format=ModelFormat.ONNX)
```
FastestDet模型加载和初始化，其中model_file为导出的ONNX模型格式
**参数**
> * **model_file**(str): 模型文件路径
> * **params_file**(str): 参数文件路径，当模型格式为ONNX格式时，此参数无需设定
> * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置
> * **model_format**(ModelFormat): 模型格式，默认为ONNX
### predict函数
> ```python
> FastestDet.predict(image_data)
> ```
>
> 模型预测接口，输入图像直接输出检测结果。
>
> **参数**
>
> > * **image_data**(np.ndarray): 输入数据，注意需为HWC，BGR格式
> **返回**
>
> > 返回`fastdeploy.vision.DetectionResult`结构体，结构体说明参考文档[视觉模型预测结果](../../../../../docs/api/vision_results/)
### 类成员属性
#### 预处理参数
用户可按照自己的实际需求，修改下列预处理参数，从而影响最终的推理和部署效果
> > * **size**(list[int]): 通过此参数修改预处理过程中resize的大小，包含两个整型元素，表示[width, height], 默认值为[352, 352]
## 其它文档
- [FastestDet 模型介绍](..)
- [FastestDet C++部署](../cpp)
- [模型预测结果说明](../../../../../docs/api/vision_results/)
- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)

51
examples/vision/detection/fastestdet/python/infer.py Normal file
View File

@@ -0,0 +1,51 @@
import fastdeploy as fd
import cv2
def parse_arguments():
import argparse
import ast
parser = argparse.ArgumentParser()
parser.add_argument(
"--model", required=True, help="Path of FastestDet onnx model.")
parser.add_argument(
"--image", required=True, help="Path of test image file.")
parser.add_argument(
"--device",
type=str,
default='cpu',
help="Type of inference device, support 'cpu' or 'gpu'.")
parser.add_argument(
"--use_trt",
type=ast.literal_eval,
default=False,
help="Wether to use tensorrt.")
return parser.parse_args()
def build_option(args):
option = fd.RuntimeOption()
if args.device.lower() == "gpu":
option.use_gpu()
if args.use_trt:
option.use_trt_backend()
option.set_trt_input_shape("images", [1, 3, 352, 352])
return option
args = parse_arguments()
# Configure runtime and load model
runtime_option = build_option(args)
model = fd.vision.detection.FastestDet(args.model, runtime_option=runtime_option)
# Predict picture detection results
im = cv2.imread(args.image)
result = model.predict(im)
# Visualization of prediction results
vis_im = fd.vision.vis_detection(im, result)
cv2.imwrite("visualized_result.jpg", vis_im)
print("Visualized result save in ./visualized_result.jpg")

13
examples/vision/detection/paddledetection/python/serving/README_CN.md
View File

@@ -17,16 +17,9 @@ cd FastDeploy/examples/vision/detection/paddledetection/python/serving
wget https://bj.bcebos.com/paddlehub/fastdeploy/ppyoloe_crn_l_300e_coco.tgz
tar xvf ppyoloe_crn_l_300e_coco.tgz
# 安装uvicorn
pip install uvicorn
# 启动服务，可选择是否使用GPU和TensorRT，可根据uvicorn --help配置IP、端口号等
# CPU
MODEL_DIR=ppyoloe_crn_l_300e_coco DEVICE=cpu uvicorn server:app
# GPU
MODEL_DIR=ppyoloe_crn_l_300e_coco DEVICE=gpu uvicorn server:app
# GPU上使用TensorRT （注意：TensorRT推理第一次运行，有序列化模型的操作，有一定耗时，需要耐心等待）
MODEL_DIR=ppyoloe_crn_l_300e_coco DEVICE=gpu USE_TRT=true uvicorn server:app
# 启动服务，可修改server.py中的配置项来指定硬件、后端等
# 可通过--host、--port指定IP和端口号
fastdeploy simple_serving --app server:app
```
客户端：

14
examples/vision/detection/paddledetection/python/serving/README_EN.md
View File

@@ -17,17 +17,9 @@ cd FastDeploy/examples/vision/detection/paddledetection/python/serving
wget https://bj.bcebos.com/paddlehub/fastdeploy/ppyoloe_crn_l_300e_coco.tgz
tar xvf ppyoloe_crn_l_300e_coco.tgz
# Install uvicorn
pip install uvicorn
# Launch server, it's configurable to use GPU and TensorRT,
# and run 'uvicorn --help' to check how to specify IP and port, etc.
# CPU
MODEL_DIR=ppyoloe_crn_l_300e_coco DEVICE=cpu uvicorn server:app
# GPU
MODEL_DIR=ppyoloe_crn_l_300e_coco DEVICE=gpu uvicorn server:app
# GPU and TensorRT
MODEL_DIR=ppyoloe_crn_l_300e_coco DEVICE=gpu USE_TRT=true uvicorn server:app
# Launch server, change the configurations in server.py to select hardware, backend, etc.
# and use --host, --port to specify IP and port
fastdeploy simple_serving --app server:app
```
Client:

9
examples/vision/detection/paddledetection/python/serving/client.py
View File

@@ -1,20 +1,15 @@
import requests
import json
import cv2
import base64
import fastdeploy as fd
from fastdeploy.serving.utils import cv2_to_base64
if __name__ == '__main__':
url = "http://127.0.0.1:8000/fd/ppyoloe"
headers = {"Content-Type": "application/json"}
im = cv2.imread("000000014439.jpg")
data = {
"data": {
"image": fd.serving.utils.cv2_to_base64(im)
},
"parameters": {}
}
data = {"data": {"image": cv2_to_base64(im)}, "parameters": {}}
resp = requests.post(url=url, headers=headers, data=json.dumps(data))
if resp.status_code == 200:

16
examples/vision/detection/paddledetection/python/serving/server.py
View File

@@ -1,18 +1,16 @@
import fastdeploy as fd
from fastdeploy.serving.server import SimpleServer
import os
import logging
logging.getLogger().setLevel(logging.INFO)
# Get arguments from envrionment variables
model_dir = os.environ.get('MODEL_DIR')
device = os.environ.get('DEVICE', 'cpu')
use_trt = os.environ.get('USE_TRT', False)
# Prepare model, download from hub or use local dir
if model_dir is None:
model_dir = fd.download_model(name='ppyoloe_crn_l_300e_coco')
# Configurations
model_dir = 'ppyoloe_crn_l_300e_coco'
device = 'cpu'
use_trt = False
# Prepare model
model_file = os.path.join(model_dir, "model.pdmodel")
params_file = os.path.join(model_dir, "model.pdiparams")
config_file = os.path.join(model_dir, "infer_cfg.yml")
@@ -33,7 +31,7 @@ model_instance = fd.vision.detection.PPYOLOE(
runtime_option=option)
# Create server, setup REST API
app = fd.serving.SimpleServer()
app = SimpleServer()
app.register(
task_name="fd/ppyoloe",
model_handler=fd.serving.handler.VisionModelHandler,

16
examples/vision/detection/paddledetection/rv1126/cpp/CMakeLists.txt
View File

@@ -20,19 +20,11 @@ install(TARGETS infer_demo DESTINATION ./)
install(DIRECTORY models DESTINATION ./)
install(DIRECTORY images DESTINATION ./)
# install(DIRECTORY run_with_adb.sh DESTINATION ./)
file(GLOB FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/*)
install(PROGRAMS ${FASTDEPLOY_LIBS} DESTINATION lib)
file(GLOB OPENCV_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/opencv/lib/lib*)
install(PROGRAMS ${OPENCV_LIBS} DESTINATION lib)
file(GLOB PADDLELITE_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/lib*)
install(PROGRAMS ${PADDLELITE_LIBS} DESTINATION lib)
file(GLOB TIMVX_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/verisilicon_timvx/*)
install(PROGRAMS ${TIMVX_LIBS} DESTINATION lib)
file(GLOB_RECURSE FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/lib*.so*)
file(GLOB_RECURSE ALL_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/lib*.so*)
list(APPEND ALL_LIBS ${FASTDEPLOY_LIBS})
install(PROGRAMS ${ALL_LIBS} DESTINATION lib)
file(GLOB ADB_TOOLS run_with_adb.sh)
install(PROGRAMS ${ADB_TOOLS} DESTINATION ./)

2
examples/vision/detection/paddledetection/rv1126/cpp/infer_ppyoloe.cc
View File

@@ -24,7 +24,7 @@ void InitAndInfer(const std::string& model_dir, const std::string& image_file) {
auto params_file = model_dir + sep + "model.pdiparams";
auto config_file = model_dir + sep + "infer_cfg.yml";
auto subgraph_file = model_dir + sep + "subgraph.txt";
fastdeploy::vision::EnableFlyCV();
fastdeploy::RuntimeOption option;
option.UseTimVX();
option.SetLiteSubgraphPartitionPath(subgraph_file);

4
examples/vision/detection/paddledetection/serving/README.md
View File

@@ -37,13 +37,13 @@ cp models/runtime/ppyoloe_runtime_config.pbtxt models/runtime/config.pbtxt
# 拉取fastdeploy镜像(x.y.z为镜像版本号，需替换成fastdeploy版本数字)
# GPU镜像
docker pull paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
# CPU镜像
docker pull paddlepaddle/fastdeploy:z.y.z-cpu-only-21.10
# 运行容器.容器名字为 fd_serving, 并挂载当前目录为容器的 /serving 目录
nvidia-docker run -it --net=host --name fd_serving --shm-size="1g" -v `pwd`/:/serving paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
nvidia-docker run -it --net=host --name fd_serving --shm-size="1g" -v `pwd`/:/serving registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
# 启动服务(不设置CUDA_VISIBLE_DEVICES环境变量，会拥有所有GPU卡的调度权限)
CUDA_VISIBLE_DEVICES=0 fastdeployserver --model-repository=/serving/models

1
examples/vision/detection/yolov5/python/serving/README.md Symbolic link
View File

@@ -0,0 +1 @@
README_CN.md

36
examples/vision/detection/yolov5/python/serving/README_CN.md Normal file
View File

@@ -0,0 +1,36 @@
简体中文 | [English](README_EN.md)
# YOLOv5 Python轻量服务化部署示例
在部署前，需确认以下两个步骤
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
- 2. FastDeploy Python whl包安装，参考[FastDeploy Python安装](../../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
服务端：
```bash
# 下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/detection/yolov5/python/serving
# 下载模型文件
wget https://bj.bcebos.com/paddlehub/fastdeploy/yolov5s_infer.tar
tar xvf yolov5s_infer.tar
# 启动服务，可修改server.py中的配置项来指定硬件、后端等
# 可通过--host、--port指定IP和端口号
fastdeploy simple_serving --app server:app
```
客户端：
```bash
# 下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/detection/yolov5/python/serving
# 下载测试图片
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg
# 请求服务，获取推理结果（如有必要，请修改脚本中的IP和端口号）
python client.py
```

36
examples/vision/detection/yolov5/python/serving/README_EN.md Normal file
View File

@@ -0,0 +1,36 @@
English | [简体中文](README_CN.md)
# YOLOv5 Python Simple Serving Demo
## Environment
- 1. Prepare environment and install FastDeploy Python whl, refer to [download_prebuilt_libraries](../../../../../../docs/en/build_and_install/download_prebuilt_libraries.md)
Server:
```bash
# Download demo code
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/detection/yolov5/python/serving
# Download model
wget https://bj.bcebos.com/paddlehub/fastdeploy/yolov5s_infer.tar
tar xvf yolov5s_infer.tar
# Launch server, change the configurations in server.py to select hardware, backend, etc.
# and use --host, --port to specify IP and port
fastdeploy simple_serving --app server:app
```
Client:
```bash
# Download demo code
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/detection/yolov5/python/serving
# Download test image
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg
# Send request and get inference result (Please adapt the IP and port if necessary)
python client.py
```

23
examples/vision/detection/yolov5/python/serving/client.py Normal file
View File

@@ -0,0 +1,23 @@
import requests
import json
import cv2
import fastdeploy as fd
from fastdeploy.serving.utils import cv2_to_base64
if __name__ == '__main__':
url = "http://127.0.0.1:8000/fd/yolov5s"
headers = {"Content-Type": "application/json"}
im = cv2.imread("000000014439.jpg")
data = {"data": {"image": cv2_to_base64(im)}, "parameters": {}}
resp = requests.post(url=url, headers=headers, data=json.dumps(data))
if resp.status_code == 200:
r_json = json.loads(resp.json()["result"])
det_result = fd.vision.utils.json_to_detection(r_json)
vis_im = fd.vision.vis_detection(im, det_result, score_threshold=0.5)
cv2.imwrite("visualized_result.jpg", vis_im)
print("Visualized result save in ./visualized_result.jpg")
else:
print("Error code:", resp.status_code)
print(resp.text)

38
examples/vision/detection/yolov5/python/serving/server.py Normal file
View File

@@ -0,0 +1,38 @@
import fastdeploy as fd
from fastdeploy.serving.server import SimpleServer
import os
import logging
logging.getLogger().setLevel(logging.INFO)
# Configurations
model_dir = 'yolov5s_infer'
device = 'cpu'
use_trt = False
# Prepare model
model_file = os.path.join(model_dir, "model.pdmodel")
params_file = os.path.join(model_dir, "model.pdiparams")
# Setup runtime option to select hardware, backend, etc.
option = fd.RuntimeOption()
if device.lower() == 'gpu':
option.use_gpu()
if use_trt:
option.use_trt_backend()
option.set_trt_input_shape("images", [1, 3, 640, 640])
option.set_trt_cache_file('yolov5s.trt')
# Create model instance
model_instance = fd.vision.detection.YOLOv5(
model_file,
params_file,
runtime_option=option,
model_format=fd.ModelFormat.PADDLE)
# Create server, setup REST API
app = SimpleServer()
app.register(
task_name="fd/yolov5s",
model_handler=fd.serving.handler.VisionModelHandler,
predictor=model_instance)

15
examples/vision/detection/yolov5/rv1126/cpp/CMakeLists.txt
View File

@@ -21,17 +21,10 @@ install(TARGETS infer_demo DESTINATION ./)
install(DIRECTORY models DESTINATION ./)
install(DIRECTORY images DESTINATION ./)
file(GLOB FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/*)
install(PROGRAMS ${FASTDEPLOY_LIBS} DESTINATION lib)
file(GLOB OPENCV_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/opencv/lib/lib*)
install(PROGRAMS ${OPENCV_LIBS} DESTINATION lib)
file(GLOB PADDLELITE_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/lib*)
install(PROGRAMS ${PADDLELITE_LIBS} DESTINATION lib)
file(GLOB TIMVX_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/verisilicon_timvx/*)
install(PROGRAMS ${TIMVX_LIBS} DESTINATION lib)
file(GLOB_RECURSE FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/lib*.so*)
file(GLOB_RECURSE ALL_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/lib*.so*)
list(APPEND ALL_LIBS ${FASTDEPLOY_LIBS})
install(PROGRAMS ${ALL_LIBS} DESTINATION lib)
file(GLOB ADB_TOOLS run_with_adb.sh)
install(PROGRAMS ${ADB_TOOLS} DESTINATION ./)

2
examples/vision/detection/yolov5/rv1126/cpp/infer.cc
View File

@@ -23,7 +23,7 @@ void InitAndInfer(const std::string& model_dir, const std::string& image_file) {
auto model_file = model_dir + sep + "model.pdmodel";
auto params_file = model_dir + sep + "model.pdiparams";
auto subgraph_file = model_dir + sep + "subgraph.txt";
fastdeploy::vision::EnableFlyCV();
fastdeploy::RuntimeOption option;
option.UseTimVX();
option.SetLiteSubgraphPartitionPath(subgraph_file);

6
examples/vision/detection/yolov5/serving/README.md
View File

@@ -20,12 +20,12 @@ mv yolov5s.onnx models/runtime/1/model.onnx
# 拉取fastdeploy镜像(x.y.z为镜像版本号，需参照serving文档替换为数字)
# GPU镜像
docker pull paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
# CPU镜像
docker pull paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-cpu-only-21.10
# 运行容器.容器名字为 fd_serving, 并挂载当前目录为容器的 /yolov5_serving 目录
nvidia-docker run -it --net=host --name fd_serving -v `pwd`/:/yolov5_serving paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
nvidia-docker run -it --net=host --name fd_serving -v `pwd`/:/yolov5_serving registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
# 启动服务(不设置CUDA_VISIBLE_DEVICES环境变量，会拥有所有GPU卡的调度权限)
CUDA_VISIBLE_DEVICES=0 fastdeployserver --model-repository=/yolov5_serving/models --backend-config=python,shm-default-byte-size=10485760

12
examples/vision/keypointdetection/det_keypoint_unite/cpp/det_keypoint_unite_infer.cc
View File

@@ -55,10 +55,9 @@ void CpuInfer(const std::string& det_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测框结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.2);
cv::imwrite("vis_result.jpg", vis_im);
@@ -104,10 +103,9 @@ void KunlunXinInfer(const std::string& det_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测框结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.2);
cv::imwrite("vis_result.jpg", vis_im);
@@ -153,10 +151,9 @@ void GpuInfer(const std::string& det_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测框结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.2);
cv::imwrite("vis_result.jpg", vis_im);
@@ -210,10 +207,9 @@ void TrtInfer(const std::string& det_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测关键点结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.2);
cv::imwrite("vis_result.jpg", vis_im);

12
examples/vision/keypointdetection/tiny_pose/cpp/pptinypose_infer.cc
View File

@@ -42,10 +42,9 @@ void CpuInfer(const std::string& tinypose_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测框结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto tinypose_vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.5);
cv::imwrite("tinypose_vis_result.jpg", tinypose_vis_im);
@@ -75,10 +74,9 @@ void KunlunXinInfer(const std::string& tinypose_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测框结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto tinypose_vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.5);
cv::imwrite("tinypose_vis_result.jpg", tinypose_vis_im);
@@ -110,10 +108,9 @@ void GpuInfer(const std::string& tinypose_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测框结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto tinypose_vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.5);
cv::imwrite("tinypose_vis_result.jpg", tinypose_vis_im);
@@ -145,10 +142,9 @@ void TrtInfer(const std::string& tinypose_model_dir,
} else {
std::cout << "TinyPose Prediction Done!" << std::endl;
}
// 输出预测框结果
std::cout << res.Str() << std::endl;
// 可视化预测结果
auto tinypose_vis_im =
fastdeploy::vision::VisKeypointDetection(im, res, 0.5);
cv::imwrite("tinypose_vis_result.jpg", tinypose_vis_im);

1
examples/vision/ocr/PP-OCRv3/python/serving/README.md Symbolic link
View File

@@ -0,0 +1 @@
README_CN.md

44
examples/vision/ocr/PP-OCRv3/python/serving/README_CN.md Normal file
View File

@@ -0,0 +1,44 @@
简体中文 | [English](README_EN.md)
# PP-OCRv3 Python轻量服务化部署示例
在部署前，需确认以下两个步骤
- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
- 2. FastDeploy Python whl包安装，参考[FastDeploy Python安装](../../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
服务端：
```bash
# 下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/ocr/PP-OCRv3/python/serving
# 下载模型和字典文件
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar
tar xvf ch_PP-OCRv3_det_infer.tar
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar
tar -xvf ch_ppocr_mobile_v2.0_cls_infer.tar
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar
tar xvf ch_PP-OCRv3_rec_infer.tar
wget https://gitee.com/paddlepaddle/PaddleOCR/raw/release/2.6/ppocr/utils/ppocr_keys_v1.txt
# 启动服务，可修改server.py中的配置项来指定硬件、后端等
# 可通过--host、--port指定IP和端口号
fastdeploy simple_serving --app server:app
```
客户端：
```bash
# 下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/ocr/PP-OCRv3/python/serving
# 下载测试图片
wget https://gitee.com/paddlepaddle/PaddleOCR/raw/release/2.6/doc/imgs/12.jpg
# 请求服务，获取推理结果（如有必要，请修改脚本中的IP和端口号）
python client.py
```

43
examples/vision/ocr/PP-OCRv3/python/serving/README_EN.md Normal file
View File

@@ -0,0 +1,43 @@
English | [简体中文](README_CN.md)
# PP-OCRv3 Python Simple Serving Demo
## Environment
- 1. Prepare environment and install FastDeploy Python whl, refer to [download_prebuilt_libraries](../../../../../../docs/en/build_and_install/download_prebuilt_libraries.md)
Server:
```bash
# Download demo code
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/ocr/PP-OCRv3/python/serving
# Download models and labels
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar
tar xvf ch_PP-OCRv3_det_infer.tar
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar
tar -xvf ch_ppocr_mobile_v2.0_cls_infer.tar
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar
tar xvf ch_PP-OCRv3_rec_infer.tar
wget https://gitee.com/paddlepaddle/PaddleOCR/raw/release/2.6/ppocr/utils/ppocr_keys_v1.txt
# Launch server, change the configurations in server.py to select hardware, backend, etc.
# and use --host, --port to specify IP and port
fastdeploy simple_serving --app server:app
```
Client:
```bash
# Download demo code
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/ocr/PP-OCRv3/python/serving
# Download test image
wget https://gitee.com/paddlepaddle/PaddleOCR/raw/release/2.6/doc/imgs/12.jpg
# Send request and get inference result (Please adapt the IP and port if necessary)
python client.py
```

24
examples/vision/ocr/PP-OCRv3/python/serving/client.py Normal file
View File

@@ -0,0 +1,24 @@
import requests
import json
import cv2
import fastdeploy as fd
from fastdeploy.serving.utils import cv2_to_base64
if __name__ == '__main__':
url = "http://127.0.0.1:8000/fd/ppocrv3"
headers = {"Content-Type": "application/json"}
im = cv2.imread("12.jpg")
data = {"data": {"image": cv2_to_base64(im)}, "parameters": {}}
resp = requests.post(url=url, headers=headers, data=json.dumps(data))
if resp.status_code == 200:
r_json = json.loads(resp.json()["result"])
print(r_json)
ocr_result = fd.vision.utils.json_to_ocr(r_json)
vis_im = fd.vision.vis_ppocr(im, ocr_result)
cv2.imwrite("visualized_result.jpg", vis_im)
print("Visualized result save in ./visualized_result.jpg")
else:
print("Error code:", resp.status_code)
print(resp.text)

80
examples/vision/ocr/PP-OCRv3/python/serving/server.py Normal file
View File

@@ -0,0 +1,80 @@
import fastdeploy as fd
from fastdeploy.serving.server import SimpleServer
import os
import logging
logging.getLogger().setLevel(logging.INFO)
# Configurations
det_model_dir = 'ch_PP-OCRv3_det_infer'
cls_model_dir = 'ch_ppocr_mobile_v2.0_cls_infer'
rec_model_dir = 'ch_PP-OCRv3_rec_infer'
rec_label_file = 'ppocr_keys_v1.txt'
device = 'cpu'
# backend: ['paddle', 'trt'], you can also use other backends, but need to modify
# the runtime option below
backend = 'paddle'
# Prepare models
# Detection model
det_model_file = os.path.join(det_model_dir, "inference.pdmodel")
det_params_file = os.path.join(det_model_dir, "inference.pdiparams")
# Classification model
cls_model_file = os.path.join(cls_model_dir, "inference.pdmodel")
cls_params_file = os.path.join(cls_model_dir, "inference.pdiparams")
# Recognition model
rec_model_file = os.path.join(rec_model_dir, "inference.pdmodel")
rec_params_file = os.path.join(rec_model_dir, "inference.pdiparams")
# Setup runtime option to select hardware, backend, etc.
option = fd.RuntimeOption()
if device.lower() == 'gpu':
option.use_gpu()
if backend == 'trt':
option.use_trt_backend()
else:
option.use_paddle_infer_backend()
det_option = option
det_option.set_trt_input_shape("x", [1, 3, 64, 64], [1, 3, 640, 640],
[1, 3, 960, 960])
# det_option.set_trt_cache_file("det_trt_cache.trt")
print(det_model_file, det_params_file)
det_model = fd.vision.ocr.DBDetector(
det_model_file, det_params_file, runtime_option=det_option)
cls_batch_size = 1
rec_batch_size = 6
cls_option = option
cls_option.set_trt_input_shape("x", [1, 3, 48, 10],
[cls_batch_size, 3, 48, 320],
[cls_batch_size, 3, 48, 1024])
# cls_option.set_trt_cache_file("cls_trt_cache.trt")
cls_model = fd.vision.ocr.Classifier(
cls_model_file, cls_params_file, runtime_option=cls_option)
rec_option = option
rec_option.set_trt_input_shape("x", [1, 3, 48, 10],
[rec_batch_size, 3, 48, 320],
[rec_batch_size, 3, 48, 2304])
# rec_option.set_trt_cache_file("rec_trt_cache.trt")
rec_model = fd.vision.ocr.Recognizer(
rec_model_file, rec_params_file, rec_label_file, runtime_option=rec_option)
# Create PPOCRv3 pipeline
ppocr_v3 = fd.vision.ocr.PPOCRv3(
det_model=det_model, cls_model=cls_model, rec_model=rec_model)
ppocr_v3.cls_batch_size = cls_batch_size
ppocr_v3.rec_batch_size = rec_batch_size
# Create server, setup REST API
app = SimpleServer()
app.register(
task_name="fd/ppocrv3",
model_handler=fd.serving.handler.VisionModelHandler,
predictor=ppocr_v3)

16
examples/vision/segmentation/paddleseg/rv1126/cpp/CMakeLists.txt
View File

@@ -20,19 +20,11 @@ install(TARGETS infer_demo DESTINATION ./)
install(DIRECTORY models DESTINATION ./)
install(DIRECTORY images DESTINATION ./)
# install(DIRECTORY run_with_adb.sh DESTINATION ./)
file(GLOB FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/*)
install(PROGRAMS ${FASTDEPLOY_LIBS} DESTINATION lib)
file(GLOB OPENCV_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/opencv/lib/lib*)
install(PROGRAMS ${OPENCV_LIBS} DESTINATION lib)
file(GLOB PADDLELITE_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/lib*)
install(PROGRAMS ${PADDLELITE_LIBS} DESTINATION lib)
file(GLOB TIMVX_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/paddlelite/lib/verisilicon_timvx/*)
install(PROGRAMS ${TIMVX_LIBS} DESTINATION lib)
file(GLOB_RECURSE FASTDEPLOY_LIBS ${FASTDEPLOY_INSTALL_DIR}/lib/lib*.so*)
file(GLOB_RECURSE ALL_LIBS ${FASTDEPLOY_INSTALL_DIR}/third_libs/install/lib*.so*)
list(APPEND ALL_LIBS ${FASTDEPLOY_LIBS})
install(PROGRAMS ${ALL_LIBS} DESTINATION lib)
file(GLOB ADB_TOOLS run_with_adb.sh)
install(PROGRAMS ${ADB_TOOLS} DESTINATION ./)

2
examples/vision/segmentation/paddleseg/rv1126/cpp/infer.cc
View File

@@ -24,7 +24,7 @@ void InitAndInfer(const std::string& model_dir, const std::string& image_file) {
auto params_file = model_dir + sep + "model.pdiparams";
auto config_file = model_dir + sep + "deploy.yaml";
auto subgraph_file = model_dir + sep + "subgraph.txt";
fastdeploy::vision::EnableFlyCV();
fastdeploy::RuntimeOption option;
option.UseTimVX();
option.SetLiteSubgraphPartitionPath(subgraph_file);

6
fastdeploy/runtime.cc
View File

@@ -660,7 +660,11 @@ bool Runtime::Infer(std::vector<FDTensor>& input_tensors,
}
bool Runtime::Infer() {
return backend_->Infer(input_tensors_, &output_tensors_, false);
bool result = backend_->Infer(input_tensors_, &output_tensors_, false);
for (auto& tensor : output_tensors_) {
tensor.device_id = option.device_id;
}
return result;
}
void Runtime::BindInputTensor(const std::string& name, FDTensor& input) {

68
fastdeploy/text/uie/model.cc
View File

@@ -13,6 +13,8 @@
// limitations under the License.
#include "fastdeploy/text/uie/model.h"
#include "fastdeploy/function/concat.h"
#include "fastdeploy/function/split.h"
#include <algorithm>
#include <codecvt>
#include <locale>
@@ -42,8 +44,7 @@ static std::string DBC2SBC(const std::string& content) {
result.append(content.data() + content_utf8_len, content_char_width);
} else {
char dst_char[5] = {0};
uint32_t utf8_uint32 =
fast_tokenizer::utils::UnicodeToUTF8(content_char);
uint32_t utf8_uint32 = fast_tokenizer::utils::UnicodeToUTF8(content_char);
uint32_t utf8_char_count =
fast_tokenizer::utils::UnicodeToUTF8Char(utf8_uint32, dst_char);
result.append(dst_char, utf8_char_count);
@@ -164,12 +165,12 @@ UIEModel::UIEModel(const std::string& model_file,
const std::string& params_file,
const std::string& vocab_file, float position_prob,
size_t max_length, const std::vector<std::string>& schema,
int batch_size,
const fastdeploy::RuntimeOption& custom_option,
const fastdeploy::ModelFormat& model_format,
SchemaLanguage schema_language)
: max_length_(max_length),
position_prob_(position_prob),
schema_language_(schema_language),
: max_length_(max_length), position_prob_(position_prob),
schema_language_(schema_language), batch_size_(batch_size),
tokenizer_(vocab_file) {
runtime_option = custom_option;
runtime_option.model_format = model_format;
@@ -185,12 +186,12 @@ UIEModel::UIEModel(const std::string& model_file,
const std::string& params_file,
const std::string& vocab_file, float position_prob,
size_t max_length, const std::vector<SchemaNode>& schema,
int batch_size,
const fastdeploy::RuntimeOption& custom_option,
const fastdeploy::ModelFormat& model_format,
SchemaLanguage schema_language)
: max_length_(max_length),
position_prob_(position_prob),
schema_language_(schema_language),
: max_length_(max_length), position_prob_(position_prob),
schema_language_(schema_language), batch_size_(batch_size),
tokenizer_(vocab_file) {
runtime_option = custom_option;
runtime_option.model_format = model_format;
@@ -205,13 +206,12 @@ UIEModel::UIEModel(const std::string& model_file,
UIEModel::UIEModel(const std::string& model_file,
const std::string& params_file,
const std::string& vocab_file, float position_prob,
size_t max_length, const SchemaNode& schema,
size_t max_length, const SchemaNode& schema, int batch_size,
const fastdeploy::RuntimeOption& custom_option,
const fastdeploy::ModelFormat& model_format,
SchemaLanguage schema_language)
: max_length_(max_length),
position_prob_(position_prob),
schema_language_(schema_language),
: max_length_(max_length), position_prob_(position_prob),
schema_language_(schema_language), batch_size_(batch_size),
tokenizer_(vocab_file) {
runtime_option = custom_option;
runtime_option.model_format = model_format;
@@ -230,7 +230,8 @@ bool UIEModel::Initialize() {
void UIEModel::SetValidBackend() {
// TODO(zhoushunjie): Add lite backend in future
valid_cpu_backends = {Backend::ORT, Backend::OPENVINO, Backend::PDINFER, Backend::LITE};
valid_cpu_backends = {Backend::ORT, Backend::OPENVINO, Backend::PDINFER,
Backend::LITE};
valid_gpu_backends = {Backend::ORT, Backend::PDINFER, Backend::TRT};
}
@@ -253,8 +254,8 @@ void UIEModel::AutoSplitter(const std::vector<std::string>& texts,
size_t cnt_org = 0;
size_t cnt_short = 0;
for (auto& text : texts) {
auto text_len = fast_tokenizer::utils::GetUnicodeLenFromUTF8(
text.c_str(), text.length());
auto text_len = fast_tokenizer::utils::GetUnicodeLenFromUTF8(text.c_str(),
text.length());
if (text_len <= max_length) {
short_texts->push_back(text);
if (input_mapping->size() <= cnt_org) {
@@ -264,8 +265,7 @@ void UIEModel::AutoSplitter(const std::vector<std::string>& texts,
}
cnt_short += 1;
} else {
fast_tokenizer::pretokenizers::CharToBytesOffsetConverter converter(
text);
fast_tokenizer::pretokenizers::CharToBytesOffsetConverter converter(text);
for (size_t start = 0; start < text_len; start += max_length) {
size_t end = start + max_length;
if (end > text_len) {
@@ -742,13 +742,37 @@ void UIEModel::Predict(
std::vector<fast_tokenizer::core::Encoding> encodings;
Preprocess(short_input_texts, short_prompts, &encodings, &inputs);
// 3. Infer
std::vector<fastdeploy::FDTensor> outputs(NumOutputsOfRuntime());
if (!Infer(inputs, &outputs)) {
FDERROR << "Failed to inference while using model:" << ModelName()
<< "." << std::endl;
std::vector<std::vector<FDTensor>> inputs_vec(NumInputsOfRuntime());
int encoding_size = encodings.size();
std::vector<int> num_or_sections;
for (int i = 0; i < encoding_size; i += batch_size_) {
int actual_batch_size = (std::min)(batch_size_, encoding_size - i);
num_or_sections.push_back(actual_batch_size);
}
for (int i = 0; i < NumInputsOfRuntime(); ++i) {
function::Split(inputs[i], num_or_sections, &inputs_vec[i]);
}
// 3. Infer
std::vector<fastdeploy::FDTensor> outputs(NumOutputsOfRuntime());
std::vector<fastdeploy::FDTensor> outputs0, outputs1;
for (int i = 0; i < inputs_vec[0].size(); ++i) {
std::vector<fastdeploy::FDTensor> curr_inputs(NumInputsOfRuntime());
std::vector<fastdeploy::FDTensor> curr_outputs(NumOutputsOfRuntime());
for (int j = 0; j < NumInputsOfRuntime(); ++j) {
curr_inputs[j] = std::move(inputs_vec[j][i]);
curr_inputs[j].name = inputs[j].name;
}
if (!Infer(curr_inputs, &curr_outputs)) {
FDERROR << "Failed to inference while using model:" << ModelName()
<< "." << std::endl;
}
outputs0.push_back(curr_outputs[0]);
outputs1.push_back(curr_outputs[1]);
}
function::Concat(outputs0, &outputs[0]);
function::Concat(outputs1, &outputs[1]);
// 4. Convert FDTensor to UIEResult
Postprocess(outputs, encodings, short_input_texts, short_prompts,
input_mapping_with_short_text, &results_list);

31
fastdeploy/text/uie/model.h
View File

@@ -14,14 +14,14 @@
#pragma once
#include "fast_tokenizer/tokenizers/ernie_fast_tokenizer.h"
#include "fastdeploy/fastdeploy_model.h"
#include "fastdeploy/utils/unique_ptr.h"
#include <ostream>
#include <set>
#include <string>
#include <unordered_map>
#include <vector>
#include "fastdeploy/fastdeploy_model.h"
#include "fastdeploy/utils/unique_ptr.h"
#include "fast_tokenizer/tokenizers/ernie_fast_tokenizer.h"
using namespace paddlenlp;
@@ -99,6 +99,7 @@ struct FASTDEPLOY_DECL UIEModel : public FastDeployModel {
UIEModel(const std::string& model_file, const std::string& params_file,
const std::string& vocab_file, float position_prob,
size_t max_length, const std::vector<std::string>& schema,
int batch_size,
const fastdeploy::RuntimeOption& custom_option =
fastdeploy::RuntimeOption(),
const fastdeploy::ModelFormat& model_format =
@@ -106,7 +107,7 @@ struct FASTDEPLOY_DECL UIEModel : public FastDeployModel {
SchemaLanguage schema_language = SchemaLanguage::ZH);
UIEModel(const std::string& model_file, const std::string& params_file,
const std::string& vocab_file, float position_prob,
size_t max_length, const SchemaNode& schema,
size_t max_length, const SchemaNode& schema, int batch_size,
const fastdeploy::RuntimeOption& custom_option =
fastdeploy::RuntimeOption(),
const fastdeploy::ModelFormat& model_format =
@@ -115,6 +116,7 @@ struct FASTDEPLOY_DECL UIEModel : public FastDeployModel {
UIEModel(const std::string& model_file, const std::string& params_file,
const std::string& vocab_file, float position_prob,
size_t max_length, const std::vector<SchemaNode>& schema,
int batch_size,
const fastdeploy::RuntimeOption& custom_option =
fastdeploy::RuntimeOption(),
const fastdeploy::ModelFormat& model_format =
@@ -154,10 +156,10 @@ struct FASTDEPLOY_DECL UIEModel : public FastDeployModel {
std::vector<std::unordered_map<std::string, std::vector<UIEResult>>>*
results,
std::vector<std::vector<UIEResult*>>* new_relations);
void Predict(
const std::vector<std::string>& texts,
std::vector<std::unordered_map<std::string, std::vector<UIEResult>>>*
results);
void
Predict(const std::vector<std::string>& texts,
std::vector<std::unordered_map<std::string, std::vector<UIEResult>>>*
results);
protected:
using IDX_PROB = std::pair<int64_t, float>;
@@ -190,15 +192,16 @@ struct FASTDEPLOY_DECL UIEModel : public FastDeployModel {
const SPAN_SET& span_set,
const std::vector<fast_tokenizer::core::Offset>& offset_mapping,
std::vector<SpanIdx>* span_idxs, std::vector<float>* probs) const;
void ConvertSpanToUIEResult(
const std::vector<std::string>& texts,
const std::vector<std::string>& prompts,
const std::vector<std::vector<SpanIdx>>& span_idxs,
const std::vector<std::vector<float>>& probs,
std::vector<std::vector<UIEResult>>* results) const;
void
ConvertSpanToUIEResult(const std::vector<std::string>& texts,
const std::vector<std::string>& prompts,
const std::vector<std::vector<SpanIdx>>& span_idxs,
const std::vector<std::vector<float>>& probs,
std::vector<std::vector<UIEResult>>* results) const;
std::unique_ptr<Schema> schema_;
size_t max_length_;
float position_prob_;
int batch_size_;
SchemaLanguage schema_language_;
fast_tokenizer::tokenizers_impl::ErnieFastTokenizer tokenizer_;
};

51
fastdeploy/text/uie/uie_pybind.cc
View File

@@ -35,24 +35,29 @@ void BindUIE(pybind11::module& m) {
py::class_<text::UIEModel, FastDeployModel>(m, "UIEModel")
.def(py::init<std::string, std::string, std::string, float, size_t,
std::vector<std::string>, RuntimeOption, ModelFormat, text::SchemaLanguage>(),
std::vector<std::string>, int, RuntimeOption, ModelFormat,
text::SchemaLanguage>(),
py::arg("model_file"), py::arg("params_file"), py::arg("vocab_file"),
py::arg("position_prob"), py::arg("max_length"), py::arg("schema"),
py::arg("batch_size"),
py::arg("custom_option") = fastdeploy::RuntimeOption(),
py::arg("model_format") = fastdeploy::ModelFormat::PADDLE,
py::arg("schema_language") = text::SchemaLanguage::ZH)
.def(
py::init<std::string, std::string, std::string, float, size_t,
std::vector<text::SchemaNode>, RuntimeOption, ModelFormat, text::SchemaLanguage>(),
py::arg("model_file"), py::arg("params_file"), py::arg("vocab_file"),
py::arg("position_prob"), py::arg("max_length"), py::arg("schema"),
py::arg("custom_option") = fastdeploy::RuntimeOption(),
py::arg("model_format") = fastdeploy::ModelFormat::PADDLE,
py::arg("schema_language") = text::SchemaLanguage::ZH)
.def(py::init<std::string, std::string, std::string, float, size_t,
text::SchemaNode, RuntimeOption, ModelFormat, text::SchemaLanguage>(),
std::vector<text::SchemaNode>, int, RuntimeOption,
ModelFormat, text::SchemaLanguage>(),
py::arg("model_file"), py::arg("params_file"), py::arg("vocab_file"),
py::arg("position_prob"), py::arg("max_length"), py::arg("schema"),
py::arg("batch_size"),
py::arg("custom_option") = fastdeploy::RuntimeOption(),
py::arg("model_format") = fastdeploy::ModelFormat::PADDLE,
py::arg("schema_language") = text::SchemaLanguage::ZH)
.def(py::init<std::string, std::string, std::string, float, size_t,
text::SchemaNode, int, RuntimeOption, ModelFormat,
text::SchemaLanguage>(),
py::arg("model_file"), py::arg("params_file"), py::arg("vocab_file"),
py::arg("position_prob"), py::arg("max_length"), py::arg("schema"),
py::arg("batch_size"),
py::arg("custom_option") = fastdeploy::RuntimeOption(),
py::arg("model_format") = fastdeploy::ModelFormat::PADDLE,
py::arg("schema_language") = text::SchemaLanguage::ZH)
@@ -60,23 +65,25 @@ void BindUIE(pybind11::module& m) {
static_cast<void (text::UIEModel::*)(
const std::vector<std::string>&)>(&text::UIEModel::SetSchema),
py::arg("schema"))
.def("set_schema", static_cast<void (text::UIEModel::*)(
const std::vector<text::SchemaNode>&)>(
&text::UIEModel::SetSchema),
.def("set_schema",
static_cast<void (text::UIEModel::*)(
const std::vector<text::SchemaNode>&)>(
&text::UIEModel::SetSchema),
py::arg("schema"))
.def("set_schema",
static_cast<void (text::UIEModel::*)(const text::SchemaNode&)>(
&text::UIEModel::SetSchema),
py::arg("schema"))
.def("predict",
[](text::UIEModel& self, const std::vector<std::string>& texts) {
std::vector<
std::unordered_map<std::string, std::vector<text::UIEResult>>>
results;
self.Predict(texts, &results);
return results;
},
py::arg("text"));
.def(
"predict",
[](text::UIEModel& self, const std::vector<std::string>& texts) {
std::vector<
std::unordered_map<std::string, std::vector<text::UIEResult>>>
results;
self.Predict(texts, &results);
return results;
},
py::arg("text"));
}
} // namespace fastdeploy

1
fastdeploy/vision.h
View File

@@ -22,6 +22,7 @@
#include "fastdeploy/vision/detection/contrib/scaledyolov4.h"
#include "fastdeploy/vision/detection/contrib/yolor.h"
#include "fastdeploy/vision/detection/contrib/yolov5/yolov5.h"
#include "fastdeploy/vision/detection/contrib/fastestdet/fastestdet.h"
#include "fastdeploy/vision/detection/contrib/yolov5lite.h"
#include "fastdeploy/vision/detection/contrib/yolov6.h"
#include "fastdeploy/vision/detection/contrib/yolov7/yolov7.h"

3
fastdeploy/vision/classification/ppcls/model.cc
View File

@@ -32,9 +32,10 @@ PaddleClasModel::PaddleClasModel(const std::string& model_file,
valid_ascend_backends = {Backend::LITE};
valid_kunlunxin_backends = {Backend::LITE};
valid_ipu_backends = {Backend::PDINFER};
} else if (model_format == ModelFormat::ONNX) {
} else {
valid_cpu_backends = {Backend::ORT, Backend::OPENVINO};
valid_gpu_backends = {Backend::ORT, Backend::TRT};
valid_rknpu_backends = {Backend::RKNPU2};
}
runtime_option = custom_option;

6
fastdeploy/vision/classification/ppcls/ppcls_pybind.cc
View File

@@ -36,6 +36,12 @@ void BindPaddleClas(pybind11::module& m) {
})
.def("use_gpu", [](vision::classification::PaddleClasPreprocessor& self, int gpu_id = -1) {
self.UseGpu(gpu_id);
})
.def("disable_normalize", [](vision::classification::PaddleClasPreprocessor& self) {
self.DisableNormalize();
})
.def("disable_permute", [](vision::classification::PaddleClasPreprocessor& self) {
self.DisablePermute();
});
pybind11::class_<vision::classification::PaddleClasPostprocessor>(

45
fastdeploy/vision/classification/ppcls/preprocessor.cc
View File

@@ -24,19 +24,19 @@ namespace vision {
namespace classification {
PaddleClasPreprocessor::PaddleClasPreprocessor(const std::string& config_file) {
FDASSERT(BuildPreprocessPipelineFromConfig(config_file),
this->config_file_ = config_file;
FDASSERT(BuildPreprocessPipelineFromConfig(),
"Failed to create PaddleClasPreprocessor.");
initialized_ = true;
}
bool PaddleClasPreprocessor::BuildPreprocessPipelineFromConfig(
const std::string& config_file) {
bool PaddleClasPreprocessor::BuildPreprocessPipelineFromConfig() {
processors_.clear();
YAML::Node cfg;
try {
cfg = YAML::LoadFile(config_file);
cfg = YAML::LoadFile(config_file_);
} catch (YAML::BadFile& e) {
FDERROR << "Failed to load yaml file " << config_file
FDERROR << "Failed to load yaml file " << config_file_
<< ", maybe you should check this file." << std::endl;
return false;
}
@@ -57,15 +57,19 @@ bool PaddleClasPreprocessor::BuildPreprocessPipelineFromConfig(
int height = op.begin()->second["size"].as<int>();
processors_.push_back(std::make_shared<CenterCrop>(width, height));
} else if (op_name == "NormalizeImage") {
auto mean = op.begin()->second["mean"].as<std::vector<float>>();
auto std = op.begin()->second["std"].as<std::vector<float>>();
auto scale = op.begin()->second["scale"].as<float>();
FDASSERT((scale - 0.00392157) < 1e-06 && (scale - 0.00392157) > -1e-06,
"Only support scale in Normalize be 0.00392157, means the pixel "
"is in range of [0, 255].");
processors_.push_back(std::make_shared<Normalize>(mean, std));
if (!disable_normalize) {
auto mean = op.begin()->second["mean"].as<std::vector<float>>();
auto std = op.begin()->second["std"].as<std::vector<float>>();
auto scale = op.begin()->second["scale"].as<float>();
FDASSERT((scale - 0.00392157) < 1e-06 && (scale - 0.00392157) > -1e-06,
"Only support scale in Normalize be 0.00392157, means the pixel "
"is in range of [0, 255].");
processors_.push_back(std::make_shared<Normalize>(mean, std));
}
} else if (op_name == "ToCHWImage") {
processors_.push_back(std::make_shared<HWC2CHW>());
if (!disable_permute) {
processors_.push_back(std::make_shared<HWC2CHW>());
}
} else {
FDERROR << "Unexcepted preprocess operator: " << op_name << "."
<< std::endl;
@@ -78,6 +82,21 @@ bool PaddleClasPreprocessor::BuildPreprocessPipelineFromConfig(
return true;
}
void PaddleClasPreprocessor::DisableNormalize() {
this->disable_normalize = true;
// the DisableNormalize function will be invalid if the configuration file is loaded during preprocessing
if (!BuildPreprocessPipelineFromConfig()) {
FDERROR << "Failed to build preprocess pipeline from configuration file." << std::endl;
}
}
void PaddleClasPreprocessor::DisablePermute() {
this->disable_permute = true;
// the DisablePermute function will be invalid if the configuration file is loaded during preprocessing
if (!BuildPreprocessPipelineFromConfig()) {
FDERROR << "Failed to build preprocess pipeline from configuration file." << std::endl;
}
}
void PaddleClasPreprocessor::UseGpu(int gpu_id) {
#ifdef WITH_GPU
use_cuda_ = true;

13
fastdeploy/vision/classification/ppcls/preprocessor.h
View File

@@ -46,13 +46,24 @@ class FASTDEPLOY_DECL PaddleClasPreprocessor {
bool WithGpu() { return use_cuda_; }
/// This function will disable normalize in preprocessing step.
void DisableNormalize();
/// This function will disable hwc2chw in preprocessing step.
void DisablePermute();
private:
bool BuildPreprocessPipelineFromConfig(const std::string& config_file);
bool BuildPreprocessPipelineFromConfig();
std::vector<std::shared_ptr<Processor>> processors_;
bool initialized_ = false;
bool use_cuda_ = false;
// GPU device id
int device_id_ = -1;
// for recording the switch of hwc2chw
bool disable_permute = false;
// for recording the switch of normalize
bool disable_normalize = false;
// read config file
std::string config_file_;
};
} // namespace classification

79
fastdeploy/vision/detection/contrib/fastestdet/fastestdet.cc Normal file
View File

@@ -0,0 +1,79 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "fastdeploy/vision/detection/contrib/fastestdet/fastestdet.h"
namespace fastdeploy {
namespace vision {
namespace detection {
FastestDet::FastestDet(const std::string& model_file, const std::string& params_file,
const RuntimeOption& custom_option,
const ModelFormat& model_format) {
if (model_format == ModelFormat::ONNX) {
valid_cpu_backends = {Backend::OPENVINO, Backend::ORT};
valid_gpu_backends = {Backend::ORT, Backend::TRT};
} else {
valid_cpu_backends = {Backend::PDINFER, Backend::ORT, Backend::LITE};
valid_gpu_backends = {Backend::PDINFER, Backend::ORT, Backend::TRT};
}
runtime_option = custom_option;
runtime_option.model_format = model_format;
runtime_option.model_file = model_file;
runtime_option.params_file = params_file;
initialized = Initialize();
}
bool FastestDet::Initialize() {
if (!InitRuntime()) {
FDERROR << "Failed to initialize fastdeploy backend." << std::endl;
return false;
}
return true;
}
bool FastestDet::Predict(const cv::Mat& im, DetectionResult* result) {
std::vector<DetectionResult> results;
if (!BatchPredict({im}, &results)) {
return false;
}
*result = std::move(results[0]);
return true;
}
bool FastestDet::BatchPredict(const std::vector<cv::Mat>& images, std::vector<DetectionResult>* results) {
std::vector<std::map<std::string, std::array<float, 2>>> ims_info;
std::vector<FDMat> fd_images = WrapMat(images);
if (!preprocessor_.Run(&fd_images, &reused_input_tensors_, &ims_info)) {
FDERROR << "Failed to preprocess the input image." << std::endl;
return false;
}
reused_input_tensors_[0].name = InputInfoOfRuntime(0).name;
if (!Infer(reused_input_tensors_, &reused_output_tensors_)) {
FDERROR << "Failed to inference by runtime." << std::endl;
return false;
}
if (!postprocessor_.Run(reused_output_tensors_, results, ims_info)) {
FDERROR << "Failed to postprocess the inference results by runtime." << std::endl;
return false;
}
return true;
}
} // namespace detection
} // namespace vision
} // namespace fastdeploy

76
fastdeploy/vision/detection/contrib/fastestdet/fastestdet.h Normal file
View File

@@ -0,0 +1,76 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved. //NOLINT
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#pragma once
#include "fastdeploy/fastdeploy_model.h"
#include "fastdeploy/vision/detection/contrib/fastestdet/preprocessor.h"
#include "fastdeploy/vision/detection/contrib/fastestdet/postprocessor.h"
namespace fastdeploy {
namespace vision {
namespace detection {
/*! @brief FastestDet model object used when to load a FastestDet model exported by FastestDet.
*/
class FASTDEPLOY_DECL FastestDet : public FastDeployModel {
public:
/** \brief Set path of model file and the configuration of runtime.
*
* \param[in] model_file Path of model file, e.g ./fastestdet.onnx
* \param[in] params_file Path of parameter file, e.g ppyoloe/model.pdiparams, if the model format is ONNX, this parameter will be ignored
* \param[in] custom_option RuntimeOption for inference, the default will use cpu, and choose the backend defined in "valid_cpu_backends"
* \param[in] model_format Model format of the loaded model, default is ONNX format
*/
FastestDet(const std::string& model_file, const std::string& params_file = "",
const RuntimeOption& custom_option = RuntimeOption(),
const ModelFormat& model_format = ModelFormat::ONNX);
std::string ModelName() const { return "fastestdet"; }
/** \brief Predict the detection result for an input image
*
* \param[in] img The input image data, comes from cv::imread(), is a 3-D array with layout HWC, BGR format
* \param[in] result The output detection result will be writen to this structure
* \return true if the prediction successed, otherwise false
*/
virtual bool Predict(const cv::Mat& img, DetectionResult* result);
/** \brief Predict the detection results for a batch of input images
*
* \param[in] imgs, The input image list, each element comes from cv::imread()
* \param[in] results The output detection result list
* \return true if the prediction successed, otherwise false
*/
virtual bool BatchPredict(const std::vector<cv::Mat>& imgs,
std::vector<DetectionResult>* results);
/// Get preprocessor reference of FastestDet
virtual FastestDetPreprocessor& GetPreprocessor() {
return preprocessor_;
}
/// Get postprocessor reference of FastestDet
virtual FastestDetPostprocessor& GetPostprocessor() {
return postprocessor_;
}
protected:
bool Initialize();
FastestDetPreprocessor preprocessor_;
FastestDetPostprocessor postprocessor_;
};
} // namespace detection
} // namespace vision
} // namespace fastdeploy

85
fastdeploy/vision/detection/contrib/fastestdet/fastestdet_pybind.cc Normal file
View File

@@ -0,0 +1,85 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "fastdeploy/pybind/main.h"
namespace fastdeploy {
void BindFastestDet(pybind11::module& m) {
pybind11::class_<vision::detection::FastestDetPreprocessor>(
m, "FastestDetPreprocessor")
.def(pybind11::init<>())
.def("run", [](vision::detection::FastestDetPreprocessor& self, std::vector<pybind11::array>& im_list) {
std::vector<vision::FDMat> images;
for (size_t i = 0; i < im_list.size(); ++i) {
images.push_back(vision::WrapMat(PyArrayToCvMat(im_list[i])));
}
std::vector<FDTensor> outputs;
std::vector<std::map<std::string, std::array<float, 2>>> ims_info;
if (!self.Run(&images, &outputs, &ims_info)) {
throw std::runtime_error("raise Exception('Failed to preprocess the input data in FastestDetPreprocessor.')");
}
for (size_t i = 0; i < outputs.size(); ++i) {
outputs[i].StopSharing();
}
return make_pair(outputs, ims_info);
})
.def_property("size", &vision::detection::FastestDetPreprocessor::GetSize, &vision::detection::FastestDetPreprocessor::SetSize);
pybind11::class_<vision::detection::FastestDetPostprocessor>(
m, "FastestDetPostprocessor")
.def(pybind11::init<>())
.def("run", [](vision::detection::FastestDetPostprocessor& self, std::vector<FDTensor>& inputs,
const std::vector<std::map<std::string, std::array<float, 2>>>& ims_info) {
std::vector<vision::DetectionResult> results;
if (!self.Run(inputs, &results, ims_info)) {
throw std::runtime_error("raise Exception('Failed to postprocess the runtime result in FastestDetPostprocessor.')");
}
return results;
})
.def("run", [](vision::detection::FastestDetPostprocessor& self, std::vector<pybind11::array>& input_array,
const std::vector<std::map<std::string, std::array<float, 2>>>& ims_info) {
std::vector<vision::DetectionResult> results;
std::vector<FDTensor> inputs;
PyArrayToTensorList(input_array, &inputs, /*share_buffer=*/true);
if (!self.Run(inputs, &results, ims_info)) {
throw std::runtime_error("raise Exception('Failed to postprocess the runtime result in FastestDetPostprocessor.')");
}
return results;
})
.def_property("conf_threshold", &vision::detection::FastestDetPostprocessor::GetConfThreshold, &vision::detection::FastestDetPostprocessor::SetConfThreshold)
.def_property("nms_threshold", &vision::detection::FastestDetPostprocessor::GetNMSThreshold, &vision::detection::FastestDetPostprocessor::SetNMSThreshold);
pybind11::class_<vision::detection::FastestDet, FastDeployModel>(m, "FastestDet")
.def(pybind11::init<std::string, std::string, RuntimeOption,
ModelFormat>())
.def("predict",
[](vision::detection::FastestDet& self, pybind11::array& data) {
auto mat = PyArrayToCvMat(data);
vision::DetectionResult res;
self.Predict(mat, &res);
return res;
})
.def("batch_predict", [](vision::detection::FastestDet& self, std::vector<pybind11::array>& data) {
std::vector<cv::Mat> images;
for (size_t i = 0; i < data.size(); ++i) {
images.push_back(PyArrayToCvMat(data[i]));
}
std::vector<vision::DetectionResult> results;
self.BatchPredict(images, &results);
return results;
})
.def_property_readonly("preprocessor", &vision::detection::FastestDet::GetPreprocessor)
.def_property_readonly("postprocessor", &vision::detection::FastestDet::GetPostprocessor);
}
} // namespace fastdeploy

132
fastdeploy/vision/detection/contrib/fastestdet/postprocessor.cc Normal file
View File

@@ -0,0 +1,132 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "fastdeploy/vision/detection/contrib/fastestdet/postprocessor.h"
#include "fastdeploy/vision/utils/utils.h"
namespace fastdeploy {
namespace vision {
namespace detection {
FastestDetPostprocessor::FastestDetPostprocessor() {
conf_threshold_ = 0.65;
nms_threshold_ = 0.45;
}
float FastestDetPostprocessor::Sigmoid(float x) {
return 1.0f / (1.0f + exp(-x));
}
float FastestDetPostprocessor::Tanh(float x) {
return 2.0f / (1.0f + exp(-2 * x)) - 1;
}
bool FastestDetPostprocessor::Run(
const std::vector<FDTensor> &tensors, std::vector<DetectionResult> *results,
const std::vector<std::map<std::string, std::array<float, 2>>> &ims_info) {
int batch = 1;
results->resize(batch);
for (size_t bs = 0; bs < batch; ++bs) {
(*results)[bs].Clear();
// output (1,85,22,22) CHW
const float* output = reinterpret_cast<const float*>(tensors[0].Data()) + bs * tensors[0].shape[1] * tensors[0].shape[2] * tensors[0].shape[3];
int output_h = tensors[0].shape[2]; // out map height
int output_w = tensors[0].shape[3]; // out map weight
auto iter_out = ims_info[bs].find("output_shape");
auto iter_ipt = ims_info[bs].find("input_shape");
FDASSERT(iter_out != ims_info[bs].end() && iter_ipt != ims_info[bs].end(),
"Cannot find input_shape or output_shape from im_info.");
float ipt_h = iter_ipt->second[0];
float ipt_w = iter_ipt->second[1];
// handle output boxes from out map
for (int h = 0; h < output_h; h++) {
for (int w = 0; w < output_w; w++) {
// object score
int obj_score_index = (h * output_w) + w;
float obj_score = output[obj_score_index];
// find max class
int category = 0;
float max_score = 0.0f;
int class_num = tensors[0].shape[1]-5;
for (size_t i = 0; i < class_num; i++) {
obj_score_index =((5 + i) * output_h * output_w) + (h * output_w) + w;
float cls_score = output[obj_score_index];
if (cls_score > max_score) {
max_score = cls_score;
category = i;
}
}
float score = pow(max_score, 0.4) * pow(obj_score, 0.6);
// score threshold
if (score <= conf_threshold_) {
continue;
}
if (score > conf_threshold_) {
// handle box x y w h
int x_offset_index = (1 * output_h * output_w) + (h * output_w) + w;
int y_offset_index = (2 * output_h * output_w) + (h * output_w) + w;
int box_width_index = (3 * output_h * output_w) + (h * output_w) + w;
int box_height_index = (4 * output_h * output_w) + (h * output_w) + w;
float x_offset = Tanh(output[x_offset_index]);
float y_offset = Tanh(output[y_offset_index]);
float box_width = Sigmoid(output[box_width_index]);
float box_height = Sigmoid(output[box_height_index]);
float cx = (w + x_offset) / output_w;
float cy = (h + y_offset) / output_h;
// convert from [x, y, w, h] to [x1, y1, x2, y2]
(*results)[bs].boxes.emplace_back(std::array<float, 4>{
cx - box_width / 2.0f,
cy - box_height / 2.0f,
cx + box_width / 2.0f,
cy + box_height / 2.0f});
(*results)[bs].label_ids.push_back(category);
(*results)[bs].scores.push_back(score);
}
}
}
if ((*results)[bs].boxes.size() == 0) {
return true;
}
// scale boxes to origin shape
for (size_t i = 0; i < (*results)[bs].boxes.size(); ++i) {
(*results)[bs].boxes[i][0] = ((*results)[bs].boxes[i][0]) * ipt_w;
(*results)[bs].boxes[i][1] = ((*results)[bs].boxes[i][1]) * ipt_h;
(*results)[bs].boxes[i][2] = ((*results)[bs].boxes[i][2]) * ipt_w;
(*results)[bs].boxes[i][3] = ((*results)[bs].boxes[i][3]) * ipt_h;
}
//NMS
utils::NMS(&((*results)[bs]), nms_threshold_);
//clip box
for (size_t i = 0; i < (*results)[bs].boxes.size(); ++i) {
(*results)[bs].boxes[i][0] = std::max((*results)[bs].boxes[i][0], 0.0f);
(*results)[bs].boxes[i][1] = std::max((*results)[bs].boxes[i][1], 0.0f);
(*results)[bs].boxes[i][2] = std::min((*results)[bs].boxes[i][2], ipt_w);
(*results)[bs].boxes[i][3] = std::min((*results)[bs].boxes[i][3], ipt_h);
}
}
return true;
}
} // namespace detection
} // namespace vision
} // namespace fastdeploy

67
fastdeploy/vision/detection/contrib/fastestdet/postprocessor.h Normal file
View File

@@ -0,0 +1,67 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#pragma once
#include "fastdeploy/vision/common/processors/transform.h"
#include "fastdeploy/vision/common/result.h"
namespace fastdeploy {
namespace vision {
namespace detection {
/*! @brief Postprocessor object for FastestDet serials model.
*/
class FASTDEPLOY_DECL FastestDetPostprocessor {
public:
/** \brief Create a postprocessor instance for FastestDet serials model
*/
FastestDetPostprocessor();
/** \brief Process the result of runtime and fill to DetectionResult structure
*
* \param[in] tensors The inference result from runtime
* \param[in] result The output result of detection
* \param[in] ims_info The shape info list, record input_shape and output_shape
* \return true if the postprocess successed, otherwise false
*/
bool Run(const std::vector<FDTensor>& tensors,
std::vector<DetectionResult>* results,
const std::vector<std::map<std::string, std::array<float, 2>>>& ims_info);
/// Set conf_threshold, default 0.65
void SetConfThreshold(const float& conf_threshold) {
conf_threshold_ = conf_threshold;
}
/// Get conf_threshold, default 0.65
float GetConfThreshold() const { return conf_threshold_; }
/// Set nms_threshold, default 0.45
void SetNMSThreshold(const float& nms_threshold) {
nms_threshold_ = nms_threshold;
}
/// Get nms_threshold, default 0.45
float GetNMSThreshold() const { return nms_threshold_; }
protected:
float conf_threshold_;
float nms_threshold_;
float Sigmoid(float x);
float Tanh(float x);
};
} // namespace detection
} // namespace vision
} // namespace fastdeploy

81
fastdeploy/vision/detection/contrib/fastestdet/preprocessor.cc Normal file
View File

@@ -0,0 +1,81 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "fastdeploy/vision/detection/contrib/fastestdet/preprocessor.h"
#include "fastdeploy/function/concat.h"
namespace fastdeploy {
namespace vision {
namespace detection {
FastestDetPreprocessor::FastestDetPreprocessor() {
size_ = {352, 352}; //{h,w}
}
bool FastestDetPreprocessor::Preprocess(FDMat* mat, FDTensor* output,
std::map<std::string, std::array<float, 2>>* im_info) {
// Record the shape of image and the shape of preprocessed image
(*im_info)["input_shape"] = {static_cast<float>(mat->Height()),
static_cast<float>(mat->Width())};
// process after image load
double ratio = (size_[0] * 1.0) / std::max(static_cast<float>(mat->Height()),
static_cast<float>(mat->Width()));
// fastestdet's preprocess steps
// 1. resize
// 2. convert_and_permute(swap_rb=false)
Resize::Run(mat, size_[0], size_[1]); //resize
std::vector<float> alpha = {1.0f / 255.0f, 1.0f / 255.0f, 1.0f / 255.0f};
std::vector<float> beta = {0.0f, 0.0f, 0.0f};
//convert to float and HWC2CHW
ConvertAndPermute::Run(mat, alpha, beta, false);
// Record output shape of preprocessed image
(*im_info)["output_shape"] = {static_cast<float>(mat->Height()),
static_cast<float>(mat->Width())};
mat->ShareWithTensor(output);
output->ExpandDim(0); // reshape to n, h, w, c
return true;
}
bool FastestDetPreprocessor::Run(std::vector<FDMat>* images, std::vector<FDTensor>* outputs,
std::vector<std::map<std::string, std::array<float, 2>>>* ims_info) {
if (images->size() == 0) {
FDERROR << "The size of input images should be greater than 0." << std::endl;
return false;
}
ims_info->resize(images->size());
outputs->resize(1);
// Concat all the preprocessed data to a batch tensor
std::vector<FDTensor> tensors(images->size());
for (size_t i = 0; i < images->size(); ++i) {
if (!Preprocess(&(*images)[i], &tensors[i], &(*ims_info)[i])) {
FDERROR << "Failed to preprocess input image." << std::endl;
return false;
}
}
if (tensors.size() == 1) {
(*outputs)[0] = std::move(tensors[0]);
} else {
function::Concat(tensors, &((*outputs)[0]), 0);
}
return true;
}
} // namespace detection
} // namespace vision
} // namespace fastdeploy

57
fastdeploy/vision/detection/contrib/fastestdet/preprocessor.h Normal file
View File

@@ -0,0 +1,57 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#pragma once
#include "fastdeploy/vision/common/processors/transform.h"
#include "fastdeploy/vision/common/result.h"
namespace fastdeploy {
namespace vision {
namespace detection {
/*! @brief Preprocessor object for FastestDet serials model.
*/
class FASTDEPLOY_DECL FastestDetPreprocessor {
public:
/** \brief Create a preprocessor instance for FastestDet serials model
*/
FastestDetPreprocessor();
/** \brief Process the input image and prepare input tensors for runtime
*
* \param[in] images The input image data list, all the elements are returned by cv::imread()
* \param[in] outputs The output tensors which will feed in runtime
* \param[in] ims_info The shape info list, record input_shape and output_shape
* \return true if the preprocess successed, otherwise false
*/
bool Run(std::vector<FDMat>* images, std::vector<FDTensor>* outputs,
std::vector<std::map<std::string, std::array<float, 2>>>* ims_info);
/// Set target size, tuple of (width, height), default size = {352, 352}
void SetSize(const std::vector<int>& size) { size_ = size; }
/// Get target size, tuple of (width, height), default size = {352, 352}
std::vector<int> GetSize() const { return size_; }
protected:
bool Preprocess(FDMat* mat, FDTensor* output,
std::map<std::string, std::array<float, 2>>* im_info);
// target size, tuple of (width, height), default size = {352, 352}
std::vector<int> size_;
};
} // namespace detection
} // namespace vision
} // namespace fastdeploy

2
fastdeploy/vision/detection/detection_pybind.cc
View File

@@ -22,6 +22,7 @@ void BindYOLOR(pybind11::module& m);
void BindYOLOv6(pybind11::module& m);
void BindYOLOv5Lite(pybind11::module& m);
void BindYOLOv5(pybind11::module& m);
void BindFastestDet(pybind11::module& m);
void BindYOLOX(pybind11::module& m);
void BindNanoDetPlus(pybind11::module& m);
void BindPPDet(pybind11::module& m);
@@ -39,6 +40,7 @@ void BindDetection(pybind11::module& m) {
BindYOLOv6(detection_module);
BindYOLOv5Lite(detection_module);
BindYOLOv5(detection_module);
BindFastestDet(detection_module);
BindYOLOX(detection_module);
BindNanoDetPlus(detection_module);
BindYOLOv7End2EndTRT(detection_module);

2
java/android/app/src/main/java/com/baidu/paddle/fastdeploy/app/examples/text/uie/UIEMainActivity.java
View File

@@ -204,7 +204,7 @@ public class UIEMainActivity extends Activity implements View.OnClickListener {
option.enableLiteFp16();
}
predictor.init(modelFile, paramsFile, vocabFile,
0.3f, 128, schemaTexts,
0.3f, 128, schemaTexts, 64,
option, SchemaLanguage.ZH);
}
}

129
java/android/fastdeploy/src/main/cpp/fastdeploy_jni/text/uie/uie_model_jni.cc
View File

@@ -12,12 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#include <jni.h> // NOLINT
#include "fastdeploy_jni/perf_jni.h" // NOLINT
#include "fastdeploy_jni/convert_jni.h" // NOLINT
#include "fastdeploy_jni/runtime_option_jni.h" // NOLINT
#include "fastdeploy_jni/text/text_results_jni.h" // NOLINT
#include "fastdeploy_jni/text/uie/uie_utils_jni.h" // NOLINT
#include "fastdeploy_jni/convert_jni.h" // NOLINT
#include "fastdeploy_jni/perf_jni.h" // NOLINT
#include "fastdeploy_jni/runtime_option_jni.h" // NOLINT
#include "fastdeploy_jni/text/text_results_jni.h" // NOLINT
#include "fastdeploy_jni/text/uie/uie_utils_jni.h" // NOLINT
#include <jni.h> // NOLINT
#ifdef ENABLE_TEXT
#include "fastdeploy/text.h" // NOLINT
#endif
@@ -32,16 +32,11 @@ extern "C" {
#endif
JNIEXPORT jlong JNICALL
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_bindNative(JNIEnv *env,
jobject thiz,
jstring model_file,
jstring params_file,
jstring vocab_file,
jfloat position_prob,
jint max_length,
jobjectArray schema,
jobject runtime_option,
jint schema_language) {
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_bindNative(
JNIEnv* env, jobject thiz, jstring model_file, jstring params_file,
jstring vocab_file, jfloat position_prob, jint max_length,
jobjectArray schema, jint batch_size, jobject runtime_option,
jint schema_language) {
#ifndef ENABLE_TEXT
return 0;
#else
@@ -51,18 +46,14 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_bindNative(JNIEnv *env,
auto c_position_prob = static_cast<jfloat>(position_prob);
auto c_max_length = static_cast<size_t>(max_length);
auto c_schema = fni::ConvertTo<std::vector<std::string>>(env, schema);
auto c_batch_size = static_cast<int>(batch_size);
auto c_runtime_option = fni::NewCxxRuntimeOption(env, runtime_option);
auto c_schema_language = static_cast<text::SchemaLanguage>(schema_language);
auto c_paddle_model_format = fastdeploy::ModelFormat::PADDLE;
auto c_model_ptr = new text::UIEModel(c_model_file,
c_params_file,
c_vocab_file,
c_position_prob,
c_max_length,
c_schema,
c_runtime_option,
c_paddle_model_format,
c_schema_language);
auto c_model_ptr = new text::UIEModel(
c_model_file, c_params_file, c_vocab_file, c_position_prob, c_max_length,
c_schema, c_batch_size, c_runtime_option, c_paddle_model_format,
c_schema_language);
INITIALIZED_OR_RETURN(c_model_ptr)
#ifdef ENABLE_RUNTIME_PERF
@@ -73,17 +64,15 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_bindNative(JNIEnv *env,
}
JNIEXPORT jobjectArray JNICALL
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_predictNative(JNIEnv *env,
jobject thiz,
jlong cxx_context,
jobjectArray texts) {
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_predictNative(
JNIEnv* env, jobject thiz, jlong cxx_context, jobjectArray texts) {
#ifndef ENABLE_TEXT
return NULL;
#else
if (cxx_context == 0) {
return NULL;
}
auto c_model_ptr = reinterpret_cast<text::UIEModel *>(cxx_context);
auto c_model_ptr = reinterpret_cast<text::UIEModel*>(cxx_context);
auto c_texts = fni::ConvertTo<std::vector<std::string>>(env, texts);
if (c_texts.empty()) {
LOGE("c_texts is empty!");
@@ -91,8 +80,8 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_predictNative(JNIEnv *env,
}
LOGD("c_texts: %s", fni::UIETextsStr(c_texts).c_str());
std::vector<std::unordered_map<
std::string, std::vector<text::UIEResult>>> c_results;
std::vector<std::unordered_map<std::string, std::vector<text::UIEResult>>>
c_results;
auto t = fni::GetCurrentTime();
c_model_ptr->Predict(c_texts, &c_results);
@@ -107,50 +96,46 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_predictNative(JNIEnv *env,
// Push results to HashMap array
const char* j_hashmap_put_signature =
"(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;";
const jclass j_hashmap_clazz = env->FindClass(
"java/util/HashMap");
const jclass j_uie_result_clazz = env->FindClass(
"com/baidu/paddle/fastdeploy/text/UIEResult");
const jclass j_hashmap_clazz = env->FindClass("java/util/HashMap");
const jclass j_uie_result_clazz =
env->FindClass("com/baidu/paddle/fastdeploy/text/UIEResult");
// Get HashMap method id
const jmethodID j_hashmap_init = env->GetMethodID(
j_hashmap_clazz, "<init>", "()V");
const jmethodID j_hashmap_put = env->GetMethodID(
j_hashmap_clazz,"put", j_hashmap_put_signature);
const jmethodID j_hashmap_init =
env->GetMethodID(j_hashmap_clazz, "<init>", "()V");
const jmethodID j_hashmap_put =
env->GetMethodID(j_hashmap_clazz, "put", j_hashmap_put_signature);
const int c_uie_result_hashmap_size = c_results.size();
jobjectArray j_hashmap_uie_result_arr = env->NewObjectArray(
c_uie_result_hashmap_size, j_hashmap_clazz, NULL);
jobjectArray j_hashmap_uie_result_arr =
env->NewObjectArray(c_uie_result_hashmap_size, j_hashmap_clazz, NULL);
for (int i = 0; i < c_uie_result_hashmap_size; ++i) {
auto& curr_c_uie_result_map = c_results[i];
// Convert unordered_map<string, vector<UIEResult>>
// -> HashMap<String, UIEResult[]>
jobject curr_j_uie_result_hashmap = env->NewObject(
j_hashmap_clazz, j_hashmap_init);
jobject curr_j_uie_result_hashmap =
env->NewObject(j_hashmap_clazz, j_hashmap_init);
for (auto&& curr_c_uie_result: curr_c_uie_result_map) {
for (auto&& curr_c_uie_result : curr_c_uie_result_map) {
const auto& curr_inner_c_uie_key = curr_c_uie_result.first;
jstring curr_inner_j_uie_key = fni::ConvertTo<jstring>(
env, curr_inner_c_uie_key); // Key of HashMap
jstring curr_inner_j_uie_key =
fni::ConvertTo<jstring>(env, curr_inner_c_uie_key); // Key of HashMap
if (curr_c_uie_result.second.size() > 0) {
// Value of HashMap: HashMap<String, UIEResult[]>
jobjectArray curr_inner_j_uie_result_values =
env->NewObjectArray(curr_c_uie_result.second.size(),
j_uie_result_clazz,
NULL);
jobjectArray curr_inner_j_uie_result_values = env->NewObjectArray(
curr_c_uie_result.second.size(), j_uie_result_clazz, NULL);
// Convert vector<UIEResult> -> Java UIEResult[]
for (int j = 0; j < curr_c_uie_result.second.size(); ++j) {
text::UIEResult* inner_c_uie_result = (
&(curr_c_uie_result.second[j]));
text::UIEResult* inner_c_uie_result =
(&(curr_c_uie_result.second[j]));
jobject curr_inner_j_uie_result_obj =
fni::NewUIEJavaResultFromCxx(
env, reinterpret_cast<void *>(inner_c_uie_result));
jobject curr_inner_j_uie_result_obj = fni::NewUIEJavaResultFromCxx(
env, reinterpret_cast<void*>(inner_c_uie_result));
env->SetObjectArrayElement(curr_inner_j_uie_result_values, j,
curr_inner_j_uie_result_obj);
@@ -159,14 +144,14 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_predictNative(JNIEnv *env,
// Set element of 'curr_j_uie_result_hashmap':
// HashMap<String, UIEResult[]>
env->CallObjectMethod(
curr_j_uie_result_hashmap, j_hashmap_put,
curr_inner_j_uie_key, curr_inner_j_uie_result_values);
env->CallObjectMethod(curr_j_uie_result_hashmap, j_hashmap_put,
curr_inner_j_uie_key,
curr_inner_j_uie_result_values);
env->DeleteLocalRef(curr_inner_j_uie_key);
env->DeleteLocalRef(curr_inner_j_uie_result_values);
} // end if
} // end for
} // end if
} // end for
// Set current HashMap<String, UIEResult[]> to HashMap[i]
env->SetObjectArrayElement(j_hashmap_uie_result_arr, i,
@@ -179,16 +164,15 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_predictNative(JNIEnv *env,
}
JNIEXPORT jboolean JNICALL
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_releaseNative(JNIEnv *env,
jobject thiz,
jlong cxx_context) {
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_releaseNative(
JNIEnv* env, jobject thiz, jlong cxx_context) {
#ifndef ENABLE_TEXT
return JNI_FALSE;
#else
if (cxx_context == 0) {
return JNI_FALSE;
}
auto c_model_ptr = reinterpret_cast<text::UIEModel *>(cxx_context);
auto c_model_ptr = reinterpret_cast<text::UIEModel*>(cxx_context);
PERF_TIME_OF_RUNTIME(c_model_ptr, -1)
delete c_model_ptr;
@@ -199,15 +183,14 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_releaseNative(JNIEnv *env,
JNIEXPORT jboolean JNICALL
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_setSchemaStringNative(
JNIEnv *env, jobject thiz, jlong cxx_context,
jobjectArray schema) {
JNIEnv* env, jobject thiz, jlong cxx_context, jobjectArray schema) {
#ifndef ENABLE_TEXT
return JNI_FALSE;
#else
if (cxx_context == 0) {
return JNI_FALSE;
}
auto c_model_ptr = reinterpret_cast<text::UIEModel *>(cxx_context);
auto c_model_ptr = reinterpret_cast<text::UIEModel*>(cxx_context);
auto c_schema = fni::ConvertTo<std::vector<std::string>>(env, schema);
if (c_schema.empty()) {
LOGE("c_schema is empty!");
@@ -221,8 +204,7 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_setSchemaStringNative(
JNIEXPORT jboolean JNICALL
Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_setSchemaNodeNative(
JNIEnv *env, jobject thiz, jlong cxx_context,
jobjectArray schema) {
JNIEnv* env, jobject thiz, jlong cxx_context, jobjectArray schema) {
#ifndef ENABLE_TEXT
return JNI_FALSE;
#else
@@ -236,15 +218,15 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_setSchemaNodeNative(
if (cxx_context == 0) {
return JNI_FALSE;
}
auto c_model_ptr = reinterpret_cast<text::UIEModel *>(cxx_context);
auto c_model_ptr = reinterpret_cast<text::UIEModel*>(cxx_context);
std::vector<text::SchemaNode> c_schema;
for (int i = 0; i < j_schema_size; ++i) {
jobject curr_j_schema_node = env->GetObjectArrayElement(schema, i);
text::SchemaNode curr_c_schema_node;
if (fni::AllocateUIECxxSchemaNodeFromJava(
env, curr_j_schema_node, reinterpret_cast<void *>(
&curr_c_schema_node))) {
env, curr_j_schema_node,
reinterpret_cast<void*>(&curr_c_schema_node))) {
c_schema.push_back(curr_c_schema_node);
}
env->DeleteLocalRef(curr_j_schema_node);
@@ -264,4 +246,3 @@ Java_com_baidu_paddle_fastdeploy_text_uie_UIEModel_setSchemaNodeNative(
#ifdef __cplusplus
}
#endif

14
java/android/fastdeploy/src/main/java/com/baidu/paddle/fastdeploy/text/uie/UIEModel.java
View File

@@ -22,7 +22,7 @@ public class UIEModel {
String vocabFile,
String[] schema) {
init_(modelFile, paramsFile, vocabFile, 0.5f, 128,
schema, new RuntimeOption(), SchemaLanguage.ZH);
schema, 64, new RuntimeOption(), SchemaLanguage.ZH);
}
// Constructor with custom runtime option
@@ -32,10 +32,11 @@ public class UIEModel {
float positionProb,
int maxLength,
String[] schema,
int batchSize,
RuntimeOption runtimeOption,
SchemaLanguage schemaLanguage) {
init_(modelFile, paramsFile, vocabFile, positionProb, maxLength,
schema, runtimeOption, schemaLanguage);
schema, batchSize, runtimeOption, schemaLanguage);
}
// Call init manually with label file
@@ -44,7 +45,7 @@ public class UIEModel {
String vocabFile,
String[] schema) {
return init_(modelFile, paramsFile, vocabFile, 0.5f, 128,
schema, new RuntimeOption(), SchemaLanguage.ZH);
schema, 64, new RuntimeOption(), SchemaLanguage.ZH);
}
public boolean init(String modelFile,
@@ -53,10 +54,11 @@ public class UIEModel {
float positionProb,
int maxLength,
String[] schema,
int batchSize,
RuntimeOption runtimeOption,
SchemaLanguage schemaLanguage) {
return init_(modelFile, paramsFile, vocabFile, positionProb, maxLength,
schema, runtimeOption, schemaLanguage);
schema, batchSize, runtimeOption, schemaLanguage);
}
public boolean release() {
@@ -103,6 +105,7 @@ public class UIEModel {
float positionProb,
int maxLength,
String[] schema,
int batchSize,
RuntimeOption runtimeOption,
SchemaLanguage schemaLanguage) {
if (!mInitialized) {
@@ -113,6 +116,7 @@ public class UIEModel {
positionProb,
maxLength,
schema,
batchSize,
runtimeOption,
schemaLanguage.ordinal()
);
@@ -130,6 +134,7 @@ public class UIEModel {
positionProb,
maxLength,
schema,
batchSize,
runtimeOption,
schemaLanguage.ordinal()
);
@@ -149,6 +154,7 @@ public class UIEModel {
float positionProb,
int maxLength,
String[] schema,
int batchSize,
RuntimeOption runtimeOption,
int schemaLanguage);

2
python/fastdeploy/serving/__init__.py
View File

@@ -12,5 +12,3 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import absolute_import
from .server import SimpleServer

8
python/fastdeploy/text/uie/__init__.py
View File

@@ -50,6 +50,7 @@ class UIEModel(FastDeployModel):
position_prob=0.5,
max_length=128,
schema=[],
batch_size=64,
runtime_option=RuntimeOption(),
model_format=ModelFormat.PADDLE,
schema_language=SchemaLanguage.ZH):
@@ -63,9 +64,10 @@ class UIEModel(FastDeployModel):
else:
assert "The type of schema should be list or dict."
schema_language = C.text.SchemaLanguage(schema_language)
self._model = C.text.UIEModel(
model_file, params_file, vocab_file, position_prob, max_length,
schema, runtime_option._option, model_format, schema_language)
self._model = C.text.UIEModel(model_file, params_file, vocab_file,
position_prob, max_length, schema,
batch_size, runtime_option._option,
model_format, schema_language)
assert self.initialized, "UIEModel initialize failed."
def set_schema(self, schema):

14
python/fastdeploy/vision/classification/ppcls/__init__.py
View File

@@ -42,6 +42,18 @@ class PaddleClasPreprocessor:
"""
return self._preprocessor.use_gpu(gpu_id)
def disable_normalize(self):
"""
This function will disable normalize in preprocessing step.
"""
self._preprocessor.disable_normalize()
def disable_permute(self):
"""
This function will disable hwc2chw in preprocessing step.
"""
self._preprocessor.disable_permute()
class PaddleClasPostprocessor:
def __init__(self, topk=1):
@@ -78,8 +90,6 @@ class PaddleClasModel(FastDeployModel):
"""
super(PaddleClasModel, self).__init__(runtime_option)
assert model_format == ModelFormat.PADDLE, "PaddleClasModel only support model format of ModelFormat.PADDLE now."
self._model = C.vision.classification.PaddleClasModel(
model_file, params_file, config_file, self._runtime_option,
model_format)

1
python/fastdeploy/vision/detection/__init__.py
View File

@@ -19,6 +19,7 @@ from .contrib.scaled_yolov4 import ScaledYOLOv4
from .contrib.nanodet_plus import NanoDetPlus
from .contrib.yolox import YOLOX
from .contrib.yolov5 import *
from .contrib.fastestdet import *
from .contrib.yolov5lite import YOLOv5Lite
from .contrib.yolov6 import YOLOv6
from .contrib.yolov7end2end_trt import YOLOv7End2EndTRT

149
python/fastdeploy/vision/detection/contrib/fastestdet.py Normal file
View File

@@ -0,0 +1,149 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import absolute_import
import logging
from .... import FastDeployModel, ModelFormat
from .... import c_lib_wrap as C
class FastestDetPreprocessor:
def __init__(self):
"""Create a preprocessor for FastestDet
"""
self._preprocessor = C.vision.detection.FastestDetPreprocessor()
def run(self, input_ims):
"""Preprocess input images for FastestDet
:param: input_ims: (list of numpy.ndarray)The input image
:return: list of FDTensor
"""
return self._preprocessor.run(input_ims)
@property
def size(self):
"""
Argument for image preprocessing step, the preprocess image size, tuple of (width, height), default size = [352, 352]
"""
return self._preprocessor.size
@size.setter
def size(self, wh):
assert isinstance(wh, (list, tuple)),\
"The value to set `size` must be type of tuple or list."
assert len(wh) == 2,\
"The value to set `size` must contatins 2 elements means [width, height], but now it contains {} elements.".format(
len(wh))
self._preprocessor.size = wh
class FastestDetPostprocessor:
def __init__(self):
"""Create a postprocessor for FastestDet
"""
self._postprocessor = C.vision.detection.FastestDetPostprocessor()
def run(self, runtime_results, ims_info):
"""Postprocess the runtime results for FastestDet
:param: runtime_results: (list of FDTensor)The output FDTensor results from runtime
:param: ims_info: (list of dict)Record input_shape and output_shape
:return: list of DetectionResult(If the runtime_results is predict by batched samples, the length of this list equals to the batch size)
"""
return self._postprocessor.run(runtime_results, ims_info)
@property
def conf_threshold(self):
"""
confidence threshold for postprocessing, default is 0.65
"""
return self._postprocessor.conf_threshold
@property
def nms_threshold(self):
"""
nms threshold for postprocessing, default is 0.45
"""
return self._postprocessor.nms_threshold
@conf_threshold.setter
def conf_threshold(self, conf_threshold):
assert isinstance(conf_threshold, float),\
"The value to set `conf_threshold` must be type of float."
self._postprocessor.conf_threshold = conf_threshold
@nms_threshold.setter
def nms_threshold(self, nms_threshold):
assert isinstance(nms_threshold, float),\
"The value to set `nms_threshold` must be type of float."
self._postprocessor.nms_threshold = nms_threshold
class FastestDet(FastDeployModel):
def __init__(self,
model_file,
params_file="",
runtime_option=None,
model_format=ModelFormat.ONNX):
"""Load a FastestDet model exported by FastestDet.
:param model_file: (str)Path of model file, e.g ./FastestDet.onnx
:param params_file: (str)Path of parameters file, e.g yolox/model.pdiparams, if the model_fomat is ModelFormat.ONNX, this param will be ignored, can be set as empty string
:param runtime_option: (fastdeploy.RuntimeOption)RuntimeOption for inference this model, if it's None, will use the default backend on CPU
:param model_format: (fastdeploy.ModelForamt)Model format of the loaded model
"""
super(FastestDet, self).__init__(runtime_option)
assert model_format == ModelFormat.ONNX, "FastestDet only support model format of ModelFormat.ONNX now."
self._model = C.vision.detection.FastestDet(
model_file, params_file, self._runtime_option, model_format)
assert self.initialized, "FastestDet initialize failed."
def predict(self, input_image):
"""Detect an input image
:param input_image: (numpy.ndarray)The input image data, 3-D array with layout HWC, BGR format
:return: DetectionResult
"""
assert input_image is not None, "Input image is None."
return self._model.predict(input_image)
def batch_predict(self, images):
assert len(images) == 1,"FastestDet is only support 1 image in batch_predict"
"""Classify a batch of input image
:param im: (list of numpy.ndarray) The input image list, each element is a 3-D array with layout HWC, BGR format
:return list of DetectionResult
"""
return self._model.batch_predict(images)
@property
def preprocessor(self):
"""Get FastestDetPreprocessor object of the loaded model
:return FastestDetPreprocessor
"""
return self._model.preprocessor
@property
def postprocessor(self):
"""Get FastestDetPostprocessor object of the loaded model
:return FastestDetPostprocessor
"""
return self._model.postprocessor

2
python/requirements.txt
View File

@@ -3,6 +3,6 @@ requests
tqdm
numpy
opencv-python
fastdeploy-tools==0.0.1
fastdeploy-tools>=0.0.1
pyyaml
fastapi

11
serving/Dockerfile
View File

@@ -12,6 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
ARG http_proxy
ARG https_proxy
FROM nvcr.io/nvidia/tritonserver:21.10-py3 as full
FROM nvcr.io/nvidia/tritonserver:21.10-py3-min
@@ -24,7 +27,10 @@ COPY serving/TensorRT-8.4.1.5 /opt/TensorRT-8.4.1.5
ENV TZ=Asia/Shanghai \
DEBIAN_FRONTEND=noninteractive \
DCGM_VERSION=2.2.9
DCGM_VERSION=2.2.9 \
http_proxy=$http_proxy \
https_proxy=$http_proxy
RUN apt-get update \
&& apt-key del 7fa2af80 \
&& wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb \
@@ -46,3 +52,6 @@ COPY build/fastdeploy_install /opt/fastdeploy/
ENV LD_LIBRARY_PATH="/opt/TensorRT-8.4.1.5/lib/:/opt/fastdeploy/lib:/opt/fastdeploy/third_libs/install/onnxruntime/lib:/opt/fastdeploy/third_libs/install/paddle2onnx/lib:/opt/fastdeploy/third_libs/install/tensorrt/lib:/opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib:/opt/fastdeploy/third_libs/install/paddle_inference/third_party/install/mkldnn/lib:/opt/fastdeploy/third_libs/install/paddle_inference/third_party/install/mklml/lib:/opt/fastdeploy/third_libs/install/openvino/runtime/lib:$LD_LIBRARY_PATH"
ENV PATH="/opt/tritonserver/bin:$PATH"
# unset proxy
ENV http_proxy=
ENV https_proxy=

10
serving/Dockerfile_cpu
View File

@@ -12,10 +12,15 @@
# See the License for the specific language governing permissions and
# limitations under the License.
ARG http_proxy
ARG https_proxy
FROM paddlepaddle/fastdeploy:21.10-cpu-only-min
ENV TZ=Asia/Shanghai \
DEBIAN_FRONTEND=noninteractive
DEBIAN_FRONTEND=noninteractive \
http_proxy=$http_proxy \
https_proxy=$http_proxy
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils libgomp1 ffmpeg libsm6 libxext6 \
&& python3 -m pip install -U pip \
@@ -30,3 +35,6 @@ COPY build/fastdeploy_install /opt/fastdeploy/
RUN mv /opt/tritonserver/bin/tritonserver /opt/tritonserver/bin/fastdeployserver
ENV LD_LIBRARY_PATH="/opt/fastdeploy/lib:/opt/fastdeploy/third_libs/install/onnxruntime/lib:/opt/fastdeploy/third_libs/install/paddle2onnx/lib:/opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib:/opt/fastdeploy/third_libs/install/paddle_inference/third_party/install/mkldnn/lib:/opt/fastdeploy/third_libs/install/paddle_inference/third_party/install/mklml/lib:/opt/fastdeploy/third_libs/install/openvino/runtime/lib:$LD_LIBRARY_PATH"
# unset proxy
ENV http_proxy=
ENV https_proxy=

111
tests/models/test_fastestdet.py Normal file
View File

@@ -0,0 +1,111 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from fastdeploy import ModelFormat
import fastdeploy as fd
import cv2
import os
import pickle
import numpy as np
import runtime_config as rc
def test_detection_fastestdet():
model_url = "https://bj.bcebos.com/paddlehub/fastdeploy/FastestDet.onnx"
input_url1 = "https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg"
input_url2 = "https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000570688.jpg"
result_url1 = "https://bj.bcebos.com/paddlehub/fastdeploy/fastestdet_result1.pkl"
fd.download(model_url, "resources")
fd.download(input_url1, "resources")
fd.download(input_url2, "resources")
fd.download(result_url1, "resources")
model_file = "resources/FastestDet.onnx"
model = fd.vision.detection.FastestDet(
model_file, runtime_option=rc.test_option)
with open("resources/fastestdet_result1.pkl", "rb") as f:
expect1 = pickle.load(f)
# compare diff
im1 = cv2.imread("./resources/000000014439.jpg")
print(expect1)
for i in range(3):
# test single predict
result1 = model.predict(im1)
diff_boxes_1 = np.fabs(
np.array(result1.boxes) - np.array(expect1["boxes"]))
diff_label_1 = np.fabs(
np.array(result1.label_ids) - np.array(expect1["label_ids"]))
diff_scores_1 = np.fabs(
np.array(result1.scores) - np.array(expect1["scores"]))
print(diff_boxes_1.max(), diff_boxes_1.mean())
assert diff_boxes_1.max(
) < 1e-04, "There's difference in detection boxes 1."
assert diff_label_1.max(
) < 1e-04, "There's difference in detection label 1."
assert diff_scores_1.max(
) < 1e-05, "There's difference in detection score 1."
def test_detection_fastestdet_runtime():
model_url = "https://bj.bcebos.com/paddlehub/fastdeploy/FastestDet.onnx"
input_url1 = "https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg"
result_url1 = "https://bj.bcebos.com/paddlehub/fastdeploy/fastestdet_result1.pkl"
fd.download(model_url, "resources")
fd.download(input_url1, "resources")
fd.download(result_url1, "resources")
model_file = "resources/FastestDet.onnx"
preprocessor = fd.vision.detection.FastestDetPreprocessor()
postprocessor = fd.vision.detection.FastestDetPostprocessor()
rc.test_option.set_model_path(model_file, model_format=ModelFormat.ONNX)
rc.test_option.use_openvino_backend()
runtime = fd.Runtime(rc.test_option)
with open("resources/fastestdet_result1.pkl", "rb") as f:
expect1 = pickle.load(f)
# compare diff
im1 = cv2.imread("./resources/000000014439.jpg")
for i in range(3):
# test runtime
input_tensors, ims_info = preprocessor.run([im1.copy()])
output_tensors = runtime.infer({"input.1": input_tensors[0]})
results = postprocessor.run(output_tensors, ims_info)
result1 = results[0]
diff_boxes_1 = np.fabs(
np.array(result1.boxes) - np.array(expect1["boxes"]))
diff_label_1 = np.fabs(
np.array(result1.label_ids) - np.array(expect1["label_ids"]))
diff_scores_1 = np.fabs(
np.array(result1.scores) - np.array(expect1["scores"]))
assert diff_boxes_1.max(
) < 1e-04, "There's difference in detection boxes 1."
assert diff_label_1.max(
) < 1e-04, "There's difference in detection label 1."
assert diff_scores_1.max(
) < 1e-05, "There's difference in detection score 1."
if __name__ == "__main__":
test_detection_fastestdet()
test_detection_fastestdet_runtime()

19
tools/common_tools/common_tools.py
View File

@@ -1,10 +1,12 @@
import argparse
import ast
import uvicorn
def argsparser():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('tools', choices=['compress', 'convert'])
parser.add_argument(
'tools', choices=['compress', 'convert', 'simple_serving'])
## argumentments for auto compression
parser.add_argument(
'--config_path',
@@ -69,6 +71,19 @@ def argsparser():
type=ast.literal_eval,
default=False,
help="Turn on code optimization")
## arguments for simple serving
parser.add_argument(
"--app",
type=str,
default="server:app",
help="Simple serving app string")
parser.add_argument(
"--host",
type=str,
default="127.0.0.1",
help="Simple serving host IP address")
parser.add_argument(
"--port", type=int, default=8000, help="Simple serving host port")
## arguments for other tools
return parser
@@ -116,6 +131,8 @@ def main():
except ImportError:
print(
"Model convert failed! Please check if you have installed it!")
if args.tools == "simple_serving":
uvicorn.run(args.app, host=args.host, port=args.port, app_dir='.')
if __name__ == '__main__':

10
tools/rknpu2/config/ResNet50_vd_infer_rknn.yaml Normal file
View File

@@ -0,0 +1,10 @@
model_path: ./ResNet50_vd_infer.onnx
output_folder: ./
target_platform: RK3588
normalize:
mean: [[0, 0, 0]]
std: [[1, 1, 1]]
outputs: []
outputs_nodes: []
do_quantization: False
dataset:

5
tools/setup.py
View File

@@ -3,13 +3,16 @@ import setuptools
long_description = "fastdeploy-tools is a toolkit for FastDeploy, including auto compression .etc.\n\n"
long_description += "Usage of auto compression: fastdeploy compress --config_path=./yolov7_tiny_qat_dis.yaml --method='QAT' --save_dir='./v7_qat_outmodel/' \n"
install_requires = ['uvicorn==0.16.0']
setuptools.setup(
name="fastdeploy-tools", # name of package
version="0.0.1", #version of package
version="0.0.2", #version of package
description="A toolkit for FastDeploy.",
long_description=long_description,
long_description_content_type="text/plain",
packages=setuptools.find_packages(),
install_requires=install_requires,
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: Apache Software License",