apps/FastDeploy
1
0
Fork 0
mirror of https://github.com/PaddlePaddle/FastDeploy.git synced 2025-10-06 17:17:14 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
Files
3589c0fa94c9007edd169c438eeda9a0cea75a8c
FastDeploy/examples/vision/detection/paddledetection/quantize
History
yunyaoXYY a231c9e7f3 [Quantization] Update quantized model deployment examples and update readme. (#377) 
* Add PaddleOCR Support

* Add PaddleOCR Support

* Add PaddleOCRv3 Support

* Add PaddleOCRv3 Support

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Add PaddleOCRv3 Support

* Add PaddleOCRv3 Supports

* Add PaddleOCRv3 Suport

* Fix Rec diff

* Remove useless functions

* Remove useless comments

* Add PaddleOCRv2 Support

* Add PaddleOCRv3 & PaddleOCRv2 Support

* remove useless parameters

* Add utils of sorting det boxes

* Fix code naming convention

* Fix code naming convention

* Fix code naming convention

* Fix bug in the Classify process

* Imporve OCR Readme

* Fix diff in Cls model

* Update Model Download Link in Readme

* Fix diff in PPOCRv2

* Improve OCR readme

* Imporve OCR readme

* Improve OCR readme

* Improve OCR readme

* Imporve OCR readme

* Improve OCR readme

* Fix conflict

* Add readme for OCRResult

* Improve OCR readme

* Add OCRResult readme

* Improve OCR readme

* Improve OCR readme

* Add Model Quantization Demo

* Fix Model Quantization Readme

* Fix Model Quantization Readme

* Add the function to do PTQ quantization

* Improve quant tools readme

* Improve quant tool readme

* Improve quant tool readme

* Add PaddleInference-GPU for OCR Rec model

* Add QAT method to fastdeploy-quantization tool

* Remove examples/slim for now

* Move configs folder

* Add Quantization Support for Classification Model

* Imporve ways of importing preprocess

* Upload YOLO Benchmark on readme

* Upload YOLO Benchmark on readme

* Upload YOLO Benchmark on readme

* Improve Quantization configs and readme

* Add support for multi-inputs model

* Add backends and params file for YOLOv7

* Add quantized model deployment support for YOLO series

* Fix YOLOv5 quantize readme

* Fix YOLO quantize readme

* Fix YOLO quantize readme

* Improve quantize YOLO readme

* Improve quantize YOLO readme

* Improve quantize YOLO readme

* Improve quantize YOLO readme

* Improve quantize YOLO readme

* Fix bug, change Fronted to ModelFormat

* Change Fronted to ModelFormat

* Add examples to deploy quantized paddleclas models

* Fix readme

* Add quantize Readme

* Add quantize Readme

* Add quantize Readme

* Modify readme of quantization tools

* Modify readme of quantization tools

* Improve quantization tools readme

* Improve quantization readme

* Improve PaddleClas quantized model deployment  readme

* Add PPYOLOE-l quantized deployment examples

* Improve quantization tools readme

* Improve Quantize Readme

* Fix conflicts

* Fix conflicts

* improve readme

* Improve quantization tools and readme

* Improve quantization tools and readme

* Add quantized deployment examples for PaddleSeg model

* Fix cpp readme

* Fix memory leak of reader_wrapper function

* Fix model file name in PaddleClas quantization examples

* Update Runtime and E2E benchmark

* Update Runtime and E2E benchmark

* Rename quantization tools to auto compression tools

* Remove PPYOLOE data when deployed on MKLDNN

* Fix readme

* Support PPYOLOE with OR without NMS and update readme

* Update Readme

* Update configs and readme

* Update configs and readme

* Add Paddle-TensorRT backend in quantized model deploy examples

* Support PPYOLOE+ series
2022-11-02 20:29:29 +08:00
..
cpp
[Quantization] Update quantized model deployment examples and update readme. (#377)
2022-11-02 20:29:29 +08:00
python
[Quantization] Update quantized model deployment examples and update readme. (#377)
2022-11-02 20:29:29 +08:00
README.md
[Quantization] Update quantized model deployment examples and update readme. (#377)
2022-11-02 20:29:29 +08:00

README.md

PaddleDetection 量化模型部署

FastDeploy已支持部署量化模型,并提供一键模型自动化压缩的工具. 用户可以使用一键模型自动化压缩工具,自行对模型量化后部署, 也可以直接下载FastDeploy提供的量化模型进行部署.

FastDeploy一键模型自动化压缩工具

FastDeploy 提供了一键模型自动化压缩工具, 能够简单地通过输入一个配置文件, 对模型进行量化. 详细教程请见: 一键模型自动化压缩工具

下载量化完成的PP-YOLOE-l模型

用户也可以直接下载下表中的量化模型进行部署.(点击模型名字即可下载)

Benchmark表格说明:

  • Rtuntime时延为模型在各种Runtime上的推理时延,包含CPU->GPU数据拷贝,GPU推理,GPU->CPU数据拷贝时间. 不包含模型各自的前后处理时间.
  • 端到端时延为模型在实际推理场景中的时延, 包含模型的前后处理.
  • 所测时延均为推理1000次后求得的平均值, 单位是毫秒.
  • INT8 + FP16 为在推理INT8量化模型的同时, 给Runtime 开启FP16推理选项
  • INT8 + FP16 + PM, 为在推理INT8量化模型和开启FP16的同时, 开启使用Pinned Memory的选项,可加速GPU->CPU数据拷贝的速度
  • 最大加速比, 为FP32时延除以INT8推理的最快时延,得到最大加速比.
  • 策略为量化蒸馏训练时, 采用少量无标签数据集训练得到量化模型, 并在全量验证集上验证精度, INT8精度并不代表最高的INT8精度.
  • CPU为Intel(R) Xeon(R) Gold 6271C, 所有测试中固定CPU线程数为1. GPU为Tesla T4, TensorRT版本8.4.15.

Runtime Benchmark

模型 推理后端 部署硬件 FP32 Runtime时延 INT8 Runtime时延 INT8 + FP16 Runtime时延 INT8+FP16+PM Runtime时延 最大加速比 FP32 mAP INT8 mAP 量化方式
ppyoloe_crn_l_300e_coco TensorRT GPU 27.90 6.39 6.44 5.95 4.67 51.4 50.7 量化蒸馏训练
ppyoloe_crn_l_300e_coco Paddle-TensorRT GPU 30.89 None 13.78 14.01 2.24 51.4 50.5 量化蒸馏训练
ppyoloe_crn_l_300e_coco ONNX Runtime CPU 1057.82 449.52 None None 2.35 51.4 50.0 量化蒸馏训练

NOTE:

  • TensorRT比Paddle-TensorRT快的原因是在runtime移除了multiclass_nms3算子

端到端 Benchmark

模型 推理后端 部署硬件 FP32 Runtime时延 INT8 Runtime时延 INT8 + FP16 Runtime时延 INT8+FP16+PM Runtime时延 最大加速比 FP32 mAP INT8 mAP 量化方式
ppyoloe_crn_l_300e_coco TensorRT GPU 35.75 15.42 20.70 20.85 2.32 51.4 50.7 量化蒸馏训练
ppyoloe_crn_l_300e_coco Paddle-TensorRT GPU 33.48 None 18.47 18.03 1.81 51.4 50.5 量化蒸馏训练
ppyoloe_crn_l_300e_coco ONNX Runtime CPU 1067.17 461.037 None None 2.31 51.4 50.0 量化蒸馏训练

详细部署文档