diff --git a/examples/application/js/README.md b/examples/application/js/README.md index 23fd53e09..25fed0f1e 100644 --- a/examples/application/js/README.md +++ b/examples/application/js/README.md @@ -1,3 +1,4 @@ +[English](README_en.md) | 简体中文 # 前端AI应用 @@ -19,7 +20,7 @@ |目标检测|[ScrewDetection、FaceDetection](./web_demo/src/pages/cv/detection/)| | |人像分割背景替换|[HumanSeg](./web_demo/src/pages/cv/segmentation/HumanSeg)|| |物体识别|[GestureRecognition、ItemIdentification](./web_demo/src/pages/cv/recognition/)|| -|PP-OCRv3|[TextDetection、TextRecognition](./web_demo/src/pages/cv/ocr/)|| +|OCR|[TextDetection、TextRecognition](./web_demo/src/pages/cv/ocr/)|| ## 微信小程序Demo使用 diff --git a/examples/application/js/README_en.md b/examples/application/js/README_en.md new file mode 100644 index 000000000..9c801a63f --- /dev/null +++ b/examples/application/js/README_en.md @@ -0,0 +1,40 @@ +English | [简体中文](README.md) + +# Front-end AI application + +The development of artificial intelligence technology has led to industrial upgrading in the fields of computer vision(CV) and natural language processing(NLP). In addition, the deployment of AI models in browsers to achieve front-end intelligence has already provided good basic conditions with the steady increase in computing power on PCs and mobile devices, iterative updates of model compression technologies, and the continuous emergence of various innovative needs. +In response to the difficulty of deploying AI deep learning models on the front-end, Baidu has open-sourced the Paddle.js front-end deep learning model deployment framework, which can easily deploy deep learning models into front-end projects. + +# Introduction of Paddle.js + +[Paddle.js](https://github.com/PaddlePaddle/Paddle.js) is a web sub-project of Baidu `PaddlePaddle`, an open source deep learning framework running in the browser. `Paddle.js` can load the deep learning model trained by `PaddlePaddle`, and convert it into a browser-friendly model through the model conversion tool `paddlejs-converter` of `Paddle.js`, which is easy to use for online reasoning and prediction. `Paddle.js` supports running in browsers of `WebGL/WebGPU/WebAssembly`, and can also run in the environment of Baidu applet and WeChat applet. + +Finally, we can launch AI functions in front-end application scenarios such as browsers and mini-program using `Paddle.js`, including but not limited to AI capabilities such as object detection, image segmentation, OCR, and item classification. + +## Web Demo + +Run the official demo reference [documentation](./WebDemo_en.md) directly in the browser + +|demo|web demo directory|visualization| +|-|-|-| +|object detection|[ScrewDetection、FaceDetection](./web_demo/src/pages/cv/detection/)| | +|human segmentation|[HumanSeg](./web_demo/src/pages/cv/segmentation/HumanSeg)|| +|classification|[GestureRecognition、ItemIdentification](./web_demo/src/pages/cv/recognition/)|| +|OCR|[TextDetection、TextRecognition](./web_demo/src/pages/cv/ocr/)|| + + +## Wechat Mini-program + +Run the official demo reference in the WeChat mini-program [document](./mini_program/README.md) + +|Name|Directory| +|-|-| +|OCR Text Detection| [ocrdetecXcx](./mini_program/ocrdetectXcx/) | +|OCR Text Recognition| [ocrXcx](./mini_program/ocrXcx/) | +|object detection| coming soon | +|Image segmentation | coming soon | +|Item Category| coming soon | + +## Contributor + +Thanks to Paddle Paddle Developer Expert (PPDE) Chen Qianhe (github: [chenqianhe](https://github.com/chenqianhe)) for the Web demo, mini-program. \ No newline at end of file diff --git a/examples/application/js/WebDemo.md b/examples/application/js/WebDemo.md index a1928edf4..068a4b001 100644 --- a/examples/application/js/WebDemo.md +++ b/examples/application/js/WebDemo.md @@ -1,3 +1,5 @@ +[English](WebDemo_en.md) | 简体中文 + # Web Demo介绍 - [简介](#0) diff --git a/examples/application/js/WebDemo_en.md b/examples/application/js/WebDemo_en.md new file mode 100644 index 000000000..b2a027b81 --- /dev/null +++ b/examples/application/js/WebDemo_en.md @@ -0,0 +1,176 @@ +English | [简体中文](WebDemo.md) + +# Introduction to Web Demo + +- [Introduction](#0) +- [1. Quick Start](#1) +- [2. npm package call](#2) +- [3. Model Replacement](#3) +- [4. Customize pre- and post-processing parameters](#4) +- [5. Other](#5) + + +## Introduction + +Based on [Paddle.js](https://github.com/PaddlePaddle/Paddle.js), this project implements computer vision tasks such as target detection, portrait segmentation, OCR, and item classification in the browser. + + +|demo name|web demo component|source directory|npm package| +|-|-|-|-| +|Face Detection|[FaceDetection](./web_demo/src/pages/cv/detection/FaceDetection/)| [facedetect](./package/packages/paddlejs-models/facedetect)|[@paddle-js-models/ facedetect](https://www.npmjs.com/package/@paddle-js-models/facedetect)| +|Screw Detection|[ScrewDetection](./web_demo/src/pages/cv/detection/ScrewDetection)| [detect](./package/packages/paddlejs-models/detect)|[@paddle-js-models/detect] (https://www.npmjs.com/package/@paddle-js-models/detect)| +|Portrait segmentation background replacement|[HumanSeg](./web_demo/src/pages/cv/segmentation/HumanSeg)|[humanseg](./package/packages/paddlejs-models/humanseg)|[@paddle-js-models/ humanseg](https://www.npmjs.com/package/@paddle-js-models/humanseg)| +|Gesture Recognition AI Guessing Shell|[GestureRecognition](./web_demo/src/pages/cv/recognition/GestureRecognition)|[gesture](./package/packages/paddlejs-models/gesture)|[@paddle-js- models/gesture](https://www.npmjs.com/package/@paddle-js-models/gesture)| +|1000 Item Identification|[ItemIdentification](./web_demo/src/pages/cv/recognition/ItemIdentification)|[mobilenet](./package/packages/paddlejs-models/mobilenet)|[@paddle-js-models/ mobilenet](https://www.npmjs.com/package/@paddle-js-models/mobilenet)| +|Text Detection|[TextDetection](./web_demo/src/pages/cv/ocr/TextDetection)|[ocrdetection](./package/packages/paddlejs-models/ocrdetection)|[@paddle-js-models/ocrdet] (https://www.npmjs.com/package/@paddle-js-models/ocrdet)| +|Text Recognition|[TextRecognition](./web_demo/src/pages/cv/ocr/TextRecognition)|[ocr](./package/packages/paddlejs-models/ocr)|[@paddle-js-models/ocr] (https://www.npmjs.com/package/@paddle-js-models/ocr)| + + + +## 1. Quick Start + +This section describes how to run the official demo directly in the browser. + +**1. Install Node.js** + +Download the `Node.js` installation package suitable for your platform from the `Node.js` official website https://nodejs.org/en/download/ and install it. + +**2. Install demo dependencies and start** +Execute the following command in the `./web_demo` directory: + +```` +# install dependencies +npm install +# start demo +npm run dev +```` + +Open the URL `http://localhost:5173/main/index.html` in the browser to quickly experience running computer vision tasks in the browser. + +![22416f4a3e7d63f950b838be3cd11e80](https://user-images.githubusercontent.com/26592129/196685868-93ab53bd-cb2e-44ff-a56b-50c1781b8679.jpg) + + + +## 2. npm package call + +This section introduces how to use npm packages. Each demo provides an easy-to-use interface. Users only need to initialize and upload images to get the results. The steps are as follows: +1. Call the module +2. Initialize the model +3. Pass in input, perform prediction + +Taking OCR as an example, in a front-end project, the `@paddle-js-models/ocr` package is used as follows: + +```` +// 1. Call the ocr module +import * as ocr from '@paddle-js-models/ocr'; + +// 2. Initialize the ocr model +await ocr.init(); + +// 3. Pass in an image of type HTMLImageElement as input and get the result +const res = await ocr.recognize(img); + +// Print the text coordinates and text content obtained by the OCR model +console.log(res.text); +console.log(res.points); +```` + + +## 3. Model replacement + +Due to the limitations of the front-end environment and computing resources, when deploying deep learning models on the front-end, we have stricter requirements on the performance of the models. In short, the models need to be lightweight enough. In theory, the smaller the input shape of the model and the smaller the model size, the smaller the flops of the corresponding model, and the smoother the front-end operation. Based on experience, the model storage deployed with `Paddle.js` should not exceed *5M* as much as possible, and the actual situation depends on the hardware and computing resources. + +In practical applications, models are often customized according to vertical scenarios, and the official demo supports modifying incoming parameters to replace models. + +Take the OCR demo as an example, [ocr.init() function](https://github.com/PaddlePaddle/FastDeploy/tree/develop/examples/application/js/package/packages/paddlejs-models/ocr/src/index .ts#L52), contains the default initialization model link, if you want to replace the model, please refer to the following steps. + +Step 1: Convert the model to js format: +```` +# Install paddlejsconverter +pip3 install paddlejsconverter +# Convert the model format, the input model is the inference model +paddlejsconverter --modelPath=./inference.pdmodel --paramPath=./inference.pdiparams --outputDir=./ --useGPUOpt=True +# Note: The useGPUOpt option is not enabled by default. If the model is used on the gpu backend (webgl/webgpu), enable useGPUOpt. If the model is running on (wasm/plain js), do not enable it. +```` + +After the export is successful, files such as `model.json chunk_1.dat` will appear in the local directory, which are the network structure and model parameter binary files corresponding to the js model. + +Step 2: Upload the exported js model to a server that supports cross-domain access. For the CORS configuration of the server, refer to the following figure: +![image](https://user-images.githubusercontent.com/26592129/196612669-5233137a-969c-49eb-b8c7-71bef5088686.png) + + +Step 3: Modify the code to replace the default model. Take the OCR demo as an example, modify the [model initialization code] in the OCR web demo (https://github.com/PaddlePaddle/FastDeploy/tree/develop/examples/application/js/web_demo/src/pages/cv/ocr/TextRecognition /TextRecognition.vue#L64), i.e. + +```` +await ocr.init(); +change into: +await ocr.init({modelPath: "https://js-models.bj.bcebos.com/PaddleOCR/PP-OCRv3/ch_PP-OCRv3_det_infer_js_960/model.json"}); # The first parameter passes in the new text Check dictionary type parameter +```` + +Re-execute the following command in the demo directory to experience the new model effect. +```` +npm run dev +```` + + +## 4. Customize pre- and post-processing parameters + +**Custom preprocessing parameters** + +In different computer vision tasks, different models may have different preprocessing parameters, such as mean, std, keep_ratio and other parameters. After replacing the model, the preprocessing parameters also need to be modified. A simple solution for customizing preprocessing parameters is provided in the npm package published by paddle.js. You only need to pass in custom parameters when calling the model initialization function. + +```` +# Default parameter initialization +await model.init(); + +Custom parameter initialization +const Config = {mean: [0.5, 0.5, 0.5], std: [0.5, 0.5, 0.5], keepratio: false}; +await model.init(Config); +```` + +Taking the OCR text detection demo as an example, to modify the mean and std parameters of the model preprocessing, you only need to pass in the custom mean and std parameters when the model is initialized. +```` +await ocr.init(); +change into: +const detConfig = {mean: [0.5, 0.5, 0.5], std: [0.5, 0.5, 0.5]}; +await ocr.init(detConfig); # The first parameter passes in the new text detection model link +```` + +**Custom postprocessing parameters** + +Similarly, the npm package published by paddle.js also provides a custom solution for post-processing parameters. + +```` +# run with default parameters +await model.predict(); + +# custom post-processing parameters +const postConfig = {thresh: 0.5}; +await model.predict(Config); +```` + +Take the OCR text detection demo as an example, modify the parameters of the text detection post-processing to achieve the effect of expanding the text detection frame, and modify the OCR web demo to execute the [model prediction code](https://github.com/PaddlePaddle/FastDeploy/tree/develop /examples/application/web_demo/src/pages/cv/ocr/TextRecognition/TextRecognition.vue#L99), ie: + +```` +const res = await ocr.recognize(img, { canvas: canvas.value }); +change into: +// Define hyperparameters, increase the unclip_ratio parameter from 1.5 to 3.5 +const detConfig = {shape: 960, thresh: 0.3, box_thresh: 0.6, unclip_ratio:3.5}; +const res = await ocr.recognize(img, { canvas: canvas.value }, detConfig); +```` + +Note: Different tasks have different post-processing parameters. For detailed parameters, please refer to the API in the npm package. + + +## 5. Others + +The converted model of `Paddle.js` can not only be used in the browser, but also can be run in the Baidu mini-program and WeChat mini-program environment. + +|Name|Directory| +|-|-| +|OCR Text Detection| [ocrdetecXcx](./mini_program/ocrdetectXcx/) | +|OCR Text Recognition| [ocrXcx](./mini_program/ocrXcx/) | +|target detection| coming soon | +| Image segmentation | coming soon | +|Item Category| coming soon | + diff --git a/examples/application/js/mini_program/README.md b/examples/application/js/mini_program/README.md index 92c6ae7a3..92772571b 100644 --- a/examples/application/js/mini_program/README.md +++ b/examples/application/js/mini_program/README.md @@ -1,3 +1,4 @@ +[English](README_en.md) | 简体中文 # Paddle.js微信小程序Demo diff --git a/examples/application/js/mini_program/README_en.md b/examples/application/js/mini_program/README_en.md new file mode 100644 index 000000000..1c476dd80 --- /dev/null +++ b/examples/application/js/mini_program/README_en.md @@ -0,0 +1,125 @@ +English | [中文](README.md) + +# Paddle.js WeChat mini-program Demo + +- [1. Introduction] (#1) +- [2. Project Start](#2) + * [2.1 Preparations](#21) + * [2.2 Startup steps](#22) + * [2.3 visualization](#23) +- [3. Model inference pipeline](#3) +- [4. FAQ](#4) + + +## 1 Introduction + + +This directory contains the text detection, text recognition mini-program demo, by using [Paddle.js](https://github.com/PaddlePaddle/Paddle.js) and [Paddle.js WeChat mini-program plugin](https:// mp.weixin.qq.com/wxopen/plugindevdoc?appid=wx7138a7bb793608c3&token=956931339&lang=zh_CN) to complete the text detection frame selection effect on the mini-program using the computing power of the user terminal. + + +## 2. Project start + + +### 2.1 Preparations +* [Apply for a WeChat mini-program account](https://mp.weixin.qq.com/) +* [WeChat Mini Program Developer Tools](https://developers.weixin.qq.com/miniprogram/dev/devtools/download.html) +* Front-end development environment preparation: node, npm +* Configure the server domain name in the mini-program management background, or open the developer tool [do not verify the legal domain name] + +For details, please refer to: https://mp.weixin.qq.com/wxamp/devprofile/get_profile?token=1132303404&lang=zh_CN) + + +### 2.2 Startup steps + +#### **1. Clone the demo code** +````sh +git clone https://github.com/PaddlePaddle/FastDeploy +cd FastDeploy/examples/application/js/mini_program +```` + +#### **2. Enter the mini-program directory and install dependencies** + +````sh +# Run the text recognition demo and enter the ocrXcx directory +cd ./ocrXcx && npm install +# Run the text detection demo and enter the ocrdetectXcx directory +# cd ./ocrdetectXcx && npm install +```` + +#### **3. WeChat mini-program import code** +Open WeChat Developer Tools --> Import --> Select a directory and enter relevant information + +#### **4. Add Paddle.js WeChat mini-program plugin** +Mini Program Management Interface --> Settings --> Third Party Settings --> Plugin Management --> Add Plugins --> Search for `wx7138a7bb793608c3` and add +[Reference document](https://developers.weixin.qq.com/miniprogram/dev/framework/plugin/using.html) + +#### **5. Build dependencies** +Click on the menu bar in the developer tools: Tools --> Build npm + +Reason: The node_modules directory will not be involved in compiling, uploading and packaging. If a small program wants to use npm packages, it must go through the process of "building npm". After the construction is completed, a miniprogram_npm directory will be generated, which will store the built and packaged npm packages. It is the npm package that the mini-program actually uses. * +[Reference Documentation](https://developers.weixin.qq.com/miniprogram/dev/devtools/npm.html) + + +### 2.3 visualization + + + + +## 3. Model inference pipeline + +```typescript +// Introduce paddlejs and paddlejs-plugin, register the mini-program environment variables and the appropriate backend +import * as paddlejs from '@paddlejs/paddlejs-core'; +import '@paddlejs/paddlejs-backend-webgl'; +const plugin = requirePlugin('paddlejs-plugin'); +plugin.register(paddlejs, wx); + +// Initialize the inference engine +const runner = new paddlejs.Runner({modelPath, feedShape, mean, std}); +await runner.init(); + +// get image information +wx.canvasGetImageData({ + canvasId: canvasId, + x: 0, + y: 0, + width: canvas.width, + height: canvas.height, + success(res) { + // inference prediction + runner.predict({ + data: res.data, + width: canvas.width, + height: canvas.height, + }, function (data) { + // get the inference result + console.log(data) + }); + } +}); +```` + + +## 4. FAQ +### 4.1 An error occurs `Invalid context type [webgl2] for Canvas#getContext` + +You can leave it alone, it will not affect the normal code operation and demo function + +### 4.2 Preview can't see the result + +It is recommended to try real machine debugging + +### 4.3 A black screen appears in the WeChat developer tool, and then there are too many errors + +Restart WeChat Developer Tools + +### 4.4 The debugging results of the simulation and the real machine are inconsistent; the simulation cannot detect the text, etc. + +The real machine can prevail; + +If the simulation cannot detect the text, etc., you can try to change the code at will (add, delete, newline, etc.) and then click to compile + + +### 4.5 Prompts such as no response for a long time appear when the phone is debugged or running + +Please continue to wait, model inference will take some time \ No newline at end of file diff --git a/examples/application/js/package/README.md b/examples/application/js/package/README.md index f2430fd2d..35aa73da6 100644 --- a/examples/application/js/package/README.md +++ b/examples/application/js/package/README.md @@ -1,3 +1,5 @@ +[English](README_en.md) | 简体中文 + # Paddle.js Model Module介绍 该部分是基于 Paddle.js 进行开发的模型库，主要提供 Web 端可直接引入使用模型的能力。 diff --git a/examples/application/js/package/README_en.md b/examples/application/js/package/README_en.md new file mode 100644 index 000000000..931aaac27 --- /dev/null +++ b/examples/application/js/package/README_en.md @@ -0,0 +1,41 @@ +English | [简体中文](README.md) + +# Introduction to Paddle.js Demo Module + +This part is a model library developed based on Paddle.js, which mainly provides the ability to directly introduce and use models on the web side. + +| demo name | source directory | npm package | +| ---------------- | -------------------------------- ---------------------- | --------------------------------------- --------------------------------- | +| face detection | [facedetect](./packages/paddlejs-models/facedetect) | [@paddle-js-models/facedetect](https://www.npmjs.com/package/@paddle-js-models/ facedetect) | +| Screw detection | [detect](./packages/paddlejs-models/detect) | [@paddle-js-models/detect](https://www.npmjs.com/package/@paddle-js-models/detect ) | +| Portrait segmentation background replacement | [humanseg](./packages/paddlejs-models/humanseg) | [@paddle-js-models/humanseg](https://www.npmjs.com/package/@paddle-js-models /humanseg) | +| Gesture Recognition AI Guessing Shell | [gesture](./packages/paddlejs-models/gesture) | [@paddle-js-models/gesture](https://www.npmjs.com/package/@paddle-js -models/gesture) | +| 1000 Item Recognition | [mobilenet](./packages/paddlejs-models/mobilenet) | [@paddle-js-models/mobilenet](https://www.npmjs.com/package/@paddle-js-models /mobilenet) | +| Text Detection | [ocrdetection](./packages/paddlejs-models/ocrdetection) | [@paddle-js-models/ocrdet](https://www.npmjs.com/package/@paddle-js-models/ocrdet ) | +| Text Recognition | [ocr](./packages/paddlejs-models/ocr) | [@paddle-js-models/ocr](https://www.npmjs.com/package/@paddle-js-models/ocr ) | + +## Usage + +This part is Menorepo built with `pnpm` + +### Install dependencies + +````sh +pnpm i +```` + +### Development +See Package.json for development testing with `yalc`. + +````sh +pnpm run dev:xxx +```` + +### Overall Introduction + +1. Use rollup to package the code of commonjs and es specifications at one time; at the same time, it is extensible; at present, there are some problems with the dependent cv library; there is no configuration for umd packaging. +2. The d.ts file is generated based on api-extractor during packaging, and the introduction of ts is supported to generate our package +3. Support testing based on jest and display test related coverage, etc. +4. Maintain code style based on ts and eslint to ensure better code development +5. Generate custom keywords based on conventional-changelog-cli and generate changelog accordingly +6. Implement local packaging development and testing based on yalc \ No newline at end of file