readme_en.md 11.7 KB
Newer Older
L
LDOUBLEV 已提交
1

L
LDOUBLEV 已提交
2
# Tutorial of PaddleOCR Mobile deployment
L
LDOUBLEV 已提交
3

4
This tutorial will introduce how to use [paddle-lite](https://github.com/PaddlePaddle/Paddle-Lite) to deploy paddleOCR ultra-lightweight Chinese and English detection models on mobile phones.
L
LDOUBLEV 已提交
5

蔡舒起 已提交
6
paddle-lite is a lightweight inference engine for PaddlePaddle.
7
It provides efficient inference capabilities for mobile phones and IoTs,
L
LDOUBLEV 已提交
8 9 10
and extensively integrates cross-platform hardware to provide lightweight
deployment solutions for end-side deployment issues.

L
LDOUBLEV 已提交
11
## 1. Preparation
L
LDOUBLEV 已提交
12

L
LDOUBLEV 已提交
13
- Computer (for Compiling Paddle Lite)
L
LDOUBLEV 已提交
14 15
- Mobile phone (arm7 or arm8)

D
dyning 已提交
16
***Note: PaddleOCR lite deployment currently does not support dynamic graph models, only models saved with static graph. The static branch of PaddleOCR is `develop`.***
L
LDOUBLEV 已提交
17

M
Ming 已提交
18
## 2. Build PaddleLite library
L
LDOUBLEV 已提交
19 20 21
1. [Docker](https://paddle-lite.readthedocs.io/zh/latest/source_compile/compile_env.html#docker)
2. [Linux](https://paddle-lite.readthedocs.io/zh/latest/source_compile/compile_env.html#linux)
3. [MAC OS](https://paddle-lite.readthedocs.io/zh/latest/source_compile/compile_env.html#mac-os)
L
LDOUBLEV 已提交
22

23
## 3. Prepare prebuild library for android and ios
L
LDOUBLEV 已提交
24

25
### 3.1 Download prebuild library
L
LDOUBLEV 已提交
26 27
|Platform|Prebuild library Download Link|
|-|-|
28 29
|Android|[arm7](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.android.armv7.gcc.c++_shared.with_extra.with_cv.tar.gz) / [arm8](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.android.armv8.gcc.c++_shared.with_extra.with_cv.tar.gz)|
|IOS|[arm7](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.ios.armv7.with_cv.with_extra.with_log.tiny_publish.tar.gz) / [arm8](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.ios.armv8.with_cv.with_extra.with_log.tiny_publish.tar.gz)|
L
LDOUBLEV 已提交
30

31
note: The above pre-build inference library is compiled from the PaddleLite `release/v2.7` branch. For more information about PaddleLite 2.6.3, please refer to [link](https://github.com/PaddlePaddle/Paddle-Lite/releases/tag/v2.6.3).
L
LDOUBLEV 已提交
32

33 34 35 36 37 38 39 40
### 3.2 Compile prebuild library (Recommended)
```
git clone https://github.com/PaddlePaddle/Paddle-Lite.git
cd Paddle-Lite
# checkout to Paddle-Lite release/v2.7 branch
git checkout release/v2.7
./lite/tools/build_android.sh  --arch=armv8  --with_cv=ON --with_extra=ON
```
L
LDOUBLEV 已提交
41 42 43 44 45

The structure of the prediction library is as follows:

```
inference_lite_lib.android.armv8/
L
LDOUBLEV 已提交
46 47
|-- cxx                                        C++ prebuild library
|   |-- include                                C++
L
LDOUBLEV 已提交
48 49 50 51 52 53 54
|   |   |-- paddle_api.h
|   |   |-- paddle_image_preprocess.h
|   |   |-- paddle_lite_factory_helper.h
|   |   |-- paddle_place.h
|   |   |-- paddle_use_kernels.h
|   |   |-- paddle_use_ops.h
|   |   `-- paddle_use_passes.h
L
LDOUBLEV 已提交
55 56 57 58
|   `-- lib  
|       |-- libpaddle_api_light_bundled.a             C++ static library
|       `-- libpaddle_light_api_shared.so             C++ dynamic library
|-- java                                     Java predict library
L
LDOUBLEV 已提交
59 60 61 62 63
|   |-- jar
|   |   `-- PaddlePredictor.jar
|   |-- so
|   |   `-- libpaddle_lite_jni.so
|   `-- src
L
LDOUBLEV 已提交
64 65 66
|-- demo                                     C++ and java demo
|   |-- cxx  
|   `-- java  
L
LDOUBLEV 已提交
67 68 69 70 71 72 73
```


## 4. Inference Model Optimization

Paddle Lite provides a variety of strategies to automatically optimize the original training model, including quantization, sub-graph fusion, hybrid scheduling, Kernel optimization and so on. In order to make the optimization process more convenient and easy to use, Paddle Lite provide opt tools to automatically complete the optimization steps and output a lightweight, optimal executable model.

L
opt doc  
LDOUBLEV 已提交
74
If you have prepared the model file ending in `.nb`, you can skip this step.
L
LDOUBLEV 已提交
75

L
opt doc  
LDOUBLEV 已提交
76 77
The following table also provides a series of models that can be deployed on mobile phones to recognize Chinese.
You can directly download the optimized model.
L
LDOUBLEV 已提交
78

79 80 81 82 83
| Version | Introduction | Model size | Detection model | Text Direction model | Recognition model | Paddle Lite branch |
| - | - | - | - | - | - | - |
| V1.1 | extra-lightweight chinese OCR optimized model | 8.1M | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_det_opt.nb) | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_cls_opt.nb) | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_rec_opt.nb) | develop |
| [slim] V1.1 | extra-lightweight chinese OCR optimized model | 3.5M | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_det_prune_opt.nb) | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_cls_quant_opt.nb) | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_rec_quant_opt.nb) | develop |
| V1.0 | lightweight Chinese OCR optimized model | 8.6M | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.0_det_opt.nb) | - | [Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.0_rec_opt.nb) | develop |
L
LDOUBLEV 已提交
84 85 86 87 88 89

If the model to be deployed is not in the above table, you need to follow the steps below to obtain the optimized model.

```
git clone https://github.com/PaddlePaddle/Paddle-Lite.git
cd Paddle-Lite
90
git checkout release/v2.7
L
LDOUBLEV 已提交
91 92 93 94 95 96 97 98 99 100 101
./lite/tools/build.sh build_optimize_tool
```

The `opt` tool can be obtained by compiling Paddle Lite.

After the compilation is complete, the opt file is located under `build.opt/lite/api/`.

The `opt` can optimize the inference model saved by paddle.io.save_inference_model to get the model that the paddlelite API can use.

The usage of opt is as follows:
```
L
LDOUBLEV 已提交
102
# 【Recommend】V1.1 is better than V1.0. steps for convert V1.1 model to nb file are as follows
L
LDOUBLEV 已提交
103 104 105 106 107 108 109
wget  https://paddleocr.bj.bcebos.com/20-09-22/mobile-slim/det/ch_ppocr_mobile_v1.1_det_prune_infer.tar && tar xf  ch_ppocr_mobile_v1.1_det_prune_infer.tar
wget  https://paddleocr.bj.bcebos.com/20-09-22/mobile-slim/rec/ch_ppocr_mobile_v1.1_rec_quant_infer.tar && tar xf  ch_ppocr_mobile_v1.1_rec_quant_infer.tar

./opt --model_file=./ch_ppocr_mobile_v1.1_det_prune_infer/model  --param_file=./ch_ppocr_mobile_v1.1_det_prune_infer/params  --optimize_out=./ch_ppocr_mobile_v1.1_det_prune_opt --valid_targets=arm
./opt --model_file=./ch_ppocr_mobile_v1.1_rec_quant_infer/model  --param_file=./ch_ppocr_mobile_v1.1_rec_quant_infer/params  --optimize_out=./ch_ppocr_mobile_v1.1_rec_quant_opt --valid_targets=arm

# or use V1.0 model
L
LDOUBLEV 已提交
110 111 112 113 114 115 116 117
wget  https://paddleocr.bj.bcebos.com/ch_models/ch_det_mv3_db_infer.tar && tar xf ch_det_mv3_db_infer.tar
wget  https://paddleocr.bj.bcebos.com/ch_models/ch_rec_mv3_crnn_infer.tar && tar xf ch_rec_mv3_crnn_infer.tar

./opt --model_file=./ch_det_mv3_db/model --param_file=./ch_det_mv3_db/params --optimize_out_type=naive_buffer --optimize_out=./ch_det_mv3_db_opt --valid_targets=arm
./opt --model_file=./ch_rec_mv3_crnn/model --param_file=./ch_rec_mv3_crnn/params --optimize_out_type=naive_buffer --optimize_out=./ch_rec_mv3_crnn_opt --valid_targets=arm

```

L
LDOUBLEV 已提交
118
When the above code command is completed, there will be two more files `.nb` in the current directory, which is the converted model file.
L
LDOUBLEV 已提交
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148

## 5. Run optimized model on Phone

1. Prepare an Android phone with arm8. If the compiled prediction library and opt file are armv7, you need an arm7 phone and modify ARM_ABI = arm7 in the Makefile.

2. Make sure the phone is connected to the computer, open the USB debugging option of the phone, and select the file transfer mode.

3. Install the adb tool on the computer.
    3.1 Install ADB for MAC
    ```
    brew cask install android-platform-tools
    ```
    3.2 Install ADB for Linux
    ```
    sudo apt update
    sudo apt install -y wget adb
    ```
    3.3 Install ADB for windows
    [Download Link](https://developer.android.com/studio)

    Verify whether adb is installed successfully
    ```
    $ adb devices

    List of devices attached
    744be294    device
    ```

    If there is `device` output, it means the installation was successful.

149
4. Prepare optimized models, prediction library files, test images and dictionary files used.
L
LDOUBLEV 已提交
150 151

```
152 153 154 155
 git clone https://github.com/PaddlePaddle/PaddleOCR.git
 cd PaddleOCR/deploy/lite/
 # run prepare.sh
 sh prepare.sh /{lite prediction library path}/inference_lite_lib.android.armv8
L
LDOUBLEV 已提交
156

157 158 159 160 161 162 163 164
 #
 cd /{lite prediction library path}/inference_lite_lib.android.armv8/
 cd demo/cxx/ocr/
 # copy paddle-lite C++ .so file to debug/ directory
 cp ../../../cxx/lib/libpaddle_light_api_shared.so ./debug/

 cd inference_lite_lib.android.armv8/demo/cxx/ocr/
 cp ../../../cxx/lib/libpaddle_light_api_shared.so ./debug/
L
LDOUBLEV 已提交
165 166 167

```

168 169 170
Prepare the test image, taking `PaddleOCR/doc/imgs/11.jpg` as an example, copy the image file to the `demo/cxx/ocr/debug/` folder.
Prepare the model files optimized by the lite opt tool, `ch_det_mv3_db_opt.nb, ch_rec_mv3_crnn_opt.nb`,
and place them under the `demo/cxx/ocr/debug/` folder.
L
LDOUBLEV 已提交
171 172


173 174 175 176
The structure of the OCR demo is as follows after the above command is executed:
```
demo/cxx/ocr/
|-- debug/  
L
LDOUBLEV 已提交
177 178 179
|   |--ch_ppocr_mobile_v1.1_det_prune_opt.nb           Detection model
|   |--ch_ppocr_mobile_v1.1_rec_quant_opt.nb           Recognition model
|   |--ch_ppocr_mobile_cls_quant_opt.nb                Text direction classification model
L
typo  
LDOUBLEV 已提交
180
|   |--11.jpg                           Image for OCR
181 182 183 184 185 186 187 188 189 190 191 192
|   |--ppocr_keys_v1.txt                Dictionary file
|   |--libpaddle_light_api_shared.so    C++ .so file
|   |--config.txt                       Config file
|-- config.txt  
|-- crnn_process.cc  
|-- crnn_process.h
|-- db_post_process.cc  
|-- db_post_process.h
|-- Makefile  
|-- ocr_db_crnn.cc  

```
L
LDOUBLEV 已提交
193

L
LDOUBLEV 已提交
194 195 196 197 198
#### Note:
1. ppocr_keys_v1.txt is a Chinese dictionary file.
If the nb model is used for English recognition or other language recognition, dictionary file should be replaced with a dictionary of the corresponding language.
PaddleOCR provides a variety of dictionaries under ppocr/utils/, including:
```
T
tink2123 已提交
199 200
dict/french_dict.txt     # french
dict/german_dict.txt     # german
L
LDOUBLEV 已提交
201
ic15_dict.txt       # english
T
tink2123 已提交
202 203
dict/japan_dict.txt      # japan
dict/korean_dict.txt     # korean
L
LDOUBLEV 已提交
204 205 206 207 208 209 210 211 212
ppocr_keys_v1.txt   # chinese
```

2. `config.txt`  of the detector and classifier, as shown below:
```
max_side_len  960         #  Limit the maximum image height and width to 960
det_db_thresh  0.3        # Used to filter the binarized image of DB prediction, setting 0.-0.3 has no obvious effect on the result
det_db_box_thresh  0.5    # DDB post-processing filter box threshold, if there is a missing box detected, it can be reduced as appropriate
det_db_unclip_ratio  1.6  # Indicates the compactness of the text box, the smaller the value, the closer the text box to the text
213
use_direction_classify  0  # Whether to use the direction classifier, 0 means not to use, 1 means to use
L
LDOUBLEV 已提交
214 215
```

L
LDOUBLEV 已提交
216 217 218 219 220 221 222 223 224 225 226
5. Run Model on phone

```
cd inference_lite_lib.android.armv8/demo/cxx/ocr/
make -j
mv ocr_db_crnn ./debug/
adb push debug /data/local/tmp/
adb shell
cd /data/local/tmp/debug
export LD_LIBRARY_PATH=/data/local/tmp/debug:$LD_LIBRARY_PATH
# run model
L
LDOUBLEV 已提交
227
 ./ocr_db_crnn ch_ppocr_mobile_v1.1_det_prune_opt.nb  ch_ppocr_mobile_v1.1_rec_quant_opt.nb  ch_ppocr_mobile_cls_quant_opt.nb  ./11.jpg  ppocr_keys_v1.txt
L
LDOUBLEV 已提交
228 229 230 231 232 233 234
```

The outputs are as follows:

<div align="center">
    <img src="../imgs/demo.png" width="600">
</div>
L
add faq  
LDOUBLEV 已提交
235 236 237 238 239 240 241 242 243 244 245 246

## FAQ

Q1: What if I want to change the model, do I need to run it again according to the process?
A1: If you have performed the above steps, you only need to replace the .nb model file to complete the model replacement.

Q2: How to test with another picture?
A2: Replace the .jpg test image under `./debug` with the image you want to test, and run `adb push` to push new image to the phone.

Q3: How to package it into the mobile APP?
A3: This demo aims to provide the core algorithm part that can run OCR on mobile phones.  Further,
PaddleOCR/deploy/android_demo is an example of encapsulating this demo into a mobile app for reference.