用C++部署yolov5+deepsort+tensorrt实现目标跟踪

Posted 2023-03-27 DoubleLi

本文已参与「新人创作礼」活动，一起开启掘金创作之路。​

一、参考资料

Jetson 系列——基于yolov5和deepsort的多目标头部识别，跟踪，使用tensorrt和c++加速

二、相关介绍

2.1 重要说明

==该项目能部署在Jetson系列的产品，也能部署在X86 服务器中。==

2.2 项目结构

.
├── assets
│   └── yolosort.gif
├── build  # 编译的文件夹
│   ├── CMakeCache.txt
│   ├── CMakeFiles
│   ├── cmake_install.cmake
│   ├── libdeepsort.so
│   ├── libyolov5_trt.so
│   ├── Makefile
│   └── yolosort
├── CMakeLists_deepsort-tensorrt_win10.txt
├── CMakeLists.txt
├── CMakeLists_yolov5-deepsort-tensorrt_win10.txt
├── deepsort  # deepsort源码
│   ├── include
│   └── src
├── resources  # 存放engine引擎的文件夹
├── include
│   └── manager.hpp
├── LICENSE
├── README.md
├── requirementes-gpu.txt
├── src
│   ├── main.cpp  # 入口函数
│   └── manager.cpp
└── yolo  # yolo源码
    ├── include
    └── src
复制代码

2.3 主项目

yolov5-deepsort-tensorrt

2.4 生成 onnx 文件

deep_sort_pytorch

2.5 生成 deepsort.engine

deepsort-tensorrt

2.6 生成 yolov5s.engine

TensorRT实现yolov5推理加速（一） tensorrtx/yolov5

三、实验环境

3.1 系统环境

由于博主在PC主机上测试部分代码，以下实验环境为PC主机的环境，以后有条件，我再测试一下Jetson TX2的效果。

Environment
Operating System + Version: Ubuntu + 16.04
GPU Type: GeForce GTX1650,4GB
Nvidia Driver Version: 470.63.01
CUDA Version: 10.2.300
CUDNN Version: 7.6.5
Python Version (if applicable): 3.6.14
virtualenv：20.13.0
gcc：7.5.0
g++：7.5.0
复制代码

3.2 requirements-gpu.txt

absl-py==1.0.0
appdirs==1.4.4
backcall==0.2.0
cached-property==1.5.2
cachetools==4.2.4
certifi==2021.10.8
chardet==4.0.0
charset-normalizer==2.0.10
cycler==0.11.0
Cython==0.29.26
dataclasses==0.8
decorator==4.4.2
distlib==0.3.4
easydict==1.9
filelock==3.4.1
flake8==4.0.1
future==0.18.2
gdown==3.10.1
google-auth==2.3.3
google-auth-oauthlib==0.4.6
graphsurgeon @ file:///home/yichao/360Downloads/TensorRT-7.1.3.4/graphsurgeon/graphsurgeon-0.4.5-py2.py3-none-any.whl
grpcio==1.43.0
h5py==3.1.0
idna==3.3
imageio==2.13.5
importlib-metadata==4.2.0
importlib-resources==5.4.0
imutil==0.2.5
ipdb==0.13.9
ipython==7.16.3
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.17.2
kiwisolver==1.3.1
Mako==1.1.6
Markdown==3.3.5
MarkupSafe==2.0.1
matplotlib==3.3.4
mccabe==0.6.1
networkx==2.5.1
numpy==1.19.5
oauthlib==3.1.1
opencv-python==4.5.5.62
pandas==1.1.5
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.4.0
platformdirs==2.4.0
prompt-toolkit==3.0.24
protobuf==3.19.3
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.8.0
pycuda==2020.1
pyflakes==2.4.0
Pygments==2.11.2
pyparsing==3.0.6
PySocks==1.7.1
python-dateutil==2.8.2
pytools==2021.2.9
pytz==2021.3
PyWavelets==1.1.1
PyYAML==6.0
requests==2.27.1
requests-oauthlib==1.3.0
rsa==4.8
scikit-image==0.17.2
scipy==1.5.4
seaborn==0.11.2
six==1.16.0
tb-nightly==2.8.0a20220117
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorrt @ file:///home/yichao/360Downloads/TensorRT-7.1.3.4/python/tensorrt-7.1.3.4-cp36-none-linux_x86_64.whl
tifffile==2020.9.3
toml==0.10.2
torch @ file:///home/yichao/%E4%B8%8B%E8%BD%BD/torch-1.9.0%2Bcu102-cp36-cp36m-linux_x86_64.whl
torchvision @ file:///home/yichao/%E4%B8%8B%E8%BD%BD/torchvision-0.10.0%2Bcu102-cp36-cp36m-linux_x86_64.whl
tqdm==4.62.3
traitlets==4.3.3
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
Werkzeug==2.0.2
yacs==0.1.8
yapf==0.32.0
zipp==3.6.0
复制代码

四、生成 onnx 文件

4.1 下载github代码

git clone https://github.com/RichardoMrMu/deepsort-tensorrt.git
复制代码

4.2 下载预训练模型

ckpt.t7，46MB

下载完成后，将ckpt.t7放到 deep_sort_pytorch/deep_sort/deep/checkpoint/ckpt.t7 路径。

4.3 生成 onnx 文件

# 拷贝文件
cp deepsort-tensorrt/exportOnnx.py deep_sort_pytorch/

cd deep_sort_pytorch

# 生成onnx文件
python3 exportOnnx.py

# 拷贝onnx文件到对应目录
mv deep_sort_pytorch/deepsort.onnx deepsort-tensorrt/resources
复制代码

(venv) yichao@yichao:~/下载/deep_sort_pytorch$ time python exportOnnx.py 
==> Exporting model to ONNX format at \'deepsort.onnx\'
/home/yichao/MyDocuments/Yolov5_DeepSort_Pytorch/venv/lib/python3.6/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)

real	0m26.103s
user	0m7.749s
sys	0m2.463s

deepsort.onnx，44.7MB
复制代码

4.3.1 显卡占用情况

Fri Jan 21 15:03:09 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   30C    P0    17W /  75W |   1419MiB /  3903MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1481      G   /usr/lib/xorg/Xorg                264MiB |
|    0   N/A  N/A     10705      C   python                           1151MiB |
+-----------------------------------------------------------------------------+
复制代码

五、生成 deepsort.engine

5.1 拷贝onnx文件到对应目录

mv deep_sort_pytorch/deepsort.onnx deepsort-tensorrt/resources
复制代码

5.2 生成 CMakeLists.txt 文件

cd deepsort-tensorrt

# cmake
mkdir build
cd build

cmake ..
复制代码

5.3 修改 CMakeLists.txt 文件

deepsort-tensorrt/CMakeLists.txt 添加tensorRT库。

# tensorRT
include_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/include/)
link_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/lib/)
复制代码

deepsort-tensorrt/CMakeLists.txt 添加Eigen库。

include_directories(/usr/include/eigen3)
复制代码

5.4 编译

make -j8
复制代码

5.5 生成engine引擎

deepsort.onnx 的路径：/home/yichao/下载/deepsort-tensorrt/resources/deepsort.onnx。 deepsort.engine 的路径：/home/yichao/下载/deepsort-tensorrt/resources/deepsort.engine。

./onnx2engine ../resources/deepsort.onnx ../resources/deepsort.engine
复制代码

(venv) yichao@yichao:~/下载/deepsort-tensorrt/build$ time ./onnx2engine ../resources/deepsort.onnx ../resources/deepsort.engine
----------------------------------------------------------------
Input filename:   ../resources/deepsort.onnx
ONNX IR version:  0.0.6
Opset version:    10
Producer name:    pytorch
Producer version: 1.9
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
[01/21/2022-15:10:32] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
==============
|  SUCCESS!  |
==============

real	1m58.582s
user	0m27.711s
sys	0m3.474s

deepsort.engine，24.6MB
复制代码

5.5.1 显存占用情况

Fri Jan 21 15:11:04 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   34C    P0    44W /  75W |   1255MiB /  3903MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1481      G   /usr/lib/xorg/Xorg                264MiB |
|    0   N/A  N/A     11074      C   ./onnx2engine                     987MiB |
+-----------------------------------------------------------------------------+
复制代码

5.6 测试引擎是否生成

./demo ../resource/deepsort.engine ../resources/track.txt
复制代码

六、主项目

6.1 下载github代码

# 下载github代码
git clone https://github.com/RichardoMrMu/yolov5-deepsort-tensorrt.git
复制代码

6.2 拷贝engine到对应目录

yolov5-deepsort-tensorrt/engine/yolov5s.engine

yolov5-deepsort-tensorrt/engine/deepsort.engine
复制代码

6.3 修改配置

yolov5-deepsort-tensorrt/src/main.cpp，修改engine的路径。

// before you cmake and make, please change ./src/main.cpp char* yolo_engine = ""; char* sort_engine = ""; to your own path
char* yolo_engine = "yolov5-deepsort-tensorrt/engine/yolov5s.engine";
char* sort_engine = "yolov5-deepsort-tensorrt/engine/deepsort.engine";

// 设置摄像头索引
frame = capture.open(0);

// 显示检测结果
// 在 yolov5-deepsort-tensorrt/src/manager.cpp 文件中，取消注释
showDetection(frame,det);  # 取消注释
复制代码

6.4 生成 CMakeLists.txt 文件

cd yolov5-deepsort-tensorrt

# cmake
mkdir build
cd build

cmake ..
复制代码

6.5 修改 CMakeLists.txt 文件

deepsort-tensorrt/CMakeLists.txt 添加tensorRT库。

# tensorRT
include_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/include)
link_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/lib/)
复制代码

deepsort-tensorrt/CMakeLists.txt 添加Eigen库。

include_directories(/usr/include/eigen3)
复制代码

6.6 编译

make -j8
复制代码

6.7 运行

# 运行
./yolosort
复制代码

(venv) yichao@yichao:~/下载/yolov5-deepsort-tensorrt/build$ ./yolosort 
yolov5_trt_create  ... 
yolov5_trt_create  cuda engine... 
yolov5_trt_create  buffer ... 
yolov5_trt_create  stream ... 
yolov5_trt_create  done ... 
create yolov5-trt , instance = 0x25a10a0
delay_proress:3ms, delay_infer:7ms
delay_infer:33ms
delay_proress:6ms, delay_infer:6ms
delay_infer:21ms
delay_proress:3ms, delay_infer:6ms
delay_infer:16ms
delay_proress:12ms, delay_infer:7ms
delay_infer:29ms
delay_proress:4ms, delay_infer:7ms
delay_infer:19ms
delay_proress:12ms, delay_infer:7ms
delay_infer:30ms
delay_proress:10ms, delay_infer:7ms
delay_infer:26ms
delay_proress:5ms, delay_infer:7ms
delay_infer:18ms
...
复制代码

delay_proress：yolov5预处理时间； delay_infer：yolov5目标检测时间； delay_infer：总共的时间。

6.7.1 显存占用情况

Fri Jan 21 14:37:18 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   31C    P0    18W /  75W |   1082MiB /  3903MiB |     16%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1481      G   /usr/lib/xorg/Xorg                264MiB |
|    0   N/A  N/A      9950      C   ./yolosort                        814MiB |
+-----------------------------------------------------------------------------+
复制代码

七、可能存在的问题

Q1：yaml.load()函数缺少参数错误

运行 python exportOnnx.py 出错。

(venv) yichao@yichao:~/下载/deep_sort_pytorch$ python exportOnnx.py 
Traceback (most recent call last):
  File "exportOnnx.py", line 22, in <module>
    cfg.merge_from_file(args.config_deepsort)
  File "/home/yichao/下载/deep_sort_pytorch/utils/parser.py", line 23, in merge_from_file
    self.update(yaml.load(fo.read()))
TypeError: load() missing 1 required positional argument: \'Loader\'
复制代码

错误原因：
yaml.load() 函数的参数不对，缺少 Loader 对象。
yaml.load(CLIN,Loader=yaml.FullLoader) # 加上Loader=yaml.FullLoader 避免警告。

注：yaml版本5.1之后弃用，YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated.

解决办法：
修改 deep_sort_pytorch/utils/parser.py 中的代码。

cfg_dict.update(yaml.load(fo.read()))
改成
cfg_dict.update(yaml.load(fo, Loader=yaml.FullLoader))

复制代码

Q2：缺少 NInfer.h 文件

In file included from /home/yichao/下载/deepsort-tensorrt/include/deepsort.h:6:0,
                 from /home/yichao/下载/deepsort-tensorrt/src/deepsort.cpp:1:
/home/yichao/下载/deepsort-tensorrt/include/featuretensor.h:7:10: fatal error: NvInfer.h: 没有那个文件或目录
 #include <NvInfer.h>
          ^~~~~~~~~~~
compilation terminated.
CMakeFiles/deepsort.dir/build.make:75: recipe for target \'CMakeFiles/deepsort.dir/src/deepsort.cpp.o\' failed
make[2]: *** [CMakeFiles/deepsort.dir/src/deepsort.cpp.o] Error 1
CMakeFiles/deepsort.dir/build.make:103: recipe for target \'CMakeFiles/deepsort.dir/src/featuretensor.cpp.o\' failed
make[2]: *** [CMakeFiles/deepsort.dir/src/featuretensor.cpp.o] Error 1
CMakeFiles/Makefile2:86: recipe for target \'CMakeFiles/deepsort.dir/all\' failed
make[1]: *** [CMakeFiles/deepsort.dir/all] Error 2
Makefile:90: recipe for target \'all\' failed
make: *** [all] Error 2
复制代码

错误原因：
NvInfer.h 头文件属于 TensorRT 下的一个专有头文件，在编译C++ 代码时需要找到它。

解决办法：
deepsort-tensorrt/CMakeLists.txt，增加tensorRT的依赖库

# tensorRT
include_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/include)
link_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/lib/)
复制代码

Q3：缺少 Eigen/Core 文件

/home/yichao/360Downloads/TensorRT-7.1.3.4/include/NvInfer.h:3339:22: note: declared here
 class TRT_DEPRECATED IOutputDimensionsFormula
                      ^~~~~~~~~~~~~~~~~~~~~~~~
/home/yichao/360Downloads/TensorRT-7.1.3.4/include/NvInfer.h:5565:81: warning: ‘IPluginLayer’ is deprecated [-Wdeprecated-declarations]
         ITensor* const* inputs, int nbInputs, IPluginExt& plugin) TRTNOEXCEPT = 0;
                                                                                 ^
/home/yichao/360Downloads/TensorRT-7.1.3.4/include/NvInfer.h:3373:22: note: declared here
 class TRT_DEPRECATED IPluginLayer : public ILayer
                      ^~~~~~~~~~~~
CMakeFiles/Makefile2:86: recipe for target \'CMakeFiles/deepsort.dir/all\' failed
make[1]: *** [CMakeFiles/deepsort.dir/all] Error 2
Makefile:90: recipe for target \'all\' failed
make: *** [all] Error 2
复制代码

解决办法：重新make

make -j8
复制代码

/home/yichao/下载/deepsort-tensorrt/include/datatype.h:28:10: fatal error: Eigen/Core: 没有那个文件或目录
 #include <Eigen/Core>
          ^~~~~~~~~~~~
compilation terminated.
CMakeFiles/deepsort.dir/build.make:75: recipe for target \'CMakeFiles/deepsort.dir/src/deepsort.cpp.o\' failed
make[2]: *** [CMakeFiles/deepsort.dir/src/deepsort.cpp.o] Error 1
CMakeFiles/Makefile2:86: recipe for target \'CMakeFiles/deepsort.dir/all\' failed
make[1]: *** [CMakeFiles/deepsort.dir/all] Error 2
Makefile:90: recipe for target \'all\' failed
make: *** [all] Error 2
复制代码

错误原因：
编译找不到Eigen

解决办法：
1. 安装eigen3，默认安装在 /usr/include/eigen3
sudo apt-get install libeigen3-dev

2. deepsort-tensorrt/CMakeLists.txt，增加eigen3的依赖库
include_directories(/usr/include/eigen3)

# 如果还不成功，尝试创建软链接
sudo ln -s /usr/include/eigen3/Eigen /usr/include/Eigen

3. 重新编译
make clean
make -j8
复制代码

Q4：打不开摄像头

(venv) yichao@yichao:~/下载/yolov5-deepsort-tensorrt/build$ ./yolosort 
yolov5_trt_create  ... 
yolov5_trt_create  cuda engine... 
yolov5_trt_create  buffer ... 
yolov5_trt_create  stream ... 
yolov5_trt_create  done ... 
create yolov5-trt , instance = 0x295e000
GStreamer: Error opening bin: no element "0"
can not open
复制代码

(venv) yichao@yichao:~/下载/yolov5-deepsort-tensorrt/build$ ./yolosort 
yolov5_trt_create  ... 
yolov5_trt_create  cuda engine... 
yolov5_trt_create  buffer ... 
yolov5_trt_create  stream ... 
yolov5_trt_create  done ... 
create yolov5-trt , instance = 0x186d000
GStreamer: Error opening bin: empty pipeline not allowed
can not open
复制代码

错误原因：
在 yolov5-deepsort-tensorrt/src/main.cpp 文件中，未设置摄像头索引或者设置摄像头失败。
frame = capture.open("");  // 未设置索引
frame = capture.open("0");  // 设置失败，索引为int类型

解决办法：
在 yolov5-deepsort-tensorrt/src/main.cpp 文件中，设置摄像头索引。
frame = capture.open(0);

from：https://juejin.cn/post/7103907324682567694