Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered

Posted 2020-11-03 superxiaoying

1. 当一个层的bottom和top维度一致时，取同样的名字没问题(比如说relu)，但是当两者维度不一致时，就不能取同一个名字。建议查看log中bottom 和top名字一样的层，数据维度是否一直。
2. I just encountered this issue, in my case this seems due to the use of rectangular kernels into conv layers.
   Reshaping the kernels into square ones resolved the issue.
   Note that I also used rectangular kernels for the pooling layers and those work properly

3. I have the similar problem:
   F0309 11:30:48.307298 892 syncedmem.hpp:19] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
   *** Check failure stack trace: ***
   Aborted (core dumped)

   When I run demo.py with gpu_id = 0, it is OK. But when I set gpu_id = 1 or 2 or 3(I have 4 gpu), the problem arises.

4. if using cuda-8, the problem seems missing from my side.
5. 在我这里是这样，因为我的环境数据其实都没有问题。 原本做2D Object detetction 的网络去掉了RPN部分，在roi pooling层中gt_boxes代替了rois，一来是要注意gt_boxes（xmin,ymin,xmax,ymax,label）五元组的顺序和rois(label,xmin,ymin,xmax,ymax)有所不同,二来要注意rois的label在proposal target 这一层是直接设置为0的，需要注意，如果训练的txt中是5类（0-4）,那么就出错了。要确保数据在每一个环节都没有问题。