SS928V100:onnx到om后，sample_npu_main 无法正常load，报错 InitTbeHandle: ErrorNo: 1343225860(Internal errors)

硬件环境：
SS928V100

上位机环境：
通过百度网盘分享的文件：06.虚拟机
链接：https://pan.baidu.com/s/1Hpo7t2W881AkHX6CtzLSPQ
提取码： pl6k

Ubuntu 18.04
安装的miniconda python 3.7.10
torchvision 0.9.1 pypi_0 pypi
torch 1.8.1 pypi_0 pypi

pth转onnx
基于
https://www.hiascend.com/document/detail/zh/canncommercial/63RC2/inferapplicationdev/atctool/atctool_000005.html
https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/ACL_PyTorch/built-in/cv/Resnet50_Pytorch_Infer
执行
python3 pth2onnx.py ./resnet50-0676ba61.pth
atc --model=resnet50_official.onnx --framework=5 --output=resnet50_bs64 --input_format=NCHW --input_shape="actual_input_1:64,3,224,224" --enable_small_channel=1 --log=error --soc_version=SD3403 --insert_op_conf=aipp_resnet50.aippconfig
转码成功，得到 resnet50_bs64.om

其中atc基于 Ascend-cann-toolkit_6.0.RC1.alpha003_linux-x86_64.run 安装

sample版本为 SS928V100_REF_V2.0.2.1 运行 ./sample_npu_main 0
其中SDK中自带的 resnet50.om 可以正常推理，替换为我生成的 resnet50_bs64.om，报错
[ERROR] GE(1806,sample_npu_main):1970-01-01-01:40:39.605.575 [davinci_model.cc:3871]1806 InitTbeHandle: ErrorNo: 1343225860(Internal errors) [LOAD][DEFAULT][Check][Param] TBE: atomic_addr_clean-1_0_1 can’t find tvm bin file!

~/nfs # ./sample_npu_main 0
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.279.725 [aicpusd_cust_so_manager.cpp:73][AicpuCustSoManagerInit][tid:1806] cust so dir name is /root/cust_aicpu_0_1806/.
[EVENT] PROFILING(1806,sample_npu_main):1970-01-01-01:40:39.346.457 [msprof_callback_impl.cpp:99] >>> (tid:1806) MsprofCtrlCallback called, type: 255
[EVENT] PROFILING(1806,sample_npu_main):1970-01-01-01:40:39.346.837 [prof_acl_mgr.cpp:1328] >>> (tid:1806) Init profiling for dynamic profiling
[level]:Info,[func]:sample_svp_npu_init_resource [line]:90 [info]:acl init success.
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.021 [ae_so_manager.cc:440][LoadSo][tid:1806][AICPU_PROCESSER] Single so manager init failed, soFile is /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/libaicpu_kernels.so.
[ERROR] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.206 [ae_so_manager.cc:453][LoadSo][tid:1806][AICPU_PROCESSER] Single so manager init failed, soFile is /usr/lib64/aicpu_kernels/libaicpu_kernels.so.
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.258 [ae_kernel_lib_aicpu.cc:168][BatchLoadKernelSo][tid:1806][AICPU_PROCESSER] Load so libaicpu_kernels.so failed.
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.308 [aicpusd_interface.cpp:113][AICPUPreOpenKernels][tid:1806] Preload so libaicpu_kernels.so success.
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.374 [ae_so_manager.cc:440][LoadSo][tid:1806][AICPU_PROCESSER] Single so manager init failed, soFile is /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/libcpu_kernels.so.
[ERROR] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.433 [ae_so_manager.cc:453][LoadSo][tid:1806][AICPU_PROCESSER] Single so manager init failed, soFile is /usr/lib64/aicpu_kernels/libcpu_kernels.so.
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.480 [ae_kernel_lib_aicpu.cc:168][BatchLoadKernelSo][tid:1806][AICPU_PROCESSER] Load so libcpu_kernels.so failed.
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:39.376.525 [aicpusd_interface.cpp:113][AICPUPreOpenKernels][tid:1806] Preload so libcpu_kernels.so success.
[EVENT] PROFILING(1806,sample_npu_main):1970-01-01-01:40:39.380.240 [msprof_callback_impl.cpp:120] >>> (tid:1806) MsprofSetDeviceCallback called, is open: 1
[level]:Info,[func]:sample_svp_npu_init_resource [line]:98 [info]:open device 0 success.
[level]:Info,[func]:sample_svp_npu_init_resource [line]:106 [info]:get run mode success
[ERROR] GE(1806,sample_npu_main):1970-01-01-01:40:39.605.575 [davinci_model.cc:3871]1806 InitTbeHandle: ErrorNo: 1343225860(Internal errors) [LOAD][DEFAULT][Check][Param] TBE: atomic_addr_clean-1_0_1 can't find tvm bin file!
[ERROR] GE(1806,sample_npu_main):1970-01-01-01:40:39.605.866 [davinci_model.cc:868]1806 InitNodes: ErrorNo: 1343225860(Internal errors) [LOAD][DEFAULT][Init][TbeHandle] failed. op:atomic_addr_clean-1_0_1
[ERROR] GE(1806,sample_npu_main):1970-01-01-01:40:39.605.955 [davinci_model.cc:639]1806 Init: ErrorNo: 4294967295(failed) [LOAD][DEFAULT][Init][Nodes] failed, graph:resnet50_NPU_0.
[ERROR] GE(1806,sample_npu_main):1970-01-01-01:40:39.606.012 [model_manager.cc:1407]1806 LoadModelOffline: ErrorNo: 4294967295(failed) [LOAD][DEFAULT][Init][DavinciModel] failed, ret:1343225860.
[ERROR] GE(1806,sample_npu_main):1970-01-01-01:40:39.628.060 [graph_loader.cc:230]1806 LoadModelFromData: ErrorNo: 1343225860(Internal errors) [LOAD][DEFAULT][Load][Model] failed, model_id:1.
[ERROR] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.628.337 [model.cpp:224]1806 ModelLoadFromFileWithMem: [LOAD][DEFAULT][Model][FromData]load model from data failed, ge result[1343225860]
 [level]:Error,[func]:sample_npu_load_model_with_mem [line]:60 [info]:load model from file failed, model file is /root/nfs/data/model/resnet50.om

 [level]:Error,[func]:sample_svp_npu_load_model [line]:159 [info]:execute load model fail, model_index is:0.

[level]:Info,[func]:sample_npu_unload_model [line]:519 [info]:no model had been loaded.
[level]:Info,[func]:sample_npu_destroy_desc [line]:155 [info]:destroy model description success
[level]:Info,[func]:sample_npu_unload_model [line]:519 [info]:no model had been loaded.
[EVENT] GE(1806,sample_npu_main):1970-01-01-01:40:39.636.198 [graph_caching_allocator.cc:440]1806 PrintStatics:Called counts[malloc:          0 free:          0].
[EVENT] GE(1806,sample_npu_main):1970-01-01-01:40:39.636.376 [graph_caching_allocator.cc:390]1806 PrintCount:Malloc total[size:          0 count:          0].
[EVENT] GE(1806,sample_npu_main):1970-01-01-01:40:39.636.428 [graph_caching_allocator.cc:390]1806 PrintCount: Using total[size:          0 count:          0].
[EVENT] GE(1806,sample_npu_main):1970-01-01-01:40:39.636.474 [graph_caching_allocator.cc:390]1806 PrintCount:  Free total[size:          0 count:          0].
[EVENT] PROFILING(1806,sample_npu_main):1970-01-01-01:40:39.636.617 [msprof_callback_impl.cpp:120] >>> (tid:1806) MsprofSetDeviceCallback called, is open: 0
[level]:Info,[func]:sample_svp_npu_destroy_resource [line]:69 [info]:end to reset device is 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.905.557 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_MALLOC_FREE, applyTotal = 2, applySucc = 2, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.905.786 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_MALLOC_FREE_HOST, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.905.837 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_CONTEXT, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.905.885 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_SET_RESET_DEVICE, applyTotal = 1, applySucc = 1, releaseTotal = 1, releaseSucc = 1
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.905.934 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_EVENT, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.905.981 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_STREAM, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.029 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_DVPP_MALLOC_FREE, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.076 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_RECORD_RESET_EVENT, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.125 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DATA_BUFFER, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.173 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_TENSOR_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.220 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.267 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DATASET, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.316 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_LOAD_UNLOAD_MODEL, applyTotal = 1, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.365 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_AIPP, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.413 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_ATTR, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.460 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_HANDLE, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.509 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_CHANNEL_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:39.906.558 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_PIC_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.206 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_ROI_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.281 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_RESIZE_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.332 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_JPEGE_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.380 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_VDEC_CHANNEL_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.429 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_VENC_CHANNEL_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.476 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_STREAM_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.524 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_VDEC_FRAME_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.572 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_VENC_FRAME_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.618 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_CHANNEL, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.667 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_VDEC_CHANNEL, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.715 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_VENC_CHANNEL, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.763 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_DVPP_BATCH_PIC_DESC, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.811 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_GROUP_INFO, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.859 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_PROF_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.906 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_PROF_SUB_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.250.954 [resource_statistics.cpp:104]1806 TraverseStatistics: The ResourceType:ACL_STATISTICS_CREATE_DESTROY_MODEL_CONFIG, applyTotal = 0, applySucc = 0, releaseTotal = 0, releaseSucc = 0
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.594.587 [resource_statistics.cpp:93]1806 TraverseStatistics: The ResourceType:34 is not exist!
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.594.639 [resource_statistics.cpp:93]1806 TraverseStatistics: The ResourceType:35 is not exist!
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.594.683 [resource_statistics.cpp:93]1806 TraverseStatistics: The ResourceType:36 is not exist!
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.594.727 [resource_statistics.cpp:93]1806 TraverseStatistics: The ResourceType:37 is not exist!
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.594.769 [resource_statistics.cpp:93]1806 TraverseStatistics: The ResourceType:38 is not exist!
[EVENT] ASCENDCL(1806,sample_npu_main):1970-01-01-01:40:40.594.812 [resource_statistics.cpp:93]1806 TraverseStatistics: The ResourceType:39 is not exist!
[EVENT] PROFILING(1806,sample_npu_main):1970-01-01-01:40:40.594.860 [msprof_callback_impl.cpp:99] >>> (tid:1806) MsprofCtrlCallback called, type: 3
[level]:Info,[func]:sample_svp_npu_destroy_resource [line]:75 [info]:end to finalize acl
[EVENT] CCECPU(1806,sample_npu_main):1970-01-01-01:40:40.641.830 [aicpusd_mpi_mgr.cpp:49][PrintStatisticInfo][tid:1806] Mpi Dvpp event statistic: [0]