Introduction
Overview
The Ascend Tensor Compiler (ATC) optimizes the model during model conversion, including operator elimination, operator fusion, and operator split. As a result, the computation result of Huawei proprietary operators may be different from that of third-party operators (for example, from Caffe and TensorFlow). The Model Accuracy Analyzer is designed to compare the computation results for developers to quickly resolve the operator accuracy issues.
The Model Accuracy Analyzer compares the computation results of Huawei proprietary operators with those of the Caffe or TensorFlow counterpart operators to locate the error operators. Currently, the following vector comparison methods are provided:
Cosine similarity, maximum absolute error, accumulated relative error, Euclidean relative distance, Kullback-Leibler divergence (KLD), and standard deviation
Functions
The current version supports accuracy comparison between the dump data of a model running on the Ascend AI Processor and Ground Truth (.npy file of a model running on the GPU or CPU). Quantized and non-quantized dump data can also be compared, as shown in Table 5-1.
No. |
Data to Be Compared (My Output) |
Standard Data (Ground Truth) |
|---|---|---|
1 |
Dump data of the non-quantized offline model running on the Ascend AI Processor |
.npy file of the non-quantized Caffe model |
2 |
Dump data of the quantized offline model running on the Ascend AI Processor |
.npy file of the non-quantized Caffe model |
3 |
.npy file of the quantized Caffe model |
.npy file of the non-quantized Caffe model |
4 |
Dump data of the quantized offline model running on the Ascend AI Processor |
.npy file of the quantized Caffe model |
5 |
Dump data of the non-quantized offline model running on the Ascend AI Processor |
.npy file of the non-quantized TensorFlow model |
6 |
Dump data of the model running on the Ascend AI Processor |
Dump data of the model running on the Ascend AI Processor |
Restrictions
- Before using the Model Accuracy Analyzer, set up the environment by referring to the CANN Software Installation Guide. This document describes how to use the Model Accuracy Analyzer with the assumption that the tool is installed by HwHiAiUser in the default installation path /home/HwHiAiUser/Ascend/ascend-toolkit/latest. You can replace them as required. Ensure that HwHiAiUser has the read, or read and write permissions on the paths described in this document.
- Operating environment described in this document needs to be replaced as required.
- For an optimal experience, the following hardware configuration is recommended: 8-core CPU at 2.6 GHz with 16 GB memory.
- Accuracy comparison of a single-operator network is not supported.
- The following software should be installed in advance: Python 3.7.5.
- The following data types are supported:
- FLOAT
- FLOAT16
- DT_INT8
- DT_UINT8
- DT_INT16
- DT_UINT16
- DT_INT32
- DT_INT64
- DT_UINT32
- DT_UINT64
- DT_BOOL
- DT_DOUBLE
