Single-Operator Comparison

Command Syntax

The command for vector comparison is structured as follows:

python3.7.5 msaccucmp.pyc compare -m my_dump_path -g golden_dump_path [-f fusion_rule_file] [-q quant_fusion_rule_file] [-out output] [-op op_name] [-o output_tensor] [-i input_tensor] [-c custom_script_path]

Table 5-5 describes the command-line options.

Select the -f or -q argument based on the data prepared in Comparison Data Description.

Comparison Procedure

To conduct vector comparison, perform the following steps:

• The .json file and directory names in this section are only examples. Replace them with the actual ones. Ensure that the HwHiAiUser user has the read and write permissions on the result path specified by --out.
• This section describes how to compare the dump data of a non-quantized model running on the Ascend AI Processor and the .npy file of a non-quantized Caffe model. The following parameters are based on this example. You can replace them as required.
• Single-operator comparison between two groups of dump data generated based on the same model running on the Ascend AI Processor is not supported.

1. Log in to the development environment as the HwHiAiUser user.
2. Run the export command to set the environment variable and generate a .json file.

   Set the following environment variable:

   export LD_LIBRARY_PATH=/home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/lib64:${LD_LIBRARY_PATH}

   Generate the .json file:

   /home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/bin/atc --mode=1 --om=/home/HwHiAiUser/data/resnet50.om --json=/home/HwHiAiUser/data/resnet50.json

3. Go to the /home/HwHiAiUser/Ascend/ascend-toolkit/latest/toolkit/tools/operator_cmp/compare directory.
4. Run the vector comparison command as follows:

   python3.7.5 msaccucmp.pyc compare -m /home/HwHiAiUser/MyApp_mind/resnet50 -g /home/HwHiAiUser/Standard_caffe/resnet50 -f /home/HwHiAiUser/data/resnet50.json -out /home/HwHiAiUser/result -op pool5 -i 0

   Figure 5-5 and Figure 5-6 show the content of a vector comparison result file.

   Figure 5-5 Single-operator comparison result (summary)
   

   The single-operator comparison result summary is stored in {op_name}_input_{index}_summary.txt or {op_name}_output_{index}_summary.txt. The parameters are described as follows:

   • TotalCount: number of data records in the dump data of the operator
   • LeftOp: operator name of the My Output model
   • RightOp: operator name of the Ground Truth model
   • Format: data format
   • MinAbsoluteError & MaxAbsoluteError: minimum and maximum absolute error ranges
   • MinRelativeError & MaxRelativeError: minimum and maximum relative error ranges
   Figure 5-6 Single-operator comparison result (details)
   

   The detailed comparison result of the single-operator is stored in {op_name}_input_{index}_{file_index}.csv or {op_name}_output_{index}_{file_index}.csv. Each file records a maximum of one million data records. The parameters in Figure 5-6 are described as follows:

   • N C H W: data coordinates
   • Left: dump data of the My Output model operator
   • Right: dump data of the Ground Truth model operator
   • RelativeError: relative error. The value is obtained by dividing the AbsoluteError value by the dump value of the Ground Truth operator. If the dump value of the Ground Truth operator is 0, a hyphen (-) is displayed.
   • AbsoluteError: absolute error. The value is the difference between the dump value of the My Output operator and that of the Ground Truth operator.