Network-wide Comparison

Command Syntax

The command for vector comparison is structured as follows:

python3.7.5 msaccucmp.pyc compare -m my_dump_path -g golden_dump_path [-f fusion_rule_file] [-q quant_fusion_rule_file] [-out output] [-c custom_script_path]

Table 5-4 describes the command-line options.

Select the -f or -q argument based on the data prepared in Comparison Data Description.

Comparison Procedure

To conduct vector comparison, perform the following steps:

• The .json file and directory names in this section are only examples. Replace them with the actual ones. Ensure that the HwHiAiUser user has the read and write permissions on the result path specified by --out.
• This section describes how to compare the dump data of a non-quantized model running on the Ascend AI Processor and the .npy file of a non-quantized Caffe model. The following parameters are based on this example. You can replace them as required.
• To compare two groups of dump data generated based on the same model running on the Ascend AI Processor, ensure that the number of inputs and outputs, formats, as well as the shapes are the same. Otherwise, comparison cannot be performed. In this scenario, only the -m, -g and -out options are required.

1. Log in to the development environment as the HwHiAiUser user.
2. Run the export command to set the environment variable and generate a .json file.

   Set the following environment variable:

   export LD_LIBRARY_PATH=/home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/lib64:${LD_LIBRARY_PATH}

   Generate the .json file:

   /home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/bin/atc --mode=1 --om=/home/HwHiAiUser/data/resnet50.om --json=/home/HwHiAiUser/data/resnet50.json

3. Go to the /home/HwHiAiUser/Ascend/ascend-toolkit/latest/toolkit/tools/operator_cmp/compare directory.
4. Run the vector comparison command as follows:

   python3.7.5 msaccucmp.pyc compare -m /home/HwHiAiUser/MyApp_mind/resnet50 -g /home/HwHiAiUser/Standard_caffe/resnet50 -f /home/HwHiAiUser/data/resnet50.json -out /home/HwHiAiUser/result

   The vector comparison result is saved to the result_*.csv file, as shown in Figure 5-4.

   Figure 5-4 Comparison result
   

   The parameters are described as follows:

   • LeftOp: operator name of the My Output model
   • RightOp: operator name of the Ground Truth model
   • TensorIndex: operator input ID and output ID of the My Output model
   • CosineSimilarity: result of the cosine similarity comparison. The value range is [–1, +1]. A value closer to 1 indicates higher similarity. A value closer to –1 indicates greater difference.
   • MaxAbsoluteError: result of the maximum absolute error comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
   • AccumulatedRelativeError: result of the accumulated relative error comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
   • RelativeEuclideanDistance: result of the Euclidean relative distance comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
   • KullbackLeiblerDivergence: result of the KLD comparison. The value ranges from 0 to infinity. The smaller the KLD, the closer the approximate distribution is to the true distribution.
   • StandardDeviation: result of the standard deviation comparison. The value ranges from 0 to infinity. The smaller the standard deviation, the smaller the dispersion, and the closer the value is to the average value. The mean value and standard deviation of the dump data are displayed in the format of (mean value;standard deviation). The first set of data is the result of My Output, and the second set is the result of Ground Truth.
   • An asterisk (*) indicates a newly added operator with no third-party counterpart. NaN indicates there is no comparison result.
   • If the results of cosine similarity and KLD are NaN, and the results of other algorithms exist, at least one piece of data on the left or the right is 0. If the result of KLD is inf, one piece of data on the right is 0. If NaN is displayed, the dump data contains NaN.