compare_vector.pyc

Prerequisites

Before preparing dump data, convert the model and prepare the model file by referring to ATC Tool Instructions. If model quantization is involved, perform quantization by referring to Ascend Model Compression Toolkit Instructions (Caffe) or Ascend Model Compression Toolkit Instructions (TensorFlow) before model conversion. In addition, build and run the application project with the generated model file and ensure that the project runs properly.

• During the execution of AMCT, a quantization fusion file used for accuracy comparison is also generated, which will be needed for model comparison.
• In Docker scenarios, dump is not supported in containers.
• Data dump can be implemented by calling the aclInit() or aclmdlSetDump() API.

  Dump of single-operators can be implemented only by calling the aclmdlSetDump() API.

  For details about the aclInit() and aclmdlSetDump() APIs, see Application Software Development Guide.

Generating Dump Data

Perform the following steps to dump data of the offline model:

1. Open the project file, find the path of the acl.json file in the aclInit() or aclmdlSetDump() call.

   If aclInit() or aclmdlSetDump() is initialized to empty, pass the acl.json path created in Step 2 to the call. The acl.json path is relative to the path of the binary file generated during project build.

2. Modify the acl.json file in the directory (if the file does not exist, create it to the out directory after project build) and add the dump configuration in the following format.

   {                                                                                            
   	"dump":{                                                                                 
   		"dump_list":[                                                                        
   			{	"model_name":"ResNet-101"
   			},
   			{                                                                                
   				"model_name":"ResNet-50",
   				"layer":[
   				      "conv1conv1_relu",
   				      "res2a_branch2ares2a_branch2a_relu",
   				      "res2a_branch1",
   				      "pool1"
   				] 
   			}  
   		],  
   		"dump_path":"/home/HwHiAiUser/output",
                   "dump_mode":"output",
   		"dump_op_switch":"off"
   	}                                                                                        
   }

   The dump configuration rules are as follows:

   • In the sample, the dump, dump_list, and dump_path fields are required, while the model_name, layer, dump_mode, and dump_op_switch fields are optional.
   • To dump all operators of a model, the layer field does not need to be included.
   • To dump certain operators, configure each operator in a line in the layer field and separate them with commas (,).
   • To dump multiple models, add the dump configuration for each model and separate them with commas (,).
   • To dump a single-operator model, leave dump_list empty and set dump_op_switch to on.
   The configuration items are as follows:

   • dump_list: a list of network-wide models for data dump
     • model_name: model name

       To load a model from a file, enter the model file name without the file name extension. To load a model from memory, set this parameter to the value of the name field in the .json file after model conversion. For details about how to load a model, see AscendCL API Reference in Application Software Development Guide.

     • layer: operator name
   • dump_path: dump path in the operating environment

     Both absolute path and relative path (relative to the path where the command is run) are supported.

     • If set to an absolute path, starts with a slash (/), for example, /home/HwHiAiUser/output.
     • If set to a relative path, starts with a directory name, for example, output.

     For example, if dump_path is set to /home/HwHiAiUser/output, the dump data is generated to the /home/HwHiAiUser/output directory in the operating environment.

     The directory specified by this parameter must be created in advance and the user configured during installation must have the read and write permissions on the directory.

   • dump_mode: (optional) dump mode, either input, output (default), or all
     • input: dumps the operator input only.
     • output: dumps the operator output only.
     • all: dumps both the operator input and output.
   • dump_op_switch: (optional) dump switch of the single-operator model, either on or off (default)
     • on: enables dump for the single-operator model.
     • off: disables dump for the single-operator model.

   • For TBE and AI CPU operators that do not output results, (for example, StreamActive, Send, Recv and const), dump data is not generated. For operators that are not executed on the AI CPU or AI Core (for example, concatD) after build, dump data cannot be generated.
   • When only certain operators are to be dumped, since the Data operators are not executed on the AI CPU or AI Core, you need to select all the downstream operators of the Data operator.
   • If dump data files are generated by running in the command line, delete the data files that are no longer needed in the /home/HwHiAiUser/ide_daemon/dump directory to prevent dump data generation failures due to insufficient disk space.
   • When modifying the acl.json file, ensure that each model_name is unique.
   • If the model is loaded from a file, model_name can also be set to the value of the name field in the .json file after model conversion. Model name and operator names can be obtained using either of the following methods:

     If the value of model_name in the acl.json file contains both the model file name and the model name, the model file name takes effect.

3. Run the application project to generate a dump file.

   Find the generated dump file in {dump_path} in the operating environment. The path and format are described as follows:

   time/deviceid/model_name/model_id/data_index/dump_file

   • time: dump time, formatted as YYYYMMDDhhmmss
   • deviceid: device ID
   • model_name: model name
   • model_id: model ID
   • data_index: execution sequence number of each task, indexed starting at 0. This value is increased by 1 every dump.
   • dump_file: formatted as {op_type}.{op_name}.{taskid}.{timestamp}

   Periods (.), forward slashes (/), backslashes (\), and spaces in model_name, op_type or op_name are replaced with underscores (_).

   Dump data of a single-operator model is generated to {dump_path}/deviceid/op_name/dump_file.

Command Syntax

The command for vector comparison is structured as follows:

python3.7.5 compare_vector.pyc -l LEFT_DUMP_PATH -r RIGHT_DUMP_PATH [-f FUSION_JSON_FILE_PATH] [-q QUANT_FUSION_RULE_FILE_PATH] -o OUTPUT_PATH [-custom CUSTOM_PATH]

• -l: directory of the compared data of the My Output model
• -r: directory of the compared data of the Ground Truth model
• -f: (optional) network-wide information file of the offline model (the .om file can be converted into a .json file using ATC)
• -q: (optional) quantization fusion file
• -o: path and file name of the comparison result
• -custom: (optional) customized path to store the .py file for format conversion, which should be the upper-level directory of the format_convert directory. For details about the .py file requirements, see Preparing a Customized .py File for Format Conversion.

Select the -f or -q argument based on the data prepared in Comparison Data Description.

Comparison Procedure

To conduct vector comparison, perform the following steps:

• Replace the .json file and directory names in this example with the actual ones. The result and log paths must be created in advance and the HwHiAiUser user must have the read and write permissions on the paths.
• This section describes how to compare the dump data of a non-quantized model running on the Ascend AI Processor and the .npy file of a non-quantized Caffe model. The following parameters are based on this example. You can replace them as required.
• To compare two groups of dump data generated based on the same model running on the Ascend AI Processor, ensure that the number of inputs and outputs, formats, as well as the shapes are the same. Otherwise, comparison cannot be performed. In this scenario, only the -l, -r and -o options are required.

1. Log in to the development environment as the HwHiAiUser user.
2. Run the export command to set the environment variable and generate a .json file.

   Set the following environment variable:

   export LD_LIBRARY_PATH=/home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/lib64:${LD_LIBRARY_PATH}

   Generate the .json file:

   /home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/bin/atc --mode=1 --om=/home/HwHiAiUser/data/resnet50.om --json=/home/HwHiAiUser/data/resnet50.json

3. Go to the /home/HwHiAiUser/Ascend/ascend-toolkit/latest/toolkit/tools/operator_cmp/compare directory.
4. Run the vector comparison command as follows:

   python3.7.5 compare_vector.pyc -l /home/HwHiAiUser/MyApp_mind/resnet50 -r /home/HwHiAiUser/Standard_caffe/resnet50 -f /home/HwHiAiUser/data/resnet50.json -o /home/HwHiAiUser/result/result.txt

   To save the file in CSV format, modify the command into -o /home/HwHiAiUser/result/result.csv -csv.

5. The vector comparison result is saved to the result.txt file, as shown in Figure 5-10.
   Figure 5-10 Model comparison result
   

   The parameters are described as follows:

   • LeftOp: operator name of the My Output model
   • RightOp: operator name of the Ground Truth model
   • TensorIndex: operator input ID and output ID of the My Output model
   • CosineSimilarity: result of the cosine similarity comparison. The value range is [–1, +1]. A value closer to 1 indicates higher similarity. A value closer to –1 indicates greater difference.
   • MaxAbsoluteError: result of the maximum absolute error comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
   • AccumulatedRelativeError: result of the accumulated relative error comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
   • RelativeEuclideanDistance: result of the Euclidean relative distance comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
   • KullbackLeiblerDivergence: result of the KLD comparison. The value ranges from 0 to infinity. The smaller the KLD, the closer the approximate distribution is to the true distribution.
   • StandardDeviation: result of the standard deviation comparison. The value ranges from 0 to infinity. The smaller the standard deviation, the smaller the dispersion, and the closer the value is to the average value. The mean value and standard deviation of the dump data are displayed in the format of (mean value;standard deviation). The first set of data is the result of My Output, and the second set is the result of Ground Truth.
   • An asterisk (*) indicates a newly added operator with no third-party counterpart. NaN indicates there is no comparison result.
   • If the results of cosine similarity and KLD are NaN, and the results of other algorithms exist, at least one piece of data on the left or the right is 0. If the result of KLD is inf, one piece of data on the right is 0. If NaN is displayed, the dump data contains NaN.

Command Syntax

The command for vector comparison is structured as follows:

python3.7.5 compare_vector.pyc -l LEFT_DUMP_PATH -r RIGHT_DUMP_PATH [-f FUSION_JSON_FILE_PATH] [-q QUANT_FUSION_RULE_FILE_PATH] -o OUTPUT_PATH -d OP_NAME [-t DETAIL_TYPE] [-i DETAIL_INDEX] [-custom CUSTOM_PATH]

The command-line options are described as follows:

• -l: directory of the compared data of the My Output model
• -r: directory of the compared data of the Ground Truth model
• -f: (optional) network-wide information file of the offline model (the .om file can be converted into a .json file using ATC)
• -q: (optional) quantization fusion file
• -o: path of the comparison result

• -d: name of the single-operator layer to be compared of the My Output model
• -t: (optional) indicates whether the dump file is the input or output (default).
• -i: (optional) input_index or output_index of the operator of the My Output model
• -custom: (optional) customized path to store the .py file for format conversion, which should be the upper-level directory of the format_convert directory. For details about the .py file requirements, see Preparing a Customized .py File for Format Conversion.

Select the -f or -q argument based on the data prepared in Comparison Data Description.

Comparison Procedure

To conduct vector comparison, perform the following steps:

• Replace the .json file and directory names in this example with the actual ones. The result and log paths must be created in advance and the HwHiAiUser user must have the read and write permissions on the paths.
• This section describes how to compare the dump data of a non-quantized model running on the Ascend AI Processor and the .npy file of a non-quantized Caffe model. The following parameters are based on this example. You can replace them as required.
• Single-operator comparison between two groups of dump data generated based on the same model running on the Ascend AI Processor is not supported.

1. Log in to the development environment as the HwHiAiUser user.
2. Run the export command to set the environment variable and generate a .json file.

   Set the following environment variable:

   export LD_LIBRARY_PATH=/home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/lib64:${LD_LIBRARY_PATH}

   Generate the .json file:

   /home/HwHiAiUser/Ascend/ascend-toolkit/latest/atc/bin/atc --mode=1 --om=/home/HwHiAiUser/data/resnet50.om --json=/home/HwHiAiUser/data/resnet50.json

3. Go to the /home/HwHiAiUser/Ascend/ascend-toolkit/latest/toolkit/tools/operator_cmp/compare directory.
4. Run the vector comparison command as follows:

   python3.7.5 compare_vector.pyc -l /home/HwHiAiUser/MyApp_mind/resnet50 -r /home/HwHiAiUser/Standard_caffe/resnet50 -f /home/HwHiAiUser/data/resnet50.json -o /home/HwHiAiUser/result -d pool5 -i 0

5. Figure 5-11 and Figure 5-12 show the content of a vector comparison result file.
   Figure 5-11 Single-operator comparison result (summary)
   

   The single-operator comparison result summary is stored in {op_name}_input_{index}_summary.txt or {op_name}_output_{index}_summary.txt. The parameters are described as follows:

   • TotalCount: number of data records in the dump data of the operator
   • LeftOp: operator name of the My Output model
   • RightOp: operator name of the Ground Truth model
   • Format: data format
   • MinAbsoluteError & MaxAbsoluteError: minimum and maximum absolute error ranges
   • MinRelativeError & MaxRelativeError: minimum and maximum relative error ranges
   Figure 5-12 Single-operator comparison result (details)
   

   The detailed comparison result of the single-operator is stored in {op_name}_input_{index}_{file_index}.csv or {op_name}_output_{index}_{file_index}.csv. Each file records a maximum of one million data records. The parameters in Figure 5-12 are described as follows:

   • N C H W: data coordinates
   • Left: dump data of the My Output model operator
   • Right: dump data of the Ground Truth model operator
   • RelativeError: relative error. The value is obtained by dividing the AbsoluteError value by the dump value of the Ground Truth operator. If the dump value of the Ground Truth operator is 0, a hyphen (-) is displayed.
   • AbsoluteError: absolute error. The value is the difference between the dump value of the My Output operator and that of the Ground Truth operator.