Quantization Configuration

If the inference accuracy of the config.json quantization configuration file generated by the create_quant_config call does not meet the requirements, you need to tune the config.json file until the accuracy is as expected. The following is an example of the file content.

{
    "version": 1,
    "batch_num": 30,
    "activation_offset": true,
    "do_fusion":true,
    "skip_fusion_layers":[],
    "layer_name1":{
        "quant_enable": false,
        "activation_quant_params":[
            {
                "max_percentile":0.999999,
                "min_percentile":0.999999,
                "search_range":[0.7, 1.3],
                "search_step":0.01
            }
        ],
 
        "weight_quant_params":[
            {
                "channel_wise":true
            }
        ]
    },
     "layer_name2":{
         "quant_enable": true,
        "activation_quant_params":[
            {
                "max_percentile":0.999999,
                "min_percentile":0.999999,
                "search_range":[0.7, 1.3],
                "search_step":0.01
            }
        ],
 
        "weight_quant_params":[
            {
                "channel_wise":true
            }
        ]
    }
}

The following describes the configuration options available in the configuration file.

If the inference accuracy of the model quantized based on the default configuration in the config.json file drops significantly, perform the following steps to tune the quantization configuration file:

1. Execute the quantization script in the amct_tensorflow_sample.tar.gz package to perform quantization based on the default configuration generated by the create_quant_config API.
2. If the inference accuracy with the model quantized in 1 is as expected, configuration tuning ends. Otherwise, go to 3.
3. Tune batch_num in the quantization configuration file.

   batch_num controls the batch count for quantization. Tune it based on the batch size and the number of images required for quantization. Generally, a larger quantity of data samples used in a quantization process indicates a smaller accuracy loss after quantization. However, excessive data does not improve accuracy, but occupies more memory, reduces the quantization speed, and may result in insufficient memory, video RAM, and thread resources. Therefore, it is recommended that the product of batch_num and batch_size be 16 or 32.

4. If the inference accuracy with the model quantized in 3 is as expected, configuration tuning ends. Otherwise, go to 5.
5. Tune quant_enable in the quantization configuration file.

   quant_enable specifies whether to quantize a layer. If set to true, the layer is to be quantized. If set to false, otherwise. If the configuration of a layer is not present, the quantization of the layer is skipped. Generally, specifying less layers to quantize improves quantization accuracy. When the network accuracy is not as expected, locate the quantization-sensitive layers (whose error increases significantly after quantization, such as the top layer, bottom layer, depthwise convolutional layer, and layers with few parameters) in the network, and disable quantization on these layers as needed.

6. If the inference accuracy with the model quantized in 5 is as expected, configuration tuning ends. Otherwise, go to 7.
7. Tune the values of activation_quant_params and weight_quant_params in the quantization configuration file.
   • Data is clipped to the range [left,right] specified by the activation_quant_params parameters. Generally, values distributed near a boundary are sparse, and clip may be performed on all the values, to improve the accuracy. A larger value of min_percentile (max_percentile) indicates that left (right) is closer to the minimum value (maximum value) of the to-be-quantized data. search_range and search_step affect the range of [left, right]. Generally, a larger value of search_range and a smaller value of search_step may achieve higher quantization accuracy, but the quantization takes more time.
   • channel_wise in weight_quant_params determines whether to use a different quantization factor for each channel during weight quantization. If set to true, channels are separately quantized using different quantization factors. If set to false, all channels are quantized at a time using the same quantization factors. Generally, the inference accuracy is higher if the channels are separately quantized. However, the MatMul and AVE Pooling layers are channel-irrelevant. Therefore, this parameter does not take effect on these layers.

8. If the inference accuracy with the model quantized in 7 is as expected, configuration tuning ends. Otherwise, it indicates that quantization has severe adverse impact on the inference accuracy. In this case, remove the quantization configuration.

Figure 4-5 Configuration tuning workflow