Auto Tuning in Inference Scenario

Prerequisites

• Prepare the development environment and operating environment by referring to Environment Preparations and install the required software.
• Configure the environment variables on which the Auto Tune tool depends by referring to Environment Variable Configuration.

Tuning During Model Conversion with ATC

When using the ATC to convert models, you can enable Auto Tune by setting --auto_tune_mode="xx".

Example command:

atc --model=./tune.pb --framework=3 --output=./add_tune --output_type=FP16 --soc_version=Ascend310 --auto_tune_mode="RL,GA"

• By default, no log is generated during ATC model conversion. To generate Auto Tune log (INFO-level only), include the --log=info option in the ATC command. The Auto Tune log is generated to the /var/log/npu/slog/host-0/host-0_*.log file. For details about the ATC command-line options, see "Restrictions and Parameters" in ATC Tool Instructions.
• You can set the following tuning functions through environment variables:
  • If an operator in the network model has a match in the repository, the operator will not be tuned repeatedly by default. You can configure the REPEAT_TUNE environment variable to forcibly tune the operator again.
  • You can configure the TUNE_OPS_NAME environment variable to tune a specified operator layer.
  • The Auto Tune tool also provides other environment variable functions. For details, see Environment Variable Configuration.
  • It is allowed to start multiple ATC processes for tuning on the host. Proper process parallelism improves the tuning efficiency. However, due to resource restrictions, the tuning efficiency decreases when the number of parallel processes reaches a certain limit. The following condition should be met:

    ATC process count x TE_PARALLEL_COMPILER x 2 < Host CPU core count TE_PARALLEL_COMPILER indicates the number of parallel operator build processes.

    In the TBE operator parallel build scenario (that is, TE_PARALLEL_COMPILER > 1), it is recommended that one ATC process correspond to one device.

Tuning During IR Model Building

When using the Ascend Graph API to build an offline model, you can set the global_options parameter of the model building initialization API aclgrphBuildInitialize as follows to enable the Auto Tune tool:

std::map<std::string, std::string> global_options = {
    {ge::ir_option::SOC_VERSION, "Ascend310"},
    {ge::ir_option::EXEC_DISABLE_REUSED_MEMORY, "0"},         
    {ge::ir_option::AUTO_TUNE_MODE, "RL,GA"}
};    
auto status = aclgrphBuildInitialize( global_options );

By default, no log is generated during IR model building. To generate Auto Tune log (INFO-level only), add the following configuration to the options parameter of the model build API aclgrphBuildModel:

    {ge::ir_option::LOG_LEVEL, "info"} 

The Auto Tune log is generated to the /var/log/npu/slog/host-0/host-0_*.log file. For details about how to build an IR model, see IR Model Building Guide.

You can set the following tuning functions through environment variables:

• If an operator in the network model has a match in the repository, the operator will not be tuned repeatedly by default. You can configure the REPEAT_TUNE environment variable to forcibly tune the operator again.
• You can configure the TUNE_OPS_NAME environment variable to tune a specified operator layer.
  • The Auto Tune tool also provides other environment variable functions. For details, see Environment Variable Configuration.
  • It is allowed to start multiple graph building processes for tuning on the host. Proper process parallelism improves the tuning efficiency. However, due to resource restrictions, the tuning efficiency decreases when the number of parallel processes reaches a certain limit. The following condition should be met:

    Graph building process count x TE_PARALLEL_COMPILER x 2 < Host CPU core count TE_PARALLEL_COMPILER indicates the number of parallel operator build processes.

    In the TBE operator parallel build scenario (that is, TE_PARALLEL_COMPILER > 1), it is recommended that one graph building process correspond to one device.

Offline Tuning Based on Dump Data

The detailed operations are as follows:

1. Obtain the dump data.

   The dump data refers to the operator output description file and the operator binary file. The prerequisites for generating the dump data include:

   1. Configure dump-related environment variables. Enable dump.
      • LD_LIBRARY_PATH, PYTHONPATH, and ASCEND_OPP_PATH are required environment variables for configuring Auto Tune. For details, see Environment Variable Configuration.

        export install_path=/home/HwHiAiUser/Ascend/ascend-toolkit/latest
        export LD_LIBRARY_PATH=${install_path}/acllib/lib64:${install_path}/atc/lib64:$LD_LIBRARY_PATH
        export PYTHONPATH=${install_path}/atc/python/site-packages:${install_path}/atc/python/site-packages/auto_tune.egg/auto_tune:${install_path}/atc/python/site-packages/schedule_search.egg:$PYTHONPATH
        export ASCEND_OPP_PATH=${install_path}/opp

      • Enable dump.

         export ENABLE_TUNE_DUMP=True

      • Set the dump path.

         export TUNE_DUMP_PATH=/home/HwHiAiUser/DumpData

   2. Use the ATC tool to convert the model or build an IR model (the Auto Tune tool does not need to be enabled) to generate dump data.
      • For details about how to use the ATC tool, see ATC Tool Instructions.
      • For details about how to build an IR model, see IR Model Building Guide.

      After model conversion is complete, dump data is generated in the path specified by TUNE_DUMP_PATH.

2. Perform offline tuning based on the dump data with Auto Tune.

   The entry script for offline tuning is in the ATC installation path: atc/python/site-packages/schedule_search.egg/schedule_search/msoptune.py. Run the following command to start offline tuning by executing this Python file:

   python3.7 {msoptune.py path} --start {dump_path}

   Example:

   python3.7 /home/HwHiAiUser/Ascend/atc/python/site-packages/schedule_search.egg/schedule_search/msoptune.py --start /home/HwHiAiUser/DumpData

   • /home/HwHiAiUser/Ascend: ATC installation path. Replace it as required.
   • /home/HwHiAiUser/DumpData: dump path, either an absolute path or a path relative to the directory where the current script is executed.

   Currently, only one process is allowed for offline tuning on the host.

Custom Repository

Tuning Result File

When the tuning starts, a file named tune_result_pidxxx_{timestamp}.json is generated in the Auto Tune working directory, which records the tuning process and result.

During the tuning, a tune_show_pidxxx_{timestamp} folder is generated in the tuning working directory, which stores the flag file of each tuned operator. If you want to cancel the tuning of an operator, run the following command:

python3.7 /home/HwHiAiUser/Ascend/atc/python/site-packages/schedule_search.egg/schedule_search/msoptune.py --stop tune_show_{timestamp}_pidxxx

Select the operator that you want to cancel tuning as prompted. When tuning is complete, the tool compares the current repository with the existing repository. If the current repository outperforms the existing one, the existing repository is replaced in the custom directory or a new repository is added to the custom directory. Otherwise, no new repository is generated.

The tune_show_pidxxx_{timestamp} folder is automatically deleted after the tuning is complete.