Session Configuration in sess.run Mode

use_off_line

Whether the training is performed on the Ascend AI Processor.

• True: yes
• False (default): not (training is performed on the host CPU)

enable_data_pre_proc

Data preprocessing enable.

• True: enabled
• False (default): disabled

iterations_per_loop

Number of iterations per loop set by using set_iteration_per_loop in sess.run mode, that is, the number of iterations per training loop every sess.run() call on the device side.

The parameter value must be the same as the value of iterations_per_loop set by set_iteration_per_loop for function verification.

profiling_mode

Profiling enable.

• True: enabled. The Profiling option is determined by enable_options.
• False (default): disabled.

profiling_options

Option (or options separated by colons) to be traced in profiling.

• training_trace: iteration tracing. Collects software profile data of a training job and the AI Software Stack to profile the training job. Focuses on data augmentation, forward and backward propagation, and gradient aggregation and update.
• task_trace: task tracing. Collects the HWTS and AI Core hardware information of the Ascend AI Processor and the start and end of each task.
• op_trace: single-operator tracing. To do so, you need to construct a single-operator network and train the network using a training script. This option is exclusive with training_trace and task_trace.

fp_point

Required if training_trace is selected.

This parameter specifies the start operator on the training network for forward propagation on the iteration trace, to record the start timestamp of forward propagation.

Set the value to the name of the top operator in forward propagation. You can save the graph as a .pbtxt file by using tf.io.write_graph in the training script to obtain this name.

bp_point

Required if training_trace is selected.

This parameter specifies the end operator on the training network for backward propagation on the iteration trace, to record the end timestamp of backward propagation. bp_point and fp_point can be used to calculate the forward and backward time, respectively.

Set the value to the name of the bottom operator in backward propagation. You can save the graph as a .pbtxt file by using tf.io.write_graph in the training script to obtain this name.

enable_dump

Data dump enable

• True: enabled. The dump file path is read from dump_path. If dump_path is set to None, an exception occurs.
• False (default): disabled

dump_path

Dump path. This parameter is required when enable_dump or enable_dump_debug is set to True.

Create the specified path in advance in the environment (either in a container or on the host) where training is performed. The running user configured during installation must have the read and write permissions on this path. The path can be an absolute path or a relative path relative to the path where the training script is executed.

• An absolute path starts with a slash (/), for example, /home/HwHiAiUser/output.
• A relative path starts with a directory name, for example, output.

dump_step

Iterations to dump. Defaults to None, indicating that all iterations are dumped.

Separate multiple iterations using vertical bars (|), for example, 0|5|10. You can also use hyphens (-) to specify an iteration range, for example, 0|3-5|10.

dump_mode

Dump mode.

• input: dumps only operator inputs.
• output (default): dumps only operator outputs.
• all: dumps both operator inputs and outputs.

enable_dump_debug

Overflow detection enable.

• True: enabled. The dump file path is read from dump_path. If dump_path is set to None, an exception occurs.
• False (default): disabled

dump_debug_mode

Overflow detection mode.

• aicore_overflow: detects AI Core operator overflow, for checking whether the operator outputs abnormal maximum values (such as float16 65500, 38400, and 51200) when the inputs of the operator are normal. Once such fault is detected, analyze the cause of the overflow and modify the operator implementation based on the network requirements and operator logic.
• atomic_overflow: detects Atomic Add overflow, for checking modules involved in floating-point computing (such as SDMA) in addition to the AI Core.
• all: detects both AI Core operator overflow and Atomic Add overflow.

precision_mode

A string for the operator precision mode.

• allow_fp32_to_fp16 or None: If an operator does not support the float32 data type, the precision is reduced to float16. By default, this mode is selected.
• force_fp16: If an operator supports both float16 and float32 data types, float16 is forcibly selected.
• must_keep_origin_dtype: The original precision is retained.
• allow_mix_precision: Mixed precision is enabled. For operators of the float32 data type on a network, the precision of some float32 operators can be automatically reduced to float16 based on the built-in optimization policy. In this way, the system performance is improved and the memory usage is reduced with little accuracy loss. After this function is enabled, you can enable loss scaling to compensate for the accuracy drop caused by precision reduction. For details about the related APIs, see NPULossScaleOptimizer Constructor.

enable_reduce_precision

The current version does not support this parameter.

variable_format_optimize

Variable format optimization enable.

• True (default): enabled
• False: disabled

To improve training efficiency, during the initialization of network variables, the variables are converted into a data format that is more suitable for running on the Ascend AI Processor (for example, NCHW to NC1HWC0 conversion). However, this function should be disabled in special scenarios.

mix_compile_mode

Mixed computing enable.

• True: enabled
• False (default): disabled

In fully offloaded mode, all computing operators are offloaded to the device side. As a supplement to the fully offloaded mode, the mixed computing mode allows some operators that cannot be offloaded to be executed online in the frontend framework, improving the Ascend AI Processor flexibility for adapting to TensorFlow.

hcom_parallel

Whether to enable the AllReduce gradient update and forward and backward parallel execution.

• True: enabled
• False (default): disabled

graph_memory_max_size

Network static memory and maximum dynamic memory, which can be specified based on the network size. The value ranges from 0 to 256 x 1024 x 1024 x 1024 or 0 to 274877906944, in bytes. Due to Ascend AI Processor hardware restrictions, the sum of graph_memory_max_size and variable_memory_max_size cannot exceed 31 GB. If not set, 26 GB is used by default.

variable_memory_max_size

Variable memory, which can be specified based on the network size. The value ranges from 0 to 256 x 1024 x 1024 x 1024 or 0 to 274877906944, in bytes. Due to Ascend AI Processor hardware restrictions, the sum of graph_memory_max_size and variable_memory_max_size cannot exceed 31 GB. If not set, 5 GB is used by default.

auto_tune_mode

Auto Tune enable for tuning TBE operators during build to find the optimal performance configuration on the Ascend AI Processor.

Example: auto_tune_mode = "RL,GA"; If this parameter is not specified, auto tuning is disabled.

For details about the usage of the Auto Tune tool, see Auto Tune Tool Instructions.

stream_max_parallel_num

AI CPU and AI Core engine parallelism for parallel execution of AI CPU and AI Core operators.

Example: "DNN_VM_TF:10,DNN_V100:1"

DNN_VM_TF is the name of the AI CPU engine. In this example, the number of concurrent tasks of the AI CPU engine is 10.

DNN_V100 is the name of the AI Core engine. In this example, the number of concurrent tasks of the AI Core engine is 1.

The value range is [1, 13]. Defaults to 1.

is_tailing_optimization

Communication hangover optimization enable in distributed training scenarios to improve performance. By changing a computation dependency relationship, a computation operation that does not depend on the last AR (gradient aggregation fragment) is scheduled to be performed in parallel with the last AR, to optimize communication hangover.

• True: enabled
• False (default): disabled

This parameter must be used in pair with NPUOptimizer Constructor and the value must be the same as that of is_tailing_optimization in NPUOptimizer Constructor.

graph_run_mode

Graph run mode.

• 0: online inference
• 1 (default): training

op_debug_level

Operator debug enable.

• 0 (default): disables operator debug.
• 1: enables operator debug and generates a TBE instruction mapping file. In this case, an operator CCE file (.cce) and a Python-CCE mapping file (_loc.json), and operator .o and .json files are generated in the kernel_meta folder in the training script execution directory. You can locate the AI Core error by using the line numbers in the CCE code and TBE code of the error operator.
• 2: enables operator debug and generates a TBE instruction mapping file. In this case, an operator CCE file (.cce) and a Python-CCE mapping file (_loc.json), and operator .o and .json files are generated in the kernel_meta folder in the training script execution directory, and the build optimization is disabled by enabling the CCE compiler -O0-g. You can locate the AI Core error by using the line numbers in the CCE code and TBE code of the error operator.
• 3: disables operator debug and retains the operator .o and .json files in the kernel_meta folder in the training script execution directory.
  NOTICE:

  You are advised to set this parameter to 0 or 3 for training. If you need to locate AI Core errors, set this parameter to 1 or 2, which might compromise the network performance.

enable_scope_fusion_passes

Scope fusion pattern (or scope fusion patterns separated by commas) to take effect during build.

Both built-in and custom scope fusion pattern files are classified into the following two types:

• General: common scope fusion patterns applicable to all networks. It takes effect by default and cannot be manually invalidated.
• Non-General: scope fusion patterns applicable to specific networks. By default, it does not take effect. You can use enable_scope_fusion_passes to specify the fusion pattern list that takes effect.

enable_exception_dump

Whether to dump the inputs and outputs of error operators.

• 0 (default): disabled
• 1: enabled. Keeping this switch enabled does not compromise the performance.

op_select_implmode

Operator implementation mode select. Some operators built in the Ascend AI Processor can be implemented in either high-precision or high-performance mode.

• high_precision: high precision implementation mode. In high-precision mode, Newton's Method or Taylor's Formula is used to improve operator precision with fp16 input.
• high_performance (default): high performance implementation mode. The high-performance implementation mode refers to the optimal performance implementation without affecting the network precision with fp16 input.

optypelist_for_implmode

List of operator types. The operators in the list use the mode specified by the OP_SELECT_IMPL_MODE parameter.

This parameter is used in pair with the OP_SELECT_IMPL_MODE parameter, for example:

Set op_select_implmode to high_precision.

Set optypelist_for_implmode to Pooling.