FAQs

Symptom

In the cloud scenario, profile data fails to be collected, and the following information is displayed.

Possible Cause

The container directory or its subdirectory in the /var/log/npu/profiling/ directory is manually deleted, causing file write failure.

Solution

The Profiling tool will automatically recreate the directory. You only need to wait and start a new training job when the directory is recreated.

Symptom

During the Profiling installation, privilege escalation is performed before installing the GCC, g++, and make tools. The installations fail, and the information in Figure 3-25 is displayed.

Figure 3-25 GCC installation failure message

Possible Cause

The dependencies are installed after privilege escalation. As a result, the generated privilege escalation file is incorrect.

Solution

Perform the following steps:

1. Log in to the server as the root user and delete the /etc/sudoers.d/xxx_specific file.

   Replace xxx with the Profiling installation user name.

2. Log in to the server as the Profiling installation user.

Symptom

The error message shown in Figure 3-26 is displayed when a training job is delivered in system profiling or job profiling.

Figure 3-26 Error message: No usable temporary directory

Possible Cause

No usable temporary directory is displayed, the system disk space maybe full.

Solution

Perform the following steps:

Symptom

Profiling fails to be installed, and an error message in Figure 3-27 is displayed.

Figure 3-27 Error message: google.protobuf not installed

Possible Cause

The possible causes are as follows:

• Protobuf is not installed.
• Protobuf has been installed, but the Profiling installation user does not have enough permission.

Solution

Perform the following steps:

1. Log in to the OS as the root user.
2. Run the python3.7 command and then run import protobuf to check that protobuf is installed.
   • If protobuf is not installed, install it by referring to "Preparing the Environment."
   • If protobuf has been installed, go to the next step.

3. Run the whereis python3.7 command to find the directory where Python 3.7 is located.
4. Change the permission of the site-packages directory found in Step 3 to 550.

   chmod -R 550 site-packages

Symptom

The error message shown in Figure 3-28 is displayed, indicating that the training job has no profiling result.

Figure 3-28 No profiling result of a training job

Possible Cause

The possible cause is as follows:

The profiling result cannot be found because the corresponding data file is being analyzed while you are copying the profile data.

Solution

Perform the following steps:

1. Log in to the OS as the Profiling running user.
2. Run the ./stop.sh command to stop the Profiling process.

   cd /usr/local/Ascend/toolkit/tools/profiler/bin

   ./stop.sh

3. Delete irrelevant data from the result_dir directory.
4. Run the ./start.sh command to start the Profiling process.
5. Run the command to query the job again.