Common Data Preprocessing Faults

Symptom

Frame loss occurs during video decoding, causing ghosting or discontinuity. One or more of the following errors are found in the device-side logs.

• Log error (1)

• Log error (2)

• Log error (3)

• Log error (4)

• Log error (5)

• Log error (6)

• Log error (7)

Possible Cause

The possible causes are as follows:

• Possible cause for log error 1: The IDR frame is missing in the H.264 streams.
• Possible cause for log error 2: The I-frame is missing in the H.264 streams.
• Possible cause for log error 3: The P-frame is missing in the H.264 streams.
• Possible cause for log error 4: The B-frame is missing in the H.264 streams.
• Possible cause for log error 5: The IDR frame is missing in the H.265 streams.
• Possible cause for log error 6: The P-frame is missing in the H.265 streams.
• Possible cause for log error 7: The I-frame or B-frame is missing in the H.265 streams.

Solution

To rectify the fault, perform the following steps:

1. Check whether the input source stream is correct.

   Use a third-party tool (for example, eseye u) to check whether the input streams are normal.

2. If the source stream is normal, the stream may be damaged when being transmitted to the VDEC on the device side. In this case, save the stream transmitted to the VDEC by calling fwrite() before calling aclvdecSendFrame.

   Build and deploy the sample of the corresponding version in CLI mode, decode the reserved stream, and check whether the verification stream is normal or whether the VDEC supports the format.

   If the sample decoding is normal, the development code is incorrect. You can find the VDEC sample code in Application Software Development Guide and optimize the VDEC code.

Symptom

When the input streams are decoded by the VDEC, the obtained decoded data is incorrect, causing artifacts, as shown in Figure 2-9. In addition, information similar to dvpp_vdec_vdm_process failed and Chan 0 ErrRatio = 44 exists in the log.

Figure 2-9 Video artifacts

Possible Cause

Some frame data in the input streams is incomplete or there are bad frames, causing artifacts during hardware decoding.

Solution

To rectify the fault, perform the following steps:

1. Check whether the input source stream is correct.

   Use a third-party tool (for example, eseye u) to decode and play the input streams and check whether artifacts occur. If artifacts do not occur, go to 2. If artifacts occur, replace the input streams.

2. If the source stream is normal, the stream may be damaged when being transmitted to the VDEC on the device side. In this case, save the stream transmitted to the VDEC by calling fwrite() before calling aclvdecSendFrame.

   Build and deploy the sample of the corresponding version in CLI mode, decode the reserved streams, and check whether artifacts occur.

   If the sample decoding is normal, the development code is incorrect. You can find the VDEC sample code in Application Software Development Guide and optimize the VDEC code.

Symptom

The VDEC decoding performance deteriorates, which is less than the performance specifications released by the VDEC. As a result, frame freezing occurs.

Possible Cause

The possible causes are as follows:

• The video decoding callback function takes a long time, affecting the decoding performance.
• If the ratio of I-frames in the input streams is too large, the time required for decoding I-frames is longer than that for decoding P-frames, affecting the decoding performance.
• Abnormal frames exist in the input streams, which affect the decoding performance.

Solution

To rectify the fault, perform the following steps:

1. Record the dotting test duration in the callback function and check whether the duration is too long. (The maximum duration allowed by the callback function is related to the frame rate. The formula is as follows: Maximum duration = 1/Frame rate. For example, if the frame rate is 30 fps, the maximum duration is 1/30 fps = 0.033s.)
2. Use a third-party tool to open the input streams and check whether the percentage of I-frames is too high. Generally, the GOP value is 30 (that is, the I-frame interval is 30). If the percentage of I-frames is too high, replace the I-frames with normal streams for performance testing.
3. Use a third-party tool to open the input streams and check whether there are abnormal frames (for example, artifacts or decoding errors). If there are abnormal frames, the specifications are not met.

Symptom

The video fails to be played, and the error information shown in Figure 2-10 and Figure 2-11 is printed in logs.

Figure 2-10 Log message: Invalid width or height

Figure 2-11 Log message: bit_depth_luma(*) not epual *

Possible Cause

Based on the preceding log analysis, the input stream specifications may not meet the following requirements:

• The VDEC supports the stream width ranging from 128 to 4096 and the stream height ranging from 128 to 4096.
• The VDEC supports 8-bit H.264 streams and 8-bit or 10-bit H.265 streams.

Fault Locating

1. Check whether the width and height of the input stream meet the VDEC requirements.

2. Check whether the input stream type meets the VDEC requirements.

Solution

Check whether the streams input to the VDEC meets the following requirements:

Use a third-party tool (for example, eseye u) to check whether the stream type, width, and height meet the requirements. If the streams do not meet the requirements, replace them with those that meets the requirements.

Symptom

When streams are input to the VDEC for decoding, the callback function is not triggered for some or all frames. As a result, users cannot receive the decoding result.

Possible Cause

Some frames in the stream are bad frames. As a result, the syntax cannot parse the meaning of these frames, or these frames fail to be decoded. As a result, the callback function is not called.

Solution

To rectify the fault, perform the following steps:

1. Check whether the log contains the error information in Frame Loss or Packet Loss During Video Decoding. If yes, the callback function is not called back because the exception frames fail to be decoded.
2. If the error information in Frame Loss or Packet Loss During Video Decoding is not displayed, modify the DVPP module or change the global log level to info and check whether the total number of times that the following three logs are recorded is the same as the number of input frames.
   • (1) "The queue is empty, so call the non-intelligent pointer callback interface."
   • (2) "The queue is not empty, so call the smart pointer callback interface."
   • (3) "The queue is not empty, but hiai_data_sp is nullptr."

   The preceding three types of logs are generated in the following scenarios:

   • hiai_data_sp is not used. Data is successfully decoded and returned, log (1) is printed, and the callback function registered by the user is called.
   • Each frame corresponds to a setting of hiai_data_sp. Data is successfully decoded and returned, log (2) is printed, and the callback function registered by the user is called.
   • Set hiai_data_sp for N frames. If the first frame is successfully decoded and returned, log (2) is printed, and the callback function registered by the user is called. For other N-1 frames, log (3) is printed after successful decoding, and the callback function registered by the user is called.

   The callback function registered by the user is called in either of the preceding scenarios. That is, if the user callback function is called once, a frame is returned after decoding. Therefore, if the total number of occurrences of the preceding three logs is equal to the total number of frames input by the user, no frames are lost during decoding. In this case, you need to check whether the statistics of the decoding results received by the user are correct.

Symptom

The JPEGD module fails to decode streams. The following error information is recorded in the log:

Log message (1)

just support jpeg with YUV 444 422 420 400
do not support progressive mode
do not support arithmetic code, support huffman code only

Log message (2)

EOI segment of the stream is invalid

Possible Cause

The possible causes are as follows:

1. The data format is not supported.

   JPEGD supports only Huffman encoding (color space: YUV, subsample: 444/422/420/400), but does not support arithmetic encoding, progressive encoding, or JPEG2000 format.

2. Image data is incomplete.

Solution

To rectify the fault, perform the following steps:

1. You are advised to use third-party software to decode images in formats that are not supported currently.
2. If the image data is incomplete, use third-party software to check the binary file of the source image based on the error message.

   For example, if the error message "EOI segment of the stream is invalid" or "EOI segment of the stream is invalid, it should be FFD9. Try software decoding." is displayed, the EOI end delimiter is missing in the image. The corresponding binary image is similar to that shown in the following figure. A normal JPEG image should end with FF D9. The FF D9 mark code is missing in the data.

   If the source image data is incomplete, an error message is displayed. In this case, replace the data.

3. If the source image data is complete, the data may be damaged during transmission. Before calling acldvppJpegDecodeAsync, you need to save the streams transmitted to JPEGD by using the fwrite() function and compare the streams with the source image in binary format. If they are inconsistent, data loss occurs during the transmission. In this case, you need to further locate the data loss fault.

Symptom

The VPC module fails to be called. The following error message is displayed in the log:

Log message (1)

roiNum(0), input rightOffset(1918) is not odd

Log message (2)

if format is yuv420sp or yuv400, bareDataBufferSize(2073600) should be equel to widthStride(1920) * heightStride(1088) * 3/2

Possible Cause

The possible causes are as follows:

• Log message (1): The right offset coordinate of the VPC cropping area must be an odd number. In the log message, 1918 is an even number, which does not meet the requirement.
• Log message (2): When the input format is YUV420 SP or YUV400, the size of the input memory must be Width stride x Height stride x 3/2. The log message indicates that this requirement is not met.

• Log message (3): The width stride of the input image (memory occupied by each row of image) must be a multiple of 16. In the log, the width stride is 300, which is not a multiple of 16. You need to align the image and set the width stride to the aligned value.
• Log message (4): The input memory of VPC needs to be allocated by using acldvppMalloc.
• Log message (5): The output memory of VPC needs to be allocated by using acldvppMalloc.
• Log message (6): The cropping area of VPC must be greater than or equal to 10 x 6. The width of the cropping area in the log is only 8. Therefore, an error is reported.
• Log message (7): The VPC resizing range is [1/32, 16]. The log message indicates the resizing range, the width of the cropped image is 3840, and the output region width is 112. The calculation result is as follows: 112/3840 < 1/32. Therefore, an error is reported.

Fault Locating

1. Locate the VPC configuration parameter based on the error information in the log and modify the parameter as prompted.
2. Based on the error information in the log description, modify the VPC parameter by referring to Application Software Development Guide.

Solution

Modify the configuration file based on the error information.

1. For log message (1), the right offset of the cropped region of the input image is incorrect. Change it to an odd number.
2. For log message (2), the size of the input memory is incorrect. Check the code for allocating input memory. The size of the allocated memory should be 1920 x 1088 x 3/2, and the value of bareDataBufferSize should be 1920 x 1088 x 3/2.
3. For log message (3), the stride value of the input image does not meet the requirements. In this case, set the stride value to a multiple of 16.
4. For log message (4) and log message (5), allocate memory by using acldvppMalloc.
5. For log message (6), change the width of the cropped image.
6. For log message (7), change the resizing range.

Symptom

The JPEG image dataset is decoded into the YUV420 format by the JPEG decoding module of DVPP, resized by the VPC, converted into the RGB format by the AIPP, and then inferred by the AI Core (the inference network is ResNet-50 and YOLOv3). Compared with the following comparison experiment, the overall inference accuracy of the image dataset has dropped.

Comparison experiment: The JPEG image dataset is decoded into the RGB format by OpenCV, resized by OpenCV, and then inferred by the AI Core.

Possible Cause

The possible causes are as follows:

• The decoding algorithm of the JPEG module of DVPP uses the new libjpeg-turbo algorithm (OpenCV 3.4.4 or later). The trained model is generally obtained by training the libjpeg algorithm (decoding algorithm in OpenCV 2.4.9). As a result, this model is not the most suitable model for DVPP decoding.
• The VPC resizing algorithm of the DVPP uses the Huawei-developed algorithm. However, a trained model is usually obtained through training using the bilinear interpolation algorithm (default OpenCV algorithm). As a result, the model is not the best one for adapting to the DVPP resizing.

Fault Locating

Check whether the accuracy drop is caused by DVPP decoding and resizing. The detailed process is as follows:

1. Figure 2-12 shows the comparison flow. In the control group, OpenCV is used for decoding and resizing. If the accuracy of the model used is lower than that of the control group, the DVPP decoding and resizing are considered as the cause.
   Figure 2-12 DVPP decoding and resizing
   

2. Figure 2-13 shows the comparison flow. In the control group, OpenCV is used for decoding and resizing. If the accuracy of the model used is lower than that of the control group, the DVPP decoding is considered as the cause.
   Figure 2-13 DVPP decoding
   

3. Figure 2-14 shows the comparison flow. In the control group, OpenCV is used for decoding and resizing. If the accuracy of the model used is lower than that of the control group, the DVPP resizing is considered as the cause.
   Figure 2-14 DVPP resizing
   

Solution

You are advised to use DVPP JPEG decoding, VPC resizing, and AIPP CSC to obtain RGB images, and then perform incremental or full training on the existing model to obtain a new model for inference. For details, contact Huawei engineers.