Terminology

Synchronous/Asynchronous

Scheduling is classified into the synchronous and asynchronous modes in this document from the perspective of the caller and executor. In the current scenario, if the API calling result is returned immediately without waiting for the device to complete the execution, the host scheduling is in asynchronous mode. If the API calling result is not returned until the device completes the execution, the host scheduling is in synchronous mode.

Process/Thread

Unless otherwise specified, the processes and threads mentioned in this document refer to the processes and threads on the host.

Host

The host side refers to the x86 or Arm server connected to the device side. The host side utilizes the NN compute capability provided by the device side to implement services.

Device

The device side refers to the hardware device that provides the host with the NN computing capability over the PCIe interface.

Context

A context is a container that manages the life cycle of its objects (including streams, events, and device memory). Streams and events in different contexts are isolated, and cannot be executed synchronously.

There are two types of contexts:

• Default context: a default context created implicitly upon the acl.rt.set_device call that specifies a device for computation. Each device corresponds to a default context. A default context cannot be released by calling acl.rt.destroy_context.
• Explicitly created context (recommended): a context created explicitly upon the acl.rt.create_context call in a process or thread.

Stream

Streams preserve the execution order of a stack of asynchronous operations that are executed on the device in its original order.

Stream-based kernel execution and data transfer can implement the parallelism of computing on the host, data movement between the host and the device, and computing on the device.

Streams come in two types:

• Default stream: a default stream created implicitly upon the acl.rt.set_device call that specifies a device for computation. Each device corresponds to a default stream. A default stream cannot be released by calling acl.rt.destroy_stream.
• Explicitly created stream (recommended): a stream created explicitly upon the acl.rt.create_stream call in a process or thread.

Event

Supports task synchronization between streams by calling ACL APIs, including tasks between the host and the device, and tasks between devices.

For example, if task of stream2 needs to be executed after task in stream1 is complete, you can create an event and insert it to stream1.

AIPP

The AIPP module is introduced for image preprocessing including Color Space Conversion (CSC), image normalization (by subtracting the mean value or multiplying a factor), image cropping (by specifying the crop start and cropping the image to the size required by the neural network), and much more.

Static AIPP and dynamic AIPP modes are supported. However, the two modes are mutually exclusive.

• Static AIPP: During model conversion, set the AIPP mode to static and set the AIPP parameters. After the model is generated, the AIPP parameter values are saved in the offline model (.om). The same AIPP parameter configurations are used in each model inference phase.

  In static AIPP mode, batches share the same set of AIPP attributes.

• Dynamic AIPP: During model conversion, specify the AIPP mode to dynamic, and set different sets of dynamic AIPP parameters as required. In this way, different sets of parameters can be used for model inference. For details about the interface for setting dynamic AIPP attributes, see Dynamic AIPP Parameter Settings.

  In dynamic AIPP mode, batches can use different sets of AIPP parameters.

Dynamic batch/image size

In some scenarios, the batch size or image size is not fixed. For example, if an object recognition network is executed after object detection, the number of objects inferred each time is not fixed.

• Dynamic batch: The batch size is not fixed during inference.
• Dynamic image size: The image size (H x W) of each image is not fixed during inference.

Channel

In RGB color mode, a complete image consists of three channels: HSV stands for hue, saturation, and value (brightness) and is an alternative representation of the RGB color model.

Standard form

The device functions as an endpoint (EP) and co-works with the host (namely, an x86 or Arm servers) over the PCIe interface. In this case, CPU resources on the device can be accessed only by the host, and the related inference applications run on the host. The device provides only the NN computing capability for the server.