Terminology
Term |
Description |
---|---|
Synchronous/Asynchronous |
Scheduling is classified into the synchronous and asynchronous modes in this document from the perspective of the caller and executor. In the current scenario, if the API calling result is returned immediately without waiting for the device to complete the execution, the host scheduling is in asynchronous mode. If the API calling result is not returned until the device completes the execution, the host scheduling is in synchronous mode. |
Process/Thread |
Unless otherwise specified, the processes and threads mentioned in this document refer to the processes and threads on the host. |
Host |
The host side refers to the x86 or Arm server connected to the device side. The host side utilizes the NN compute capability provided by the device side to implement services. |
Device |
The device side refers to the hardware device that provides the host with the NN computing capability over the PCIe interface. |
Context |
A context is a container that manages the life cycle of its objects (including streams, events, and device memory). Streams and events in different contexts are isolated, and cannot be executed synchronously. There are two types of contexts:
|
Stream |
Streams preserve the execution order of a stack of asynchronous operations that are executed on the device in its original order. Stream-based kernel execution and data transfer can implement the parallelism of computing on the host, data movement between the host and the device, and computing on the device. Streams come in two types:
|
Event |
Supports task synchronization between streams by calling ACL APIs, including tasks between the host and the device, and tasks between devices. For example, if task of stream2 needs to be executed after task in stream1 is complete, you can create an event and insert it to stream1. |
AIPP |
The AIPP module is introduced for image preprocessing including Color Space Conversion (CSC), image normalization (by subtracting the mean value or multiplying a factor), image cropping (by specifying the crop start and cropping the image to the size required by the neural network), and much more. Static AIPP and dynamic AIPP modes are supported. However, the two modes are mutually exclusive.
|
Dynamic batch/image size |
In some scenarios, the batch size or image size is not fixed. For example, if an object recognition network is executed after object detection, the number of objects inferred each time is not fixed.
|
Channel |
In RGB color mode, a complete image consists of three channels: HSV stands for hue, saturation, and value (brightness) and is an alternative representation of the RGB color model. |
Standard form |
The device functions as an endpoint (EP) and co-works with the host (namely, an x86 or Arm servers) over the PCIe interface. In this case, CPU resources on the device can be accessed only by the host, and the related inference applications run on the host. The device provides only the NN computing capability for the server. |