Model Loading and Execution

aclError aclmdlLoadFromFile(const char *modelPath, uint32_t *modelId)

model_id, ret = acl.mdl.load_from_file(model_path)

Loads offline model data (adapted to the Ascend AI processor) from a file. It is a synchronous interface. The memory is managed by the system.

Returns model ID after the model is loaded. The model ID is used for model identification in subsequent operations.

model_path: str, path for storing the offline model file. The file name is contained in the path. The user who runs the application must have the permission to access the path.

The offline model file is the *.om file that adapts to the Ascend AI processor.

model_id: int, pointer object of the model ID generated after the system loads the model.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

None

aclError aclmdlLoadFromMem(const void* model, size_t modelSize, uint32_t* modelId)

model_id, ret = acl.mdl.load_from_mem(model, model_size)

Loads offline model data from memory. The model execution memory is managed by the system. It is a synchronous interface.

Returns model ID after the model is loaded. The model ID is used for model identification in subsequent operations.

model: int, pointer object of the memory address of the model data.

• If the application runs on the host, allocate the memory of the host.
• If the application runs on the device, allocate the memory of the device.
• Obtain the application running mode by calling acl.rt.get_run_mode.

model_size: int, length of the model data in the memory, in bytes.

model_id: int, model ID generated after the model is loaded.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

None

aclError aclmdlLoadFromFileWithMem(const char *modelPath, uint32_t *modelId, void *workPtr, size_t workSize, void *weightPtr, size_t weightSize)

model_id, ret = acl.mdl.load_from_file_with_mem(model_path, work_ptr, work_size, weight_ptr, weight_size)

Loads offline model data (adapted to the Ascend AI processor) from a file. It is a synchronous interface. You can manage the memory for running the model.

Returns model ID after the model is loaded. The model ID is used for model identification in subsequent operations.

model_path: str, path for storing the offline model file. The file name is contained in the path. The user who runs the application must have the permission to access the path.

The offline model file is the *.om file that adapts to the Ascend AI processor.

work_ptr: int, address of the working memory (for storing model input and output data) required by the model on the device. The address is managed by users.

work_size: int, size of the working memory required by the model, in bytes.

weight_ptr: int, pointer of the model weight memory (for storing weight data) on the device. The pointer is managed by users.

weight_size: int, size of the weight memory required by the model, in bytes.

model_id: model ID generated after the model is loaded.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

None

aclError aclmdlLoadFromMemWithMem(const void *model, size_t modelSize, uint32_t *modelId, void *workPtr, size_t workSize, void *weightPtr, size_t weightSize)

model_id, ret = acl.mdl.load_from_mem_with_mem(model, model_size, work_ptr, work_size, weight_ptr, weight_size)

Loads offline model data from memory. The model execution memory is managed by the user. It is a synchronous interface.

Returns model ID after the model is loaded. The model ID is used for model identification in subsequent operations.

model_size: int, model data length, in bytes.

work_ptr: int, address of the working memory (for storing model input and output data) required by the model on the device. The address is managed by users.

work_size: int, size of the working memory required by the model, in bytes.

weight_ptr: int, pointer of the model weight memory (for storing weight data) on the device. The pointer is managed by users.

weight_size: int, size of the weight memory required by the model, in bytes.

model_id: int, model ID generated after the model is loaded.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

None

aclError aclmdlLoadFromFileWithQ(const char *modelPath, uint32_t *modelId, const uint32_t *inputQ,size_t inputQNum, const uint32_t *outputQ, size_t outputQNum)

model_id, ret = acl.mdl.load_from_file_with_q(model_path, input_q, input_q_num, output_q, output_q_num)

Loads offline model data (adapted to Ascend AI processors) from a file. The input and output model data is stored in the queue. It is a synchronous interface. The current version does not support this interface.

Returns model ID after the model is loaded. The model ID is used for model identification in subsequent operations.

model_path: str, path for storing the offline model file. The file name is contained in the path.

The user who runs the application must have the permission to access the path.

The offline model file is the *.om file that adapts to the Ascend AI processor.

input_q: int, queue ID. Each input model corresponds to a queue ID.

input_q_num: int, input queue size.

output_q: int, queue ID. Each output model corresponds to a queue ID.

output_q_num: int, size of the output queue.

model_id: model ID generated after the model is loaded.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

None

aclError aclmdlLoadFromMemWithQ(const void *model, size_t modelSize, uint32_t *modelId,const uint32_t *inputQ, size_t inputQNum, const uint32_t *outputQ, size_t outputQNum)

model_id, ret = acl.mdl.load_from_mem_with_q(model, model_size, input_q, input_q_num, output_q, output_q_num)

Loads offline model data (adapted to Ascend AI processors) from the memory. The input and output model data is stored in the queue. It is a synchronous interface. The current version does not support this interface.

Returns model ID after the model is loaded. The model ID is used for model identification in subsequent operations.

model: int, memory address of the model data.

model_size: int, length of the model data in the memory, in bytes.

input_q: int, queue ID. Each input model corresponds to a queue ID.

input_q_num: int, input queue size.

output_q: int, queue ID. Each output model corresponds to a queue ID.

output_q_num: int, size of the output queue.

model_id: model ID generated after the model is loaded.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

None

aclError aclmdlExecute(uint32_t modelId, const aclmdlDataset *input, aclmdlDataset *output)

ret = acl.mdl.execute(model_id, input, output_)

Executes model inference until the result is returned. It is a synchronous interface.

model_id: int, ID of the model to be inferred.

input: int, pointer object of the input data of model inference.

output: int, pointer object of the output data of model inference.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

Resources (such as streams and memory) associated with a single model_id are unique. Therefore, concurrent use in a multithreaded process is not allowed. Otherwise, the service is unavailable.

The input and output pointer objects must be created in advance.

aclError aclmdlExecuteAsync(uint32_t modelId, const aclmdlDataset *input, aclmdlDataset *output, aclrtStream stream)

ret = acl.mdl.execute_async(modelId, input, output_, stream)

Executes model inference. It is an asynchronous interface.

model_id: int, ID of the model to be inferred.

input: int, pointer object of the input data of model inference.

output: int, pointer object of the output data of model inference.

stream: int, pointer object of the stream.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

• For models with the same model_id, acl.mdl.execute_async cannot be called to perform model inference in the multi-stream concurrency scenario.
• Resources (such as streams and memory) associated with a single model_id are unique. Therefore, concurrent use in a multithreaded process is not allowed. Otherwise, the service is unavailable.

The stream, input, and output pointer objects must be created in advance.

aclError aclmdlUnload(uint32_t modelId)

ret = acl.mdl.unload(model_id)

Uninstalls the model and releases resources after the model inference is complete. It is a synchronous interface.

model_id: int, ID of the model to be uninstalled.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

When calling the acl.mdl.unload API to unload a model, ensure that the model is not being used by other APIs.

None

aclError aclmdlQuerySize(const char *fileName, size_t *workSize, size_t *weightSize)

work_size, weight_size, ret = acl.mdl.query_size(file_name)

Obtains the weight memory size and working memory size required for model execution based on the model file. It is a synchronous interface.

file_name: str, path of the model that needs to obtain memory information. The file name is contained in the path. The user who runs the application must have the permission to access the path.

work_size: int, size of the working memory required for model execution, in bytes.

weight_size: int, size of the weight memory required for model execution, in bytes.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

None

aclError aclmdlQuerySizeFromMem(const void *model, size_t modelSize, size_t *workSize, size_t *weightSize)

work_size, weight_size, ret = acl.mdl.query_size_from_mem(model, model_size)

Obtains the weight memory size and memory size required for model execution based on the model data in the memory. It is a synchronous interface.

model: int, pointer object of the model data whose memory information needs to be obtained.

model_size: int, model data length, in bytes.

work_size: int, size of the working memory required for model execution, in bytes.

weight_size: int, size of the weight memory required for model execution, in bytes.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

The execution and weight memory is on the device, which needs to be allocated and released by the user.

None

aclError aclmdlSetDynamicBatchSize(uint32_t modelId, aclmdlDataset *dataset, size_t index, uint64_t batchSize)

ret = acl.mdl.set_dynamic_batch_size(model_id, dataset, index, batch_size)

Sets dynamic batch sizes (number of images processed each batch) for model inference in dynamic batching scenarios.

The batch size comes from the batch size choices specified by the dynamic_batch_size parameter during model conversion. For details about model conversion, see ATC Tool Instructions.

You can also call aclmdlGetDynamicBatch to obtain the number of batch size choices as well as the batch size of each choice supported by the model.

model_id: int, model ID.

• acl.mdl.load_from_file, acl.mdl.load_from_mem, acl.mdl.load_from_file_with_mem, and acl.mdl.load_from_mem_with_mem are called to load the model. After the model is successfully loaded, the model ID is returned.

dataset: int, pointer object of the input data of a model.

• Data of type acl.model_dataset is used to describe input data for model inference, while data of type aclDataBuffer is used to describe input memory size and address.

index: int, index of the input dynamic batch, obtained by calling acl.mdl.get_input_index_by_name. For dynamic batch and image size, the input name is fixed to ascend_mbatch_shape_data. For dynamic AIPP, the input name is fixed to ascend_dynamic_aipp_data.

batch_size: int, batch size for model inference.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

Call acl.mdl.get_input_index_by_name to obtain the index. For dynamic batch and image size, the input name is fixed to ascend_mbatch_shape_data. For dynamic AIPP, the input name is fixed to ascend_dynamic_aipp_data.

aclError aclmdlSetDynamicHWSize(uint32_t modelId, aclmdlDataset *dataset, size_t index, uint64_t height, uint64_t width)

ret = acl.mdl.set_dynamic_hw_size(model_id, dataset, index, height, width)

Sets the height and width of the input image for model inference.

The image size comes from the image size choices specified by the dynamic_image_size parameter during model conversion. For details about model conversion, see ATC Tool Instructions.

You can also call acl.mdl.get_dynamic_hw to obtain the number of image size choices as well as the image size of each choice supported by the model.

model_id: int, model ID.

• acl.mdl.load_from_file, acl.mdl.load_from_mem, acl.mdl.load_from_file_with_mem, and acl.mdl.load_from_mem_with_mem are called to load the model. After the model is successfully loaded, the model ID is returned.

dataset: input data of a model.

• Data of type acl.model_dataset is used to describe input data for model inference, while data of type aclDataBuffer is used to describe input memory size and address.

index: int, index of the input dynamic batch, obtained by calling acl.mdl.get_input_index_by_name. For dynamic batch and image size, the input name is fixed to ascend_mbatch_shape_data. For dynamic AIPP, the input name is fixed to ascend_dynamic_aipp_data.

height: int, height to be set.

width: int, width to be set.

ret: int, error code.

• 0 indicates success.
• Other values indicate failure.

None

Call acl.mdl.get_input_index_by_name to obtain the index. For dynamic batch and image size, the input name is fixed to ascend_mbatch_shape_data. For dynamic AIPP, the input name is fixed to ascend_dynamic_aipp_data.