Running the Compiled Model

Running the Compiled Model#

Choosing a Runtime#

The compiled model can be deployed on the hardware board using either of two runtimes:

  • ONNX Runtime with the AMD Vitis AI Execution Provider (Vitis AI EP): The original ONNX model is loaded in an ONNX Runtime inference session, and the Vitis AI EP dispatches the supported subgraphs to the NPU while the remaining subgraphs fall back to the CPU execution provider at run time.

  • The VART ML Runtime: The compiled model is loaded and executed directly on the NPU through the VART ML C++ API, without an ONNX Runtime dependency. Partitioning is decided at compile time, and zero-copy inference is supported.

Use the following guidance to select a path based on the compilation status of the model:

Status of the model

Recommended runtime

Rationale

The model has not been compiled yet

ONNX Runtime with the Vitis AI EP

Compilation is driven through the Vitis AI EP. Run the compilation phase first to produce the .rai file or the compiled cache directory, which the VART ML Runtime then loads.

The model is fully offloaded to the NPU

Either runtime

Both paths are functionally equivalent. Use ONNX Runtime for functional validation and initial bring-up, and the VART ML Runtime for production deployment, where its lower per-inference overhead and zero-copy inference maximize throughput.

The model is compiled with the CPU partition feature for VART ML

VART ML Runtime

Heterogeneous NPU/CPU execution is resolved at compile time, so the model runs end to end on the VART ML Runtime with production-level performance.

The model contains operators outside compiler or CPU partition support, or the workflow must integrate with the ONNX ecosystem

ONNX Runtime with the Vitis AI EP

Only the Vitis AI EP provides runtime CPU fallback, and the full ONNX graph, including the CPU-only operators, is executed by a single inference session.

For a detailed comparison of the two runtimes, refer to Choosing between the Vitis AI Execution Provider (ONNX Runtime) and the VART Runner (VART ML).

Files Needed on the Hardware Board#

The compilation phase produces the compiled model in one of two forms:

  • A .rai file, which is the default output and packages the compiled model in a single file.

  • A compiled cache directory, which holds the compiled model and the generated runtime shared libraries.

Either form can be copied to any folder on the Linux filesystem of the HW board.

To run the model on the hardware board, the following files are needed:

  • The .rai file or the compiled cache directory

  • A Python script or C++ application to run the model in the case of ONNX Runtime

  • A C++ application to run the model in the case of VART ML Runtime

In addition, when deploying with ONNX Runtime and the Vitis AI EP:

  • The original ONNX model

  • The JSON config file used to compile the model

The VART ML Runtime loads the compiled model directly and does not require the original ONNX model at run time.

Deploying with ONNX Runtime and the Vitis AI Execution Provider#

Just like model compilation, model inference is performed using an ONNX Runtime (ORT) inference session.

The ORT inference session should be initialized as follows:

  • The AMD Vitis AI Execution Provider must be used; otherwise, the model runs on the local CPU.

  • The ONNX model should be the same as the one used during the compilation phase.

  • The config_file provider option should point to the same configuration file as the one used during the compilation phase.

  • The cache_dir and cache_key provider options should point to the compiled cache directory. These settings do not have to match the ones used during compilation. The compiled model can be placed in any folder on the Linux filesystem of the HW board. The combination of cache_dir/cache_key should simply point to this folder.

If the input ONNX model does not match the compiled model, or if the Vitis AI EP cannot find the compiled model, the model is not deployed on the NPU.

If the input ONNX model matches the compiled model, the Vitis AI EP loads the compiled model and is ready to deploy it on the NPU.

Once the ORT inference session is initialized, the model is deployed using the ORT session run API. For more information about the ONNX Runtime API, refer to the online documentation.

Upon execution, the subgraphs containing operators supported by the NPU are executed on the NPU by the Vitis AI EP. The remaining subgraphs are executed on the CPU.

The following code can be used as a simple template to deploy models on the NPU:

import onnxruntime

onnx_model = ...        # Original ONNX model used during the compilation phase

provider_options_dict = {
    "config_file": ..., # Original JSON config file during the compilation phase
    "cache_dir":   ..., # Path to the cache directory
    "cache_key":   ..., # Subfolder in the cache directory with the compiled model
    "target":   ...,    # Target platform for Vitis AI execution provider
}

# Create the ORT inference session
session = onnxruntime.InferenceSession(
    onnx_model,
    providers=["VitisAIExecutionProvider"],
    provider_options=[provider_options_dict]
)

# Initialize input tensors
input_data = {}
for input in session.get_inputs():
    input_data[input.name] = ...

# Run model
outputs = session.run(None, input_data)

Deploying with the VART ML Runtime#

The VART ML Runtime executes a previously compiled model directly on the NPU, without an ONNX Runtime dependency.

  • The runtime is pointed at the .rai file produced during the compilation phase, or at the compiled cache directory. Both forms can be placed in any folder on the Linux filesystem of the HW board.

  • The original ONNX model is not needed at run time, because the compiled model already carries the information required to program the NPU.

  • The application queries the shapes and data types of the input and output tensors from the runtime, allocates the corresponding buffers, and submits inference jobs. Buffers can be shared with the NPU to enable zero-copy inference.

Note

Partitioning is resolved at compile time for the VART ML Runtime. Fully offloaded models and models compiled with the CPU partition feature for VART ML both run end to end on this path. However, there is no runtime CPU fallback: if the model contains operators outside compiler or CPU partition support, deploy it with ONNX Runtime and the Vitis AI EP instead.

For the C++ APIs, including code examples, refer to Vitis AI Runtime (VART) Overview.

Contents