RunnerType::VAIML Configuration

This page documents the model loading, configuration options, and supported formats specific to vart::RunnerType::VAIML (Versal AI Edge Gen2). Other RunnerTypes may have different configuration options, supported types, and model loading mechanisms.

vart::RunnerType::VAIML supports models compiled by the Vitis AI compiler. The compiled model may consist entirely of NPU partitions or include CPU partitions for operators handled by the Vitis AI compiler-supported CPU kernels. This partitioning is determined at compile time by the Vitis AI compiler, not at runtime.

Subgraph Topology and Tensor Type Availability

The subgraph at the model's input/output boundary determines which vart::TensorType views vart::Runner::get_tensors_info() exposes for that direction:

HW (NPU) subgraph boundary – both TensorType::HW and TensorType::CPU views are available.
CPU subgraph boundary – only TensorType::CPU is available; TensorType::HW returns an empty vector.

Common topologies:

Topology	HW input?	HW output?
`HW`	Yes	Yes
`CPU → HW`	No	Yes
`HW → CPU`	Yes	No
`CPU → HW → CPU`	No	No

When a boundary is a CPU subgraph, input_tensor_type or output_tensor_type must be set to "CPU"; using "HW" causes runner creation to fail. When a boundary is a HW subgraph, either "HW" (hardware-native, zero-copy) or "CPU" (ONNX-compatible, runner performs format conversion) can be used based on application requirements.

Model Loading Modes

vart::RunnerType::VAIML accepts the compiled model in one of two artifact forms via the model_path argument to vart::RunnerFactory::create_runner():

Directory-based – A VAI (Vitis AI) compiled model cache/directory.
**.rai file** – A single-file FlatBuffer archive of the compiled model. Pass the file path; the runner memory-maps it internally.

// Directory-based loading
auto runner = vart::RunnerFactory::create_runner(
    vart::RunnerType::VAIML, "/path/to/compiled_model_dir");
 
// .rai file loading
auto runner = vart::RunnerFactory::create_runner(
    vart::RunnerType::VAIML, "/path/to/model.rai");

Configuration Options

Configuration options are passed as a std::unordered_map<std::string, std::any> to vart::RunnerFactory::create_runner(). All options are optional; unrecognized keys are silently ignored. Default values are listed below.

Tensor Configuration

Option	Type	Default	Description
`input_tensor_type`	`std::string`	`"HW"`	`"HW"` for hardware-native format, `"CPU"` for ONNX-compatible format. Determines whether input tensors go through pre-processing. At a HW-subgraph input boundary, either `"CPU"` or `"HW"` can be used as needed. At a CPU-subgraph input boundary, use `"CPU"`; `"HW"` causes runner creation to fail.
`output_tensor_type`	`std::string`	`"HW"`	`"HW"` for hardware-native format, `"CPU"` for ONNX-compatible format. Determines whether output tensors go through post-processing. At a HW-subgraph output boundary, either `"CPU"` or `"HW"` can be used as needed. At a CPU-subgraph output boundary, use `"CPU"`; `"HW"` causes runner creation to fail.
`skip_in_bo_sync`	`bool`	`false`	Skip input buffer sync for HW tensor types. When `true`, the application is responsible for syncing input tensors before inference. For tensors allocated via vart::Runner::allocate_npu_tensor() / vart::Runner::allocate_sub_tensor(), call vart::NpuTensor::sync_buffer(). For application-allocated wrapped buffers, the application must manage the sync itself.
`skip_out_bo_sync`	`bool`	`false`	Skip output buffer sync for HW tensor types. When `true`, the application is responsible for syncing output tensors before reading. For tensors allocated via vart::Runner::allocate_npu_tensor() / vart::Runner::allocate_sub_tensor(), call vart::NpuTensor::sync_buffer(). For application-allocated wrapped buffers, the application must manage the sync itself.

NPU Resource Configuration

Option	Type	Default	Description
`cma_index`	`int32_t`	`0`	CMA memory bank index for XRT BO allocation by the runner.
`aie_columns_sharing`	`bool`	`true`	`true` = shared mode (NPU columns shared), `false` = exclusive mode (columns reserved for this runner).
`start_column`	`uint32_t`	auto	Starting column for NPU overlay execution. Defaults to first available column.

Async Execution Configuration

Option	Type	Default	Description
`async_threadpool_depth`	`uint32_t`	`10`	Number of threads in the async thread pool.
`max_concurrent_runs`	`uint32_t`	`2`	Maximum number of concurrent inference threads.
`callback_order`	`std::string`	`"submission"`	Callback invocation order. `"submission"` = strict submission order. `"completion"` = invoked as jobs complete (non-deterministic). See Callback Order Modes.

General

Option	Type	Default	Description
`log_level`	`std::string`	`"INFO"`	Logging verbosity: `"ERROR"`, `"WARNING"`, `"INFO"`, `"DEBUG"`.
`debug`	`bool`	`false`	Enable runtime-level debug messages.
`config_json`	`std::string`	empty	Path to Vitis AI configuration file (e.g., `"vitis_ai_config.json"`). This is the same JSON configuration used during model compilation.
`ai_analyzer_profiling`	`bool`	`false`	Enable AI Analyzer profiling for performance analysis.

Callback Order Modes

When using vart::Runner::execute_async() with a callback, the callback_order option controls when callbacks are invoked:

submission (default) – Callbacks fire in strict submission order. If jobs A, B, C are submitted in that order, the callback for A fires first, then B, then C – even if B finishes before A. This provides deterministic, predictable ordering. Use when applications require results in the same order as requests.
completion – Callbacks fire immediately when each job completes, regardless of submission order. If B finishes before A, B's callback fires first. This provides maximum responsiveness and throughput. Use when ordering does not matter and maximum responsiveness is preferred.

Supported Data Types

The following data types are currently supported by vart::RunnerType::VAIML. The vart::DataType enum in the public header defines the full set across all RunnerTypes.

BOOLEAN, INT8, UINT8, INT16, UINT16, BF16, FP16, INT32, UINT32, FLOAT32, INT64, UINT64

Supported Memory Layouts

The following memory layouts are currently supported by vart::RunnerType::VAIML. The vart::MemoryLayout enum in the public header defines the full set across all RunnerTypes.

NHW, NHWC, NCHW, HCWNC4, HCWNC8, HCWNC16, GENERIC

When the memory layout is GENERIC, the NpuTensorInfo::memory_layout_order vector specifies the dimension permutation order relative to the CPU tensor format.