Vitis AI Runtime (VART) Overview

Vitis AI Runtime (VART) Overview#

The Vitis AI Runtime (VART) is the software layer used to execute compiled machine learning (ML) models and surrounding pipeline stages on Versal™ AI Edge Series Gen 2 platforms. This chapter introduces how the pieces fit together and suggests an order for reading the rest of the documentation. For Vitis AI 6.3 feature highlights and Vitis AI 6.2 → Vitis AI 6.3 migration notes, see What’s new in Vitis AI 6.3 at the end of this page. Earlier VART release notes are in Changelog.

VART primarily consists of VART-ML (model execution APIs) and VART-X (video and I/O pipeline building blocks). Some reference applications in this guide also use ONNX Runtime + Vitis AI Execution Provider as an alternative inference path.

How VART-ML and VART-X connect#

VART-ML focuses on running the compiled model: loading the compiler partition (graph, metadata, and binaries from the Vitis AI compiler mapped to the AI Engine/NPU—typically a model cache directory or a .rai flatbuffer file), describing inputs and outputs as NpuTensor objects, and executing synchronous or asynchronous Runner APIs.

VART-X focuses on everything around the model in a typical vision application: frame buffers; preprocess (resize, format conversion, normalization) through the VART-X PreProcess module and its image-processing pipeline; postprocess (softmax, NMS, custom); metaconvert and overlay for visualization; helper types such as InferResult; and the Memory module, which manages raw physical memory allocations and deallocations. Preprocess can run on programmable logic (PL)—the Versal FPGA fabric—or on the CPU, depending on platform and configuration. The compiled ML model executes on the AI Engine/NPU, while PL can provide accelerated image-processing and other platform-defined kernels when integrated in the design.

ONNX Runtime with the Vitis AI Execution Provider is an alternative inference backend that can be used in similar application patterns.

Three deployment lanes#

Use this to decide where to start.

VART three deployment lanes — ONNX Runtime + Vitis AI EP, VART-ML, and VART-X pipeline

  1. ONNX Runtime + Vitis AI Execution Provider — Familiar ONNX Runtime InferenceSession APIs. Often used for quick bring-up.

  2. VART-ML — C++ Runner API for compiled models that execute on the NPU, including fully offloaded models and models compiled with the CPU partition feature for heterogeneous NPU/CPU execution where supported. Provides explicit tensors and optional zero-copy when you manage hardware buffers. Use when latency and throughput dominate.

  3. VART-X–centric pipelines — Device, VideoFrame, preprocess, postprocess, overlay, and related modules for end-to-end vision flows. You still run inference inside the pipeline using either ONNX Runtime + Vitis AI EP or VART-ML; VART-X owns the frame lifecycle and non-model stages.

On the host, the same Vitis AI quantization and compilation flow produces the compiled model artifact for all three preceding lanes. The lanes differ on the target in how you invoke inference and move data: ONNX Runtime session APIs versus VART-ML Runner APIs versus VART-X pipeline orchestration, and how tensors and buffers are allocated and shared—not in requiring a separate compiler workflow per lane.

For INT8 models on VART, compile-time flags and runtime quantization configuration (VART-X JSON, VART-ML APIs, or manual scales) are described in Quantization Parameters for VART Applications.

Suggested reading order#

Follow this order to build the story from scope to code:

  1. Choosing ONNX Runtime + Vitis AI EP vs VART-ML (and the role of VART-X) — Introduction: VART X and VART ML

  2. VART-ML architecture (Runner, tensors, zero-copy) — VART ML Architecture Overview

  3. VART-X architecture (modules and data flow) — VART X Architecture Overview

  4. Application development walkthrough — VART Application Development

  5. Prebuilt examples matrix and quick starts — Reference Applications

After that, use debugging, tensor layout reference, the NPU Format Selection Guide (preprocess-config.colour-format for vision pipelines), Post Processing Functions, Quantization Parameters for VART Applications, and the VART ML API class list and VART X API class list in this chapter as needed.

What’s new in Vitis AI 6.3#

The following summarizes notable VART-ML and VART-X additions in Vitis AI 6.3, followed by migration and compatibility notes when moving from Vitis AI 6.2, and known limitations.

Feature summary#

VART ML#

What’s new

  • Mixed tensor configuration - added runner options input_tensor_type.<tensor_name> and output_tensor_type.<tensor_name> so CPU and HW tensors can be mixed within the same direction.

  • Memory bank selection - added Runner::allocate_npu_tensor(info, memory_bank) to allocate an NpuTensor from a selected supported memory bank.

  • Runtime log control - added the VART_LOG_LEVEL environment variable to override the runner’s log_level option.

  • XRT run pooling - added the enable_xrt_run_pool runner option to reduce CPU overhead by reusing XRT run objects; it increases memory footprint and is therefore disabled by default.

  • Directional batch size - added Runner::get_batch_size(TensorDirection direction) to get the batch size based on direction.

VART X#

What’s new

  • Added new PreProcessType::PANSCAN, which center-crops the source to the output aspect ratio and resizes to fill it, with no padding and no distortion.

  • Added new overloaded PreProcess::process(const std::vector<PreProcessOp>& preprocess_ops, std::vector<PreProcessOpRes>& ops_res) API to report the applied scale, crop, and padding, which can be used during InferResult transformation.

  • Introduced the vart::Logger module, with console, file, syslog, and optional external logging system.

  • Device::get_device_hdl() now allows xclbin_loc to be left empty.

Migration and compatibility (Vitis AI 6.2 to Vitis AI 6.3)#

VART ML#

Applications must be recompiled for VART-ML 0.4.0 because of ABI changes:

  • New virtual methods were added to Runner, which changes the vtable layout.

  • Two new values were added to MemoryLayout before GENERIC, which renumbers the existing enum values.

  • The data type of the cma_index runner option changed from int32_t to uint32_t.

VART X#

Applications must be recompiled for VART-X 0.4.0 because of ABI changes:

  • New fields are added to PreProcessInfo, TensorInfo and InferResScaleInfo.

  • PreProcess::process() now takes its operation vector by const reference.

  • Applications that currently derive detection scaling, crop, or padding on their own should stop recomputing those values and bind the geometry reported by PreProcess::process(const std::vector<PreProcessOp>& preprocess_ops, std::vector<PreProcessOpRes>& ops_res) into vart::InferResScaleInfo before calling InferResult::transform().

Known limitations#

VART ML#

  • For RunnerType::VAIML, Runner::execute_async() can have lower throughput and higher latency in this release than in Vitis AI 6.2.

See also

For the high-level product view of quantization, compilation, and deployment, see Vitis AI Overview at the start of this guide.

Contents