Vitis AI EP Configuration File

Vitis AI EP Configuration File#

Provide a JSON configuration file using the config_file provider option when creating the ONNX Runtime inference session. The following example shows a complete configuration with commonly used options:

{
  "passes": [
   {
     "name": "init",
     "plugin": "vaip-pass_init"
   },
   {
    "name": "vaiml_partition",
    "plugin": "vaip-pass_vaiml_partition",
    "vaiml_config":
    {
      "device": "ve2-xc2ve3858",
      "optimize_level": 2,
      "logging_level": "info",
      "threshold_gops_percent": 20
    }
   }
  ],
  "target": "VAIML",
  "targets": [
    {
        "name": "VAIML",
        "pass": [
            "init",
            "vaiml_partition"
        ]
    }
  ]
}

This example includes all commonly used options. The device field within vaiml_config is mandatory; all other options in vaiml_config have defaults and can be omitted. See the individual option descriptions in the following sections for details on types, supported values, and defaults.

Configuration Options#

The following options can be specified in the vaiml_config section of the configuration file.

device#

Controls the target device for compilation.

Type: String
Required: Yes
Supported values: See Supported Devices for Compilation section

Example:

"vaiml_config": {
  "device": "ve2-xc2ve3558"
}

optimize_level#

Controls the compiler’s optimization level to balance performance, memory usage, and compile-time stability.

Type: Integer
Supported values: 1, 2, 3
Default: 2

"vaiml_config": {
  "optimize_level": 2
}

Optimization Levels#

Level	Description	Use Cases
1	Prioritizes stability with predictable memory management (fixed or ping-pong buffering). Maximum functional coverage with minimal compile-time risk.	Early development and debugging, very large models, maximum stability requirements
2	Enables advanced optimizations including kernel chaining and efficient L2 memory allocation. Falls back to DDR (external memory) if the model overflows L2 memory (NPU Mem Tiles). Reduces latency and minimizes DDR traffic.	Production builds, models benefiting from kernel chaining optimizations
3	Instructs the compiler to apply more aggressive latency optimizations beyond what is achieved through tensor parallelism size (`tp_size`) alone. O3 can be stacked on top of any `tp_size` setting.	When a model fails to meet its latency targets even after tuning `tp_size`. It is intended as a next-step option for squeezing out additional performance in latency-sensitive workloads.

Important: 3 is an Early Access (EA) option. As such, it might not be fully validated or production-ready, and it is recommended to use this option only when standard optimization paths have been exhausted.

Note: Kernel chaining combines multiple operations to reduce memory transfers. L2 memory refers to on-chip NPU memory tiles. DDR is external memory accessed when on-chip memory is insufficient.

Parallelism Configuration#

Data parallelism and tensor parallelism are strategies for distributing workload across the device. These can be configured independently based on your performance requirements.

dp_size#

Controls data parallelism, which instantiates the entire model multiple times across the device. With dp_size=4, four independent model instances process different inference requests simultaneously.

Type: Integer
Supported values: 1-6 (for ve2-xc2ve3558), 1-9 (for ve2-xc2ve3858)
Default: 1

Use data parallelism when:

You need to maximize throughput for concurrent requests
Your application handles multiple simultaneous inference requests (for example, processing multiple camera streams in video analytics)
Model size fits comfortably within a single processing unit’s memory

"vaiml_config": {
  "dp_size": 4
}

tp_size#

Controls tensor parallelism, which partitions a single inference request across multiple processing units. With tp_size=4, the computation for one request is divided into four parallel execution streams, reducing the time required to complete that request.

Type: Integer
Supported values: 0-6 (for ve2-xc2ve3558), 0-9 (for ve2-xc2ve3858)
Default: 0 (When set to 0, the compiler automatically selects an appropriate tp_size value based on the target device characteristics. For the ve2-xc2ve3858 device, the compiler resolves tp_size to 6.)

Use tensor parallelism when:

Minimizing per-request latency is critical
The model’s memory requirements exceed the capacity of a single processing unit
You process one inference request at a time or have low concurrency

"vaiml_config": {
  "tp_size": 4
}

Compiler Choice of tp_size and dp_size: The compiler log file records the values of tp_size and dp_size used during compilation. The compiler outputs the following info message when tp_size=0:

INFO: [VAIML-COMPILE 1000] [flexml-compile] tp_size is automatically set to 6 from the overlay because users selected tp_size auto mode (tp_size=0)

For more details on configuring data and tensor parallelism, refer to the Data Parallelism and Tensor Parallelism section.

preferred_data_storage#

Controls whether intermediate data is stored in vectorized or unvectorized format. Convolution-heavy models (CNNs) perform better with vectorized data. GEMM-heavy models (Transformers) perform better with unvectorized data. The auto mode selects the optimal layout automatically.

Type: String
Supported values: “vectorized”, “unvectorized”, “auto”
Default: “auto”

"vaiml_config": {
  "preferred_data_storage": "unvectorized"
}

threshold_gops_percent#

Directs operators to the NPU or CPU based on their GOPS (Giga Operations Per Second) performance threshold. Operators above the threshold execute on the NPU; those below execute on the CPU.

Type: Integer (percentage)
Supported values: 0-100
Default: 2

"vaiml_config": {
  "threshold_gops_percent": 30
}

logging_level#

Controls the verbosity of compiler logging output.

Type: String
Supported values: “info”, “warning”, “error”
Default: “error”

Level	Description
info	Details about significant events or actions, including comparative information between options
warning	Recoverable issues and differences between options
error	Critical failures that prevent program continuation (limited details)

Example

"vaiml_config": {
      "logging_level": "info"
}

keep_outputs#

Specifies whether to preserve intermediate compilation files or generate profiling files during execution for debugging.

Type: Boolean
Supported values: true, false
Default: false

Value	Description
true	The Vitis AI compiler preserves both the <cache-dir>/<cache-key>/<model>.rai file and the complete vaiml directory structure.
false	Only the <cache-dir>/<cache-key>/<model>.rai file is retained.

Example

"vaiml_config": {
      "keep_outputs": true
}

ai_analyzer_enhanced_profiling#

Specifies whether to enable enhanced profiling in AI Analyzer during compilation. When enabled, additional registers are set to allow detailed performance data collection during runtime.

Type: List
Supported values: control_instrumentation
Default: no enhanced profiling

Example

"vaiml_config": {
      "ai_analyzer_enhanced_profiling": ["control_instrumentation"]
}

profiling_runtime_config#

Provides a JSON field with additional configuration options for enhanced profiling during runtime. This field is optional and only applicable when ai_analyzer_enhanced_profiling is enabled. The specific configuration options within this field depend on the profiling features you wish to enable and are currently limited to “control_instrumentation”, which has 2 sub-properties:

interface_tile for DDR bandwidth analysis for a specific direction (read or write) and a specific channel (0 or 1).

aie_tile to analyze layer boundness: compute or IO.

control_instrumentation:

Type: JSON
Supported values: “interface_tile”, “aie_tile”

interface_tile:

Type: Text
Supported values: “detailed_ddr_<read|write>_bandwidth:<0|1>”

aie_tile:

Type: Text
Supported values: “compute_io_bound”

Example

"vaiml_config": {
  "profiling_runtime_config": {
      "control_instrumentation": {
          "interface_tile": "detailed_ddr_read_bandwidth:0",
          "aie_tile": "compute_io_bound"
      }
  }
}

static_batching#

Configures static batching behavior for ONNX models. Static batching sets a fixed batch size at model build or export time. Every inference request must provide exactly this number of inputs; supplying a different number results in an error. This approach is efficient, but the input batch size must remain constant.

The compiler has limited support for static batch sizes greater than 1. If compilation fails on such a model, set override_batch_size to force it to compile as batch size 1.

override_batch_size:

Type: Boolean
Description: Forces a model with a static batch size greater than 1 to compile as batch size 1. Use this when compilation fails at the model’s declared batch size.

Example:

"vaiml_config": {
   "override_batch_size": true
 }

Limitations: Static batch sizes above 6 are not supported.

Your application is unaffected: continue to submit the full batch as a single request through ONNX Runtime with the Vitis AI Execution Provider. The compiler and runtime treat the model as single-batch, and the runtime serializes the batch internally, processing the inputs one after another.

That serialization costs throughput. To process the inputs in parallel instead, set dp_size to the batch size alongside override_batch_size. For a batch-4 model:

"vaiml_config": {
  "override_batch_size": true,
  "dp_size": 4
}

Automatic FP32 Conversion for Mixed Models (VINT8 + FP32)#

Specifies how to automatically convert the remaining FP32 operations to lower precision (for example, BF16 or FP16) during model compilation for improved performance. These options are described in more detail in Mixed Precision Configuration Reference.

enable_f32_to_bf16_conversion:

Type: Boolean
Description: Enables automatic conversion of FP32 operations to BF16 during model compilation.
Default: True

enable_f32_to_fp16_conversion:

Type: Boolean
Description: Enables automatic conversion of FP32 operations to FP16 during model compilation.
Default: False

Vitis AI EP Configuration File

Contents

Vitis AI EP Configuration File#

Configuration Options#

device#

optimize_level#

Optimization Levels#

Parallelism Configuration#

dp_size#

tp_size#

preferred_data_storage#

threshold_gops_percent#

logging_level#

keep_outputs#

ai_analyzer_enhanced_profiling#

profiling_runtime_config#

static_batching#

Automatic FP32 Conversion for Mixed Models (VINT8 + FP32)#