Compiled Model Output

Compiled Model Output#

After successful compilation, the AMD Vitis™ AI Execution Provider generates compiled model artifacts in the cache_dir/cache_key directory. These files contain all the necessary binaries and metadata required for model execution on the NPU at runtime.

Understanding the Compiled Model#

When you compile a model, the entire directory structure created in cache_dir/cache_key constitutes your compiled model. All files within this directory are required for runtime inference.

What gets created:

  • Binary files optimized for NPU execution

  • Model metadata and configuration

  • Runtime execution graphs

  • Compilation logs and summary reports

Directory Structure#

After compilation, your cache_dir contains the following structure:

Flat-Buffer Format (.rai file - Default):

cache_dir/
  cache_key.rai                  # Single packaged model file

Directory Structure Format:

cache_dir/
  cache_key/
    vaiml_par_0/
      # Compiled model files
    final-vaiml-pass-summary.txt  # Compilation summary report
  [additional compilation artifacts]

Model Storage Formats#

The compiler supports two storage formats for the compiled model:

Flat-Buffer Format (.rai file)#

A single .rai file that packages all model artifacts in a compact binary format. This format uses memory-mapped file loading for efficient zero-copy access.

What to copy to the board: Copy only the single .rai file to your target board.

When to use:

  • For production deployment

  • When transferring models to target hardware

  • For large models where single-file management is easier

  • When you need faster loading times

Generating the Flat-Buffer Format

The .rai file is generated by default by the Vitis AI compiler during model compilation. No special option is required in this case.

Directory Structure Format#

The compiled model is stored in cache_dir/cache_key as a directory hierarchy containing the following:

  • vaiml_par_0/ - Partition subdirectory containing NPU-specific binaries and runtime files

  • final-vaiml-pass-summary.txt - Compilation summary with performance metrics

  • Model metadata and intermediate compilation artifacts (useful for debugging)

What to copy to the board: Copy the entire cache_key directory to your target board, maintaining the directory structure.

When to use:

  • During development and debugging

  • When you need to inspect compilation artifacts

  • When you want detailed logs and reports

Generating the compiled cache directory

The ‘keep_outputs’: ‘true’ option is required in the compilation configuration JSON file to retain the compiled cache directory.

provider_options = {
    'config_file': '/path/to/vaip_config.json',
    'target': 'VAIML',
    'cacheDir': './cache',
    'cacheKey': 'my_model',
    'keep_outputs': 'true'  # the compiled cache directory is retained
}

Compilation Summary Report#

After compilation, you can review detailed compilation results in the summary report.

Viewing the Compilation Summary#

To view compilation details, display the content of the summary file:

cat cache_dir/cache_key/final-vaiml-pass-summary.txt

Understanding the Summary Report#

The compilation summary provides critical information about how your model was compiled, which parts of it were offloaded to the NPU, and how long the compilation took.

Example Summary Report:

--------- Final Summary of VAIML Pass ----------
OS: Linux X64
Model: /workspace/MyModel.onnx
Model signature: 206d5ce5f4eb839e49dbdcaee82c36bfCompiler Information
  Target Device: ve2-xc2ve3858
  Device Data Type: bfloat16
  Flow: default
  Version: VAIP 98c936dccf6966097281d75dae58ff08e588edfc
  FlexML Version: vai_6_3 (hash: 415683bbeeb66e9461935a2aabf9e34fbdc45885, built: 2026-09-19-15:08:32)
  Overlay: aie2_6x4x4.yaml
  NPU Frequency: 1267 MHz
  AIE Single Core Compiler: peano
  Recipes: mlopslib
  DP size: 1
  TP size: 6
  Optimize Level: 2
  Preferred Data Storage: auto
  Threshold GOPs Percent: 2
  Logging Level: info
  Preemption: disabled
  Model data type: float32
Number of operators in the model: 426
GOPs of the model: 35.7351
Number of operators supported by VAIML: 426 (100.000%)
GOPs supported by VAIML: 35.735 (100.000%)
Number of subgraphs supported by VAIML: 1
  NOTE: Number of subgraph supported by VAIML does not include subgraphs below GOPs% threshold.
Number of operators offloaded by VAIML: 426 (100.000%)
GOPs offloaded by VAIML: 35.735 (100.000%)
Number of subgraphs offloaded by VAIML: 1
Number of partitions offloaded to NPU: 1
Number of partitions executed on CPU: 0
Number of subgraphs with compilation errors (fall back to CPU): 0
Number of subgraphs below 2% GOPs threshold (fall back to CPU): 0
Number of subgraphs above max number of subgraphs allowed(7): 0 (fall back to CPU)
Stats for offloaded subgraphs
Subgraph vaiml_par_0 stats:
    Type: npu
    Operators: 426 (100.000%)
    GOPs : 35.735 (100.000%)  OPs: 35,735,126,932
    fp32 ops %: 98.388Compilation Information
  Frontend (FE): 10.7 s (2.6%)
  Backend (BE): 398.0 s (95.9%)
    AIE Compile: 343.3 s (82.7%)
  Overhead (partitioner, cache, etc.): 6.4 s (1.5%)
  Total: 415.2 s

Compilation Settings#

The Compiler Information block records the target and the configuration used for this compilation.

Setting

Description

Target Device

Target NPU device and part number (for example, ve2-xc2ve3858)

Device Data Type

NPU execution precision (for example, bfloat16)

Model data type

Original model precision (for example, float32)

NPU Frequency

NPU clock frequency, in MHz, assumed by the compiler when scheduling the model

DP size

Data-parallelism count. The number of independent copies of the model running in parallel, each processing a different portion of the batch.

TP size

Tensor-parallelism count. The number of NPU partitions a single copy of the model is split across, so one inference is computed cooperatively.

Optimize Level

Optimization level applied; higher levels trade compile time for runtime performance

Threshold GOPs Percent

Minimum share of total model GOPs a subgraph must have to be offloaded. Smaller subgraphs fall back to the CPU.

Logging Level

Verbosity of the compiler log (for example, info)

Key Metrics Explained#

Metric

Description

Model signature

Unique hash identifying your model

Number of operators

Total operators in your model

GOPs

Giga Operations - computational complexity measure

Operators supported by VAIML

Percentage of operators that can run on NPU

GOPs supported by VAIML

Percentage of computation that can run on NPU

Number of subgraphs supported by VAIML

Subgraphs the Vitis AI compiler is able to compile. This count excludes subgraphs that fall below the Threshold GOPs Percent value.

Operators offloaded by VAIML

Operators actually running on NPU (after optimization)

GOPs offloaded by VAIML

Computation actually running on NPU (after optimization)

Number of subgraphs offloaded by VAIML

Model partitions created for NPU execution

Number of partitions offloaded to NPU

Partitions dispatched to the NPU at runtime

Number of partitions executed on CPU

Partitions that remain on the CPU execution provider

Subgraph type

Execution target: npu or cpu

OPs

Absolute operation count for the subgraph

fp32 ops %

Share of the subgraph’s operations that originate from float32 operators before conversion to the device data type

Total

End-to-end wall-clock compilation time

Tip

Compilation results are cached against the Model signature. Recompiling an unchanged model reuses the cached artifacts and completes far faster than the initial build.

Interpreting Compilation Results#

Optimal Compilation (100% NPU offload):

Number of operators offloaded by VAIML: 426 (100.000%)
GOPs offloaded by VAIML: 35.735 (100.000%)
Number of subgraphs offloaded by VAIML: 1
Number of partitions offloaded to NPU: 1
Number of partitions executed on CPU: 0

This indicates all operations run on the NPU for maximum performance.

Partial NPU Offload:

Number of operators offloaded by VAIML: 305 (71.596%)
GOPs offloaded by VAIML: 24.500 (68.560%)
Number of subgraphs offloaded by VAIML: 2
Number of partitions offloaded to NPU: 2
Number of partitions executed on CPU: 1
Number of subgraphs with compilation errors (fall back to CPU): 1

This indicates some operations fall back to CPU execution. Check the reasons:

  • Compilation errors: Subgraph failed to compile for NPU

  • Below GOPs threshold: Subgraph too small, CPU execution more efficient. The threshold comes from Threshold GOPs Percent (2 in the preceding example), so the line reads Number of subgraphs below 2% GOPs threshold (fall back to CPU).

  • Above max subgraphs (7): Too many partitions, excess falls back to CPU

Details on model deployment on the board can be found in ONNX Runtime Python Inference.

Compiler Stack Size Requirement#

The Vitis AI compiler requires sufficient stack space to run. If the available stack size is too small, the compiler stops and reports an error indicating that the failure may be due to inadequate stack size.

In this situation, the compiler may emit the following message in the output log:

Please check the output log for errors and fix those before you run the application. If a stack trace is printed above, refer to it for debugging. If no stack trace or errors are visible, the issue may be caused by insufficient stack size. To increase the stack size, execute the command 'ulimit -s unlimited' and re-run the compiler.
Compilation Failed
(WARNING:0, CRITICAL-WARNING:0, ERROR:1)

This message is generated by the compiler and indicates that you should:

  1. Inspect the output log for explicit errors or stack traces.

  2. If no clear cause is visible, increase the stack size, for example by running:

    ulimit -s unlimited

    and then re-run the compiler.

Contents