Running Inference with Custom Operators

Running Inference with Custom Operators#

Overview#

This section describes how to run a model containing custom operators on the VEK385 board using x_plus_ml_vart.

Before following these steps, ensure you have:

Step 1 — Prepare the Custom Ops Model#

Use the AI skill (Custom operator workflow) or manual authoring to prepare the custom ops model and generate all required artifacts. Confirm the following are present before proceeding:

  • Modified custom op ONNX model

  • Compilation cache directory (<cache_dir>/<cache_key>/)

  • vitisai_config.json

  • Custom op config folder containing the YAML and associated files (e.g., head_conv_int8/)

  • Reference data (NumPy input files), e.g., ./head_conv_int8/reference_data/

Step 2 — Identify Required Artifacts#

The complete set of artifacts needed for an on-board run is:

Artifact

Description

Modified custom op ONNX model

The ONNX model with custom op nodes wired in

Compilation cache

Output of the Vitis AI compile step (libadf.a, ctrl_pkts.xclbin, ELFs, etc.)

vitisai_config.json

Compilation and runtime configuration

Custom op config folder

Directory containing the YAML config and related files (e.g., head_conv_int8/)

custom_ops_config.json

Schema info for ORT registration (created in step 4)

Reference data

NumPy .npy input files for validation

Step 3 — Copy Artifacts to the Board#

Transfer all artifacts identified in step 2 to the target board, preserving the directory structure so that relative paths in the configs remain valid.

Step 4 — Prepare custom_ops_config.json for the Board#

Create a custom_ops_config.json that maps each custom op to its domain, name, input/output count, and YAML config path. The YAML path must be relative to where runmodel.py is invoked on the board.

{
  "mydomain.head_conv": {
    "domain": "mydomain",
    "name": "head_conv",
    "inputs": 2,
    "outputs": 1,
    "op_config": "head_conv_int8/custom_op_head_conv/head_conv.yaml"
  }
}

Tip

The YAML file is located within the op config folder you copied (e.g., head_conv_int8/). Search for .yaml files in that directory to confirm the correct relative path.

Step 5 — Update vitisai_config.json#

Edit vitisai_config.json on the board to ensure the op_config path for each custom op under vaiml_config.custom_ops points to the correct YAML location on the board filesystem.

Step 6 — Prepare runmodel.py#

Before creating the ONNX Runtime inference session, set the required environment variable and register the custom op schemas.

Schema registration and session creation:

import json
import numpy as np
import onnxruntime as ort
from onnxruntime_custom_ops import (
    register_dynamic_custom_ops_to_onnxruntime,
    vaiml_custom_op_schema,
)

# 1. Register custom op schemas BEFORE creating the session
with open("custom_ops_config.json") as f:
    custom_ops = json.load(f)

ops = [
    vaiml_custom_op_schema(
        domain=op["domain"],
        name=op["name"],
        nb_inputs=op["inputs"],
        nb_outputs=op["outputs"],
    )
    for op in custom_ops.values()
]
register_dynamic_custom_ops_to_onnxruntime(ops)

# 2. Create the inference session with VitisAIExecutionProvider
session_options = ort.SessionOptions()
providers = [
    (
        "VitisAIExecutionProvider",
        {
            "config_file": "vitisai_config.json",
            "cacheDir": "<cache_dir>",
            "cacheKey": "<cache_key>",
        },
    )
]
session = ort.InferenceSession(
    "<model>.onnx",
    sess_options=session_options,
    providers=providers,
)

# 3. Load reference inputs and run
input_name = session.get_inputs()[0].name
input_data = np.load("reference_data/input_0.npy")
outputs = session.run(None, {input_name: input_data})

Important

register_dynamic_custom_ops_to_onnxruntime must be called before ort.InferenceSession(...) is constructed. Calling it after session creation has no effect and the op will not be found.

Step 7 — Execute on the Board#

Run runmodel.py on the board, verifying the following are correctly set:

Parameter

What to verify

cacheDir / cacheKey

Points to the compilation cache copied in step 3

vitisai_config.json path

Matches the file updated in step 5

custom_ops_config.json path

Matches the file created in step 4

Reference data path

Matches the NumPy input files copied in step 3

 
python runmodel.py

Compare the output against the CPU reference data in reference_data/ to confirm numeric correctness.

Additional Resources#

Contents