NPU Runtime Hang Analysis

NPU Runtime Hang Analysis#

Overview#

An NPU hang occurs when a Vitis AI design stops responding during execution and cannot continue processing its workload. This happens when compute operations get stuck indefinitely, data transfers stop moving, or communication between AI Engine tiles becomes deadlocked.

When a hang occurs, the system generates a core dump file that captures the NPU state at the moment of failure. This guide explains how to use the vaiprofile --core-dump command to analyze these core dump files and identify the root cause in your source code.

Purpose#

The vaiprofile --core-dump command analyzes NPU core dumps from hung Vitis AI designs and creates AIE status reports. This is the first step in resolving NPU runtime hang issues.

Basic Usage#

Setting Up the Docker Environment

Launch the Vitis AI Docker container:

docker run --ulimit stack=-1:-1 -it --rm --network host \
  -v <yourLicenseDir>:/usr/licenses \
  -v <yourWorkingDir>:/host_mount \
  xcoacasharbor.amd.com/vai-docker-local/vitis_ai_2ve_2026.1:1.9.0_2026_08_28_17757_patched bash

vaiprofile \
  --core-dump <coredumpfilename> \
  --dump-aie-status status.txt

Where:

  • <coredumpfilename>

    The NPU core dump file (for example, npu_core_dump_0.bin) produced by a failing Vitis AI run.

  • status.txt

    The output AIE status dump file generated from the core dump.

What Is Core Dump Analysis#

The vaiprofile --core-dump command analyzes NPU core dumps from hung Vitis AI designs and creates AIE status reports. This is the first step in resolving NPU runtime hang issues. The command converts the binary core dump into a human-readable status file that can be used to identify the root cause of the hang.

When to Use --core-dump#

Use vaiprofile --core-dump when:

  • You have a specific NPU core dump from a failing Vitis AI design run.

Steps#

The following steps describe this process in detail:

  1. Capture the Core Dump: When your application crashes or hangs, a .bin core dump file is automatically generated containing a snapshot of the AIE state at the time of failure.

  2. Convert to Status File: Run the ML Debug utility to convert the binary core dump into a human-readable status file:

    vaiprofile --core-dump <coredumpfilename> --dump-aie-status status.txt

  3. Analyze the Status Output: The status file reveals AIE state information that can help identify obvious failure modes:

    • Error halts

    • DMA out-of-range conditions

    • Bad memory access patterns

    • Other hardware-level diagnostic signals

  4. Share the status file with AMD: The status file can be shared with AMD for further analysis.

  5. Triage the Issue (Optional): Use the status output to determine whether the failure likely originated from:

    • Your custom operation kernel

    • Other system components

Contents