Performance

Use this section to review model performance across one or more inference runs. It contains two pages: Summary and Timeline.

Summary

The Performance Summary page presents overall inference statistics, along with charts that break down runtime by operator. When ONNX Runtime profiling is enabled, the total inference time (including layers executed on the CPU) is displayed. When NPU profiling is enabled with the ai_analyzer_profiling flag, a chart of runtime by NPU operator type is shown.

At the top of the page, a summary of model performance characteristics is displayed, including:

• Number of inferences: The total number of inferences on which all reported statistics are based.

• Clock Frequency: Displays the assumed NPU clock frequency. This value is editable. When it is changed, all timestamp data (collected in clock cycles but displayed in time units) is recalculated accordingly.

• Average Time per Inference: Calculated by dividing total execution time by the number of inferences. This metric provides an overall view of model performance across multiple runs.

• Max/Min Inference Time: Displays the maximum and minimum inference times observed across all runs to highlight performance variability. These values can help identify outliers or bottlenecks in specific runs.

Performance Characteristics

NPU Inference Time

NPU inference time is generally consistent across inference operations, indicating stable accelerator-side performance.

Execution Provider Inference Time

Execution Provider inference time can vary significantly between initial and subsequent inference operations due to the following factors:

• Initialization Overhead: The first several inference operations include one-time initialization costs, which are reflected in measured inference time.

• Steady-State Performance: After initialization completes, these overhead costs are removed, resulting in improved performance for subsequent inferences.

Impact on Average Performance

The impact of initialization overhead on reported average performance depends on the total number of inference operations:

• Large Sample Sizes: For workloads with many inferences, initialization overhead has minimal impact on overall average performance.

• Small Sample Sizes: For workloads with few inferences, initialization overhead can significantly skew average performance metrics.

Note: When evaluating performance, consider the number of inference operations in your workload to interpret reported averages accurately.

The following charts are only displayed when detailed DDR profiling data is enabled, as described in DDR Throughput Profiling and provided that the environment variable AI_ANALYZER_ADVANCED is set:

• Running vs. Stall Cycles: Displays the number of cycles spent running versus stalled on the overall runtime of the model.

• Channel Running Cycles: Displays the number of cycles spent running for every column and channel of the NPU.

• Operator Running Cycles: Displays the number of cycles spent running for every operator type of the model. This chart is useful for identifying which operator types are consuming the most execution time and may benefit from optimization.

• Channel Stream Starvation Stall Cycles: Displays the number of cycles spent stalled for every column and channel of the NPU due to stream starvation. This chart is useful for identifying which columns and channels are starving due to the NoC that cannot keep pace with the NPU execution.

• Operator Stream Starvation Stall Cycles: Displays the number of cycles spent stalled for every operator type of the model due to stream starvation.

• Channel Backpressure Stall Cycles: Displays the number of cycles spent stalled for every column and channel of the NPU due to backpressure. This chart is useful for identifying which columns and channels are experiencing backpressure due to the L2 Memory that cannot keep pace with the NPU execution.

• Operator Backpressure Stall Cycles: Displays the number of cycles spent stalled for every operator type of the model due to backpressure.

Cross-probing between this summary, the detailed timeline, and timeline tables allows for quick localization of the layers that are causing performance bottlenecks. For example, if you click the pie section surrounded by a red square, the corresponding layer is automatically highlighted in the timeline and the timeline table:

Timeline

The Performance Timeline provides a layer-by-layer view of model execution. The upper section presents a graphical timeline of layer execution, while the lower section displays the same data in tabular form. The Timeline page shows either single inferences or all of them.

By default, it displays all inferences. The runtime is represented by 4 horizontal bars:

• Inference Id: Displays the inference number.

• NPU name: Displays the NPU partition name of the model. There may be multiple partitions if the model is partitioned across multiple NPU partitions.

• CPU and NPU Execution: Displays the execution time of each layer on the CPU and the overall NPU execution time.

• NPU Layer Execution: Displays the execution time of each layer on the NPU. Each layer is represented as a horizontal bar, with bar length corresponding to layer execution time and bar position indicating when the layer executed during inference.

If multiple threads in the processor are competing for access to the NPU, the timeline shows the multiple threads in parallel, with each thread represented as a horizontal bar group.

If profiling data is available for multiple inferences, use the Inferences chooser to select which inference to display:

After selecting a specific inference, the timeline displays only that inference’s execution data:

Click on Kernel Execution to concentrate the view on the NPU execution Timeline.

Click on Overview, just before the inference number, to return to the view of all inferences.

Toolbar

• Show or hide the properties panel for selected objects using the Show Properties toggle.

• Show or hide the table under the timeline using the Show Table toggle.

• Export the graphical timeline as SVG using Export to SVG.

• In the vaiml partition view, an additional control is available to change display units (cycles or time) and clock frequency (GHz) used for timeline data.

Select Show NPU Compute Block Execution to display the execution details of individual NPU compute blocks within the timeline. This view provides a detailed view of the NPU base compute block used for each layer:

In this view, each NPU compute block’s execution timeline is displayed. Each operator is shown, decomposed in multiple compute (CMP) or data movement (MOV) phases:

Table Toolbar

• Show or hide the search area using Show/Hide Search. Use this to locate specific layers in the table.

• Save the table in CSV format using Export to CSV.

• Use Reset Settings to restore default column display settings.

• Use Show/Hide by Type to select which layer types are displayed in the table.

• Use Show/Hide Columns to select which columns are displayed.

Table Columns

The table columns are grouped into generic and specific categories to provide comprehensive insight into model execution and compute performance.

• Generic columns: General information about model layers and their characteristics.

  • “Name”: Displays the name of the model layer.

  • “Start/End Time (µs)” and “Start/End Cycle”: Provide basic layer execution timing information.

  • “Inference”: Displays the inference number.

  • “ONNX RT Identifier”: Displays the ONNX Runtime identifier, the NPU name.

  • “Type”: Displays the layer type within the NPU or CPU.

  • “µC Variability (%)”: Provides temporal metrics for models processing variable-size input samples. This metric quantifies the time difference between completion of the first sample inference and completion of the final sample inference in the same input sample set, as observed by the micro-controllers. High variability might indicate routing inefficiencies or memory spilling.

• Execution Time columns: Layer execution-time metrics during inference.

  • “µs” and “Cycles”: Execution time of the layer for the selected inference.

  • “% NPU” and “% Total”: Ratio of layer execution time to NPU execution time or total inference execution time.

  • “Average”, “Min”, “Max” (cycles or µs): Execution time statistics across all inferences.

• NPU column usage (this is accurate for VART, but for ORT the starting column is just inferred)

  • “First Column”: The first column in the AI Engine array used by the model runtime execution.

  • “Total columns”: The total number of columns in the AI Engine array used by the model runtime execution.

The following options are only displayed when detailed DDR profiling data is enabled, as described in DDR Throughput Profiling:

• L3 Throughput Min Read (MB/s) or L3 Throughput Min Write (MB/s) columns: Data-transfer performance for each layer. These columns report the minimum observed throughput, indicating the layer that might be experiencing a memory-access bottleneck. A value significantly below 10 GB/s (for a 1.25 GHz AI Engine array clock) might indicate memory-access bottlenecks.

  • “MB/s”: Minimum observed throughput for the layer, in MB/s.

  • “Column:Channel”: The column and channel on which the minimum throughput was observed.

  • “Backpressure Stall (%)”: The percentage of time the layer experienced backpressure stalls due to memory access contention.

  • “Starvation Stall (%)”: The percentage of time the layer experienced starvation stalls due to insufficient data availability.

• L3 Throughput (MB/s) columns: Data-transfer performance for each layer.

  • “ChN Read/Write”: Bandwidth observed during execution of the layer for channel N.

  • “ChN Backpressure Stall (%)”: The percentage of time channel N experienced backpressure stalls due to memory access contention.

  • “ChN Starvation Stall (%)”: The percentage of time channel N experienced starvation stalls due to insufficient data availability.