DDR Throughput Profiling

This section describes the detailed DDR profiling option available in AMD Vitis™ AI and explains how to use DDR memory throughput data within AI Analyzer to evaluate the efficiency of data transfers between the L2 [AI Engine Memory Tile] and L3 [DDR] cache levels.

Configuration Example for Detailed DDR Bandwidth Profiling

    {
"passes": [
    {
        "name": "init",
        "plugin": "vaip-pass_init"
    },
    {
        "name": "vaiml_partition",
        "plugin": "vaip-pass_vaiml_partition",
        "vaiml_config": {
            "device": "ve2-xc2ve3858", # for XC2VE3858
            "keep_outputs": true,
            "log_level": "info",
            "ai_analyzer_enhanced_profiling": [
                "control_instrumentation"
            ],
            "profiling_runtime_config": {
                "control_instrumentation": {
                "interface_tile": "detailed_ddr_read_bandwidth:0"
                }
            }
        }
    }
],
"target": "VAIML",
"targets": [
    {
        "name": "VAIML",
        "pass": [
            "init",
            "vaiml_partition"
        ]
    }
]
    }

Important

Prior to executing the model on hardware, select either read-bandwidth or write-bandwidth profiling, along with the target channel (0 or 1). Note that both metrics cannot be captured simultaneously within a single run. In the preceding configuration example, read-bandwidth profiling on channel 0 is enabled via the detailed_ddr_read_bandwidth:0 interface tile setting. To capture write-bandwidth on channel 0 instead, set the interface tile to detailed_ddr_write_bandwidth:0. To target channel 1, replace the trailing 0 with 1 in either setting.

Note

For the sake of DDR throughput analysis, it is recommended to run multiple inferences to be able to obtain more accurate and representative profiling data.

The following table is a summary of the various available profiling options and their corresponding interface tile settings:

Profiling Option	Interface Tile Setting	Metric Captured
Detailed Read Bandwidth (Channel 0)	“detailed_ddr_read_bandwidth:0”	Running cycles, backpressure cycles (from AI Engine Array) and starvation cycles (from DDR memory) when reading DDR memory
Detailed Read Bandwidth (Channel 1)	“detailed_ddr_read_bandwidth:1”	Running cycles, backpressure cycles (from AI Engine Array) and starvation cycles (from DDR memory) when reading DDR memory
Detailed Write Bandwidth (Channel 0)	“detailed_ddr_write_bandwidth:0”	Running cycles, backpressure cycles (from DDR memory) and starvation cycles (from AI Engine Array) when writing DDR memory
Detailed Write Bandwidth (Channel 1)	“detailed_ddr_write_bandwidth:1”	Running cycles, backpressure cycles (from DDR memory) and starvation cycles (from AI Engine Array) when writing DDR memory

Collecting Profiling Data on Hardware

After compiling the model with the appropriate configuration, run inference on the target hardware. This generates a timestamp file and a trace-dump file containing profiling data, both of which are required for AI Analyzer performance analysis.

Execution and Trace Generation

During inference, one trace-dump file per inference is generated, and the console also displays informational messages for each inference, such as:

[2026-05-29 09:17:01.684] [console] [info] [FLEXMLRT] ../../flexml_rt/src/FlexMLDispatcher.cpp:1015 hsiSlot OFM iter=0 i=0 b=0 batch_index=0 outSlot=0 layout=batch-major name=compute_graph.ofm_ddr cpu_shape=[1,1000] elements=1000 outNumElem[i]=1000

Trace Dump Files

After inference completes, a single timestamp file and trace-dump file are generated in the current working directory:

dtrace_dump_ctx_1_run_NNN.py  ...
record_timer_inference_MMM.json
record_timer_subgraph_cpu_ts.json
record_timer_ts.json
record_timer_vaip_vaiml_par_0.json

Aggregating Profiling Data

AI Analyzer processes profiling data from a single aggregated archive. Once inference completes, run the following command on the board to collect and package the data:

vaiprofile-collect-data .

This command moves all trace-dump and timestamp files into the vai_profile_dir directory and generates a vai-profile.zip archive from that directory. Transfer this file to your host computer for subsequent analysis with AI Analyzer.

To get a full view of data transfers during inference, it is necessary to run inference multiple times, once for each direction and each channel:

Runs to perform for full data transfer view

“interface-tile” setting	vai-profile.zip renamed to:
“detailed_ddr_read_bandwidth:0”	vai-profile_0.zip
“detailed_ddr_read_bandwidth:1”	vai-profile_1.zip
“detailed_ddr_write_bandwidth:0”	vai-profile_2.zip
“detailed_ddr_write_bandwidth:1”	vai-profile_3.zip

After transferring all vai_profile_N.zip files to the host machine, process them with the vaiprofile (see vaiprofile Utility) command within the Docker container:

vaiprofile --generate-profile-report <VAIML_design_dir_path> vai_profile_0.zip vai_profile_1.zip vai_profile_2.zip vai_profile_3.zip --frequency <freqMHz>

Where <freqMHz> is the AI Engine array clock frequency in MHz, for example, 1250 for a 1.25 GHz clock.

See Basic Profiling for more details on using the vaiprofile command with multiple profiling data archives.

Using Detailed DDR Profiling Bandwidth Columns

Launch AI Analyzer from <VAIML_design_dir_path>/analyze_multiple_runs/merged_results and visualize the data in the Performance tab:

• Summary: Provides a chart summarizing DDR bandwidth usage across the entire model with the running and stall (starvation/backpressure) cycles.

• Timeline: The table of this view shows DDR bandwidth usage over time for each layer and each column of the AI Engine Array (see Performance).

You get an overall view of the DDR bandwidth usage for the entire model and stall reason distribution:

Let’s detail the L3 Read Stall Summary:

You can see here that both runs for Read direction have been consolidated into these donut charts. You can see the distribution of running cycles from a column:channel or operator point of view. Same thing for the starvation cycles and the backpressure cycles. We can see in the stall charts that the operator Conv2D-Add is the one that experiences the most stall cycles by far. We can cross-probe between the Performance Summary chart, the Mapped Graph, and the Performance Timeline to get a better understanding of what is happening and where:

If model performance is below expectations, use the DDR bandwidth columns (Timeline table) to determine whether a memory bandwidth bottleneck is present. If the minimum observed throughput for a layer is significantly below 10 GB/s (for a 1.25 GHz AI Engine array clock), that layer might be limited by memory access.

Use the reported column and channel information to identify where the bottleneck occurs, then verify whether it recurs across multiple layers.

If the same column and channel consistently show low throughput, this may indicate inefficient memory-controller parameterization or issues with AI Engine GMIO parameters. In that case, adjust GMIO parameters using gmio_train Configuring NoC Connectivity for Model Deployments or tune memory-controller settings, then reassess throughput and overall model performance.

DDR Throughput Analysis Methodology

Purpose

Memory bandwidth analysis helps determine whether an AI model deployment is:

• Memory-bound

  • Limited by DDR/memory throughput

  • Detailed Profiling gives us access to DDR/L2 bandwidth metrics but not L2/L1 data transfers

• Compute-bound

  • Limited by processing capacity

  • Low Efficiency (%) in AI Analyzer with high DDR bandwidth suggests a compute-bound layer.

• Well-balanced

  • Optimal hardware utilization

Data is available in the Performance tab of AI Analyzer, including columns for minimum observed DDR read/write throughput (MB/s) per layer and the corresponding column and channel where it occurred. These data can be processed by the chatbox to provide recommendations.

Perform this analysis during performance optimization, especially when investigating latency issues or unexpected performance degradation.

Analysis can give you answers to the following questions:

• Are there memory bottlenecks limiting performance?

• Do different columns:channel experience shared or independent bottlenecks?

• Is bandwidth variance expected (heterogeneous workload) or problematic (instability)?

• How is workload distributed across memory channels?

• Is there room for throughput optimization?

Analyzing NoC-to-AI Engine Throughput Data

The timeline table captures the memory bandwidth measured between the Network-on-Chip (NoC) and the AI Engine array, recorded at every column and channel of the model. This data can be used to identify potential memory access bottlenecks that might impact model performance.

Throughput Thresholds

Thresholds can be set in advance; however, interpretation should be based on the specific model and performance goals. The following thresholds are relative to the maximum theoretical bandwidth of 10 GB/s for a 1.25 GHz AI Engine array clock:

• ~10,000 MB/s (optimal): Full bandwidth utilization — no memory bottleneck

• ≥ 2,000 MB/s (acceptable): Within acceptable range — monitor for degradation

• < 2,000 MB/s (problematic): Potential memory access issue — further investigation required

Pattern Detection and Variance Analysis

To identify memory access issues, analyze the throughput data across all columns and channels, looking for the following patterns:

• Consistently low throughput on a specific column — A column that repeatedly measures below the acceptable threshold across multiple inferences might indicate a structural bottleneck in the memory access pattern for that part of the model.

• High variance across columns — Significant differences in throughput between columns might suggest uneven workload distribution or suboptimal data layout.

• Gradual throughput degradation — A column that shows declining throughput over successive inferences might point to resource contention or buffer saturation.

Configuration Recommendation

Bandwidth Metric

To enable memory bandwidth profiling, compile the model with the required configuration, as described previously, so the necessary profiling data is generated. This includes setting the ai_analyzer_enhanced_profiling flag in vitisai_config.json to include either detailed_ddr_read_bandwidth or detailed_ddr_write_bandwidth on either channel, depending on the metric you want to capture. Read bandwidth is typically more relevant for bottleneck analysis because many models are read-bound due to input activations and weights being fetched from memory. Write bandwidth can also be analyzed when output-activation bottlenecks are suspected.

Sample Size Considerations

• Minimum: 10 inference runs

• Recommended: 20-30 inference runs

• Production characterization: 100-1000 runs to capture variance

Rationale: A sufficient sample size is required to distinguish among:

• Normal variance (different layer types)

• Anomalies (for example, memory initialization effects)

  • Discard the first 1-3 inferences to remove initialization overhead

• Systematic issues (consistent bottlenecks)

Analysis Methodology

Step 1: Average Analysis

Calculate average bandwidth per channel:column across all layers and inferences.

Maximum throughput is 10 GB/s for a 1.25 GHz AI Engine array clock. The following thresholds are relative to this maximum and can be adapted to your performance goals:

Status	Threshold	Interpretation
Critical	< 10% of max	Severe bottleneck or mostly idle
Warning	10-50% of max	Low utilization, investigate
Good	50-80% of max	Healthy, balanced workload
Excellent	> 80% of max	High utilization, memory-intensive

Step 2: Variance Analysis

High variance is not always problematic; interpretation depends on the underlying cause.

A useful indicator is the coefficient of variation (CV), computed as follows:

\[\text{Coefficient of Variation} = \frac{\sigma}{\mu}*100\%\]

Where \(\sigma\) is the standard deviation and \(\mu\) is the mean of throughput measurements for a specific channel:column across all layers and inferences.

CV Range	Variance Level	Interpretation
CV < 15%	Low variance	Consistent behavior
CV 15-30%	Moderate variance	Check patterns
CV > 30%	High variance	Requires investigation

A high CV might result from factors that are either expected or indicative of a problem:

Expected causes of high variance:

• Bimodal distribution (high BW for memory layers, low BW for compute layers)

• Layer-dependent patterns (different ResNet blocks, attention vs FFN)

• Correlated with layer type (conv vs pool vs dense)

Problematic causes of high variance:

• Random fluctuations (no pattern)

• Degrading trend over time (thermal/throttling)

• Correlated with failures or errors

Combining average and variance analysis helps determine whether low bandwidth is a consistent bottleneck (low variance, low average) or an intermittent issue (high variance) requiring further investigation of system stability or layer-specific behavior.

Step 3: Correlation Analysis

This step helps determine whether bottlenecks are shared (undesirable) or independent (typically desirable) across channels:columns. The standard approach is to compute the Pearson correlation coefficient (r) between bandwidth measurements of different channel:column pairs across all layers and inferences.

Correlation provides additional insight into relationships among bandwidth patterns of different channel:column pairs:

Correlation	Relationship	Interpretation	Assessment
r > +0.5	Strong positive	Channels move together	Undesirable if low BW
r < -0.5	Strong negative	Inverse relationship	Good load balancing
|r| < 0.3	Weak	Independent behavior	Good; no shared bottleneck

The following conclusions can be drawn from correlation analysis:

Start
 |
 |-- Are simultaneous_bottlenecks > 10% of layers?
 |     |
 |     |-- Yes --> Shared bottleneck pattern
 |     |            |
 |     |            |-- All channels suffer together
 |     |            |-- System-wide memory limitation
 |     |
 |     |-- No
 |          |
 |          |-- Are complementary_patterns > 20% of layers?
 |                |
 |                |-- Yes --> Load-balancing pattern
 |                |            |
 |                |            |-- Channels handle different work
 |                |            |-- Efficient distribution
 |                |
 |                |-- No  --> Neutral pattern - independent operation
 |                             |
 |                             |-- No shared bottlenecks
 |                             |-- No coordination issues

Best Practices Checklist

Before Analysis

• Enable hardware profiling with bandwidth metrics

• Configure timing/trace collection

• Run sufficient warmup iterations (3-5)

• Collect 10+ inference samples for statistical significance

• Document platform specifications (max bandwidth, memory type)

• Record configuration (batch size, parallelism, precision)

During Analysis

• Calculate average, min, max, std dev per channel

• Apply appropriate thresholds (% of max bandwidth)

• Compute coefficient of variation (CV)

• Identify variance patterns (bimodal, random, trending)

• Calculate correlation between channels

• Check for simultaneous bottlenecks

• Analyze workload distribution (imbalance ratio)

• Look for temporal trends (degradation/improvement)

Interpretation

• Distinguish compute-bound vs memory-bound

• Identify if variance is expected (heterogeneous layers) or problematic

• Determine if correlation is good (load balancing) or bad (shared bottleneck)

• Verify if workload imbalance is intentional (specialization)

• Check for system issues (thermal, leaks, interference)

LLM Example Prompts to Analyze Profiling Data

Before using the LLM chatbox, ensure you have collected profiling data with AI Analyzer. Run the profiler four times to capture data for all directions and channels, then launch aianalyzer from analyze_multiple_runs/merged_results so the chatbox can access the complete dataset.

For a first global analysis, you could try something like the following:

Prompt Example

“compute mean, min, max, standard deviation and coefficient of variation of measured bandwidth for each column:channel pair over all inferences”

“I see a minimum at 318 for 0:0. Is it due to a specific layer that is consistently giving low bandwidth”

To dig into correlations between operators sharing a channel, you can ask the AI Analyzer to analyze the per-operator bandwidth across inferences.

Prompt Example

“Correlate the per-operator bandwidth across inferences for operators that share a channel — tests whether the same op is repeatably fast/slow”

To get more insights into memory-bound and compute-bound layers, you can ask the AI Analyzer for optimization suggestions based on the profiling data.

Prompt Example

“Can you propose optimizations for layers that are memory-bounded or compute-bounded”