Multi-Instances of NPU Support

Multi-Instances of NPU Support#

This section covers how to instantiate multiple NPU IPs on a single design. The Default Reference Design supports the execution of multiple models from a single application context, and it has the flexibility to enable the NPU IP in the reference design with either a subset or all available AIE columns and it allowed resource savings for lighter workloads, but any unused AIE columns remained idle and could not be leveraged for parallel execution. The multi-instance design supports multiple models being executed on multiple NPU IPs. The available AIE resources can now be partitioned into multiple sub-groups (e.g., a few columns of AIE per NPU IP) and enables the creation of multiple NPU IPs within the same design.

The multi-instance NPU supports:

  • Concurrent execution of multiple models/snapshots on separate NPU IPs.

  • Better hardware utilization by distributing workloads across different AIE sub-groups.

  • Scalable performance, enabling developers to run diverse models simultaneously with dedicated compute resources.

It unlocks true parallelism and maximizes efficiency, making it easier to deploy multi-model workloads on the same platform.

The following two NPU IPs are used for the multi-instance NPU example.

  • VE2802_NPU_IP_O00_A128_M3 (Offset 0, AIE Columns 16)

  • VE2802_NPU_IP_O16_A080_M3 (Offset 16, AIE Columns 10)

Note

  1. There is a limitation on IP choices for multi-instance NPU support. Due to the limiting number of resources on both the AIE (GMIO) and PL (LUT/Flip-Flop/NMU) and DDR bandwidth, a high number of IP is not practical. Thus only two NPU IPs are used for the multi-instance NPU example implementation.

  2. The number of columns of each of the IP is also restricted by the GMIO availability on the AIE and the routing capability of Vitis. Splitting AIEs between multiple IPs and related physical silicon IO limitations should be reviewed carefully before attempting an implementation.

Integrate Multiple NPU IPs in Reference Design#

Refer to the following section to build the multi-NPU reference design.

Build Reference Design with Multiple NPU IPs#

This section covers the steps required to build the Reference Design with multiple NPU IPs.

  1. Clone the Vitis AI source code:

    $ git clone -b release/6.3 https://github.com/amd/Vitis-AI.git

  2. Change the directory to the location of source code:

    $ cd Vitis-AI/versal_ve

  3. Source with two NPU IPs:

    $ source npu_ip/settings.sh VE2802_NPU_IP_O00_A128_M3 VE2802_NPU_IP_O16_A080_M3

  4. Source the Vitis tool:

    $ source <path_to_Vitis>/2026.1/Vitis/settings64.sh

  5. Build the reference design with two NPU IPs:

    $ cd reference_design/vek280
$ bash setup_yocto_host.sh
$ export XILINXD_LICENSE_FILE=<path_to_NPU_IP_Vivado_licenses>
$ make all

During reference design build (SD Card compilation), ResNet50 and ResNet101 models are downloaded from TensorFlow hub and a snapshot is generated for the target NPU IP. Downloading models from TensorFlow hub may not work properly depending on tfhub availability. It is possible to skip the models download and snapshot generation using the option SKIP_SNAPSHOT=1 during the SD Card generation, as shown in the following command. make all SKIP_SNAPSHOT=1 BSP_PATH=<BSP file Path>.

The SD Card VE2802_NPU_IP_O00_A128_M3__O16_A080_M3_sd_card.wic.gz gets generated after running the preceding steps, and it contains 2 NPU IPs: VE2802_NPU_IP_O00_A128_M3 (128 AIEs so 16 columns of AIE and column offset 0) and VE2802_NPU_IP_O16_A080_M3 (80 AIEs so 10 AIE columns of AIE and column offset 16). You can refer to the Set Up/Flash SD Card section in Software Installations for steps to flash the SD Card.

Compile Multiple Models For Multiple NPU IPs#

This section guides you on compiling multiple models to run on multiple NPU IPs.

From the application, the execution of multiple models can be scheduled:

  • Using multiple threads (for instance, one thread per model), in that case the execute blocking call is recommended to be used.

  • Using single thread and issuing outstanding execute_async calls, and the associated wait calls.

In case the models target the same NPU_IP, the multiple inferences are sequential and one inference starts once the NPU HW is available. In case the models target different NPU_IP, the inference on the different NPU_IP runs at the same time, asynchronously.

Theoretical Example#

Let us assume we have 2 IPs (named IP_0 and IP_1) and we have 3 models (named model_A, model_B, model_C).

Refer to the following commands to compile the models for different NPU IPs.

source npu_ip/settings.sh IP_0                        # enable compilation for IP_0
VAISW_SNAPSHOT_DIRECTORY=model_A.$NPU_IP model_a.py   # will generate a snapshot for model_A and will dump it on model_A.IP_0
VAISW_SNAPSHOT_DIRECTORY=model_B.$NPU_IP model_b.py   # will generate a snapshot for model_B and will dump it on model_B.IP_0

source npu_ip/settings.sh IP_1                        # enable compilation for IP_1
VAISW_SNAPSHOT_DIRECTORY=model_A.$NPU_IP model_a.py   # will generate a snapshot for model_A and will dump it on model_A.IP_1
VAISW_SNAPSHOT_DIRECTORY=model_C.$NPU_IP model_c.py   # will generate a snapshot for model_C and will dump it on model_C.IP_1

Loading the snapshot of model_A.IP_0 will create a handle to run model_A on IP_0.

Therefore, the following is possible by a single application:

  • Running model A either on IP_0 or IP_1 (depending on which handle is used)

  • Running model B on IP_0

  • Running model C on IP_1

The target of the IP is included in the generated snapshot, so there is no need in the embedded SW stack to specify on which target IP a snapshot has to be executed.

Note

  1. A snapshot is necessarily bound to a defined IP, different IP uses different resources and AIE addresses, so it is not possible to generate a single snapshot targeting different IP.

  2. Therefore, it is required to generate multiple snapshots, one per IP, even if the same model is used on the different IPs.

Practical Example#

As VE2802_NPU_IP_O00_A128_M3 and VE2802_NPU_IP_O16_A080_M3 IPs are used in the multi-instance of NPUs example, refer to the following steps to compile the models for these IPs.

  1. Navigate to Vitis-AI source

    cd <path_to_Vitis-AI/versal_ve>

  2. Build snapshot for VE2802_NPU_IP_O00_A128_M3

    source npu_ip/settings.sh VE2802_NPU_IP_O00_A128_M3

./docker/run.bash --acceptLicense -- /bin/bash -c "source npu_ip/settings.sh && cd examples/python_examples/batcher && VAISW_SNAPSHOT_DIRECTORY=$VAISW_HOME/SNAP.$NPU_IP/resnet50.TF ./run_classification.sh -f tensorflow2 -n resnet50 --batchSize 4"

  3. Build snapshot for VE2802_NPU_IP_O16_A080_M3

    source npu_ip/settings.sh VE2802_NPU_IP_O16_A080_M3

./docker/run.bash --acceptLicense -- /bin/bash -c "source npu_ip/settings.sh && cd examples/python_examples/batcher && VAISW_SNAPSHOT_DIRECTORY=$VAISW_HOME/SNAP.$NPU_IP/resnet50.TF ./run_classification.sh -f tensorflow2 -n resnet50 --batchSize 4"

Now the snapshots for the ResNet50 model are generated for respective NPU IPs. Refer to the following section to execute the snapshots on multiple NPU IPs.

Execute Multiple Models with Multiple NPU IPs#

Use the following steps to execute the snapshots (of the ResNet50 model) on multiple NPU IPs (VE2802_NPU_IP_O00_A128_M3 and VE2802_NPU_IP_O16_A080_M3), by using the VART Runner application.

  1. Ensure that VE2802_NPU_IP_O00_A128_M3__O16_A080_M3_sd_card.wic is flashed on SD Card.

  2. Ensure that the VEK280 board is up and running

  3. Copy the generated snapshots (of the ResNet50 model) to the target board.

    scp <snapshot_generated_for_VE2802_NPU_IP_O00_A128_M3> amd-edf@<vek280_board_ip>:/home/amd-edf
scp <snapshot_generated_for_VE2802_NPU_IP_O16_A080_M3> amd-edf@<vek280_board_ip>:/home/amd-edf

  4. Copy the ‘imagenet’ folder, which is generated during executing sample model.

    scp -r imagenet/ amd-edf@<vek280_board_ip>:/home/amd-edf

  5. Set up the Vitis AI tools environment on the board

    $ sudo -i
$ source /etc/vai.sh

  6. Run the following command to execute the models on both IPs

    cd /home/amd-edf
# sample command
# vart_ml_demo --imgPath /home/amd-edf/imagenet/ILSVRC2012_img_val/ --snapshot <snapshot_generated_for_VE2802_NPU_IP_O00_A128_M3>+<snapshot_generated_for_VE2802_NPU_IP_O16_A080_M3> --labels /etc/vai/labels/labels --goldFile /home/amd-edf/imagenet/ILSVRC_2012_val_GroundTruth_10p.txt  --nbImages 1
# actual command
vart_ml_demo --imgPath /home/amd-edf/imagenet/ILSVRC2012_img_val/ --snapshot SNAP.VE2802_NPU_IP_O00_A128_M3/resnet50.TF/+SNAP.VE2802_NPU_IP_O16_A080_M3/resnet50.TF/ --labels /etc/vai/labels/labels --goldFile /home/amd-edf/imagenet/ILSVRC_2012_val_GroundTruth_10p.txt  --nbImages 1

    The previous command generates the following output on the console.

    root@amd-edf:/home/amd-edf# vart_ml_demo --imgPath /home/amd-edf/imagenet/ILSVRC2012_img_val/ --snapshot SNAP.VE2802_NPU_IP_O00_A128_M3/resnet50.TF/+SNAP.VE2802_NPU_IP_O16_A080_M3/resnet50.TF/ --labels /etc/vai/labels/labels --goldFile /home/amd-edf/imagenet/ILSVRC_2012_val_GroundTruth_10p.txt  --nbImages 1
XAIEFAL: INFO: Resource group Avail is created.
XAIEFAL: INFO: Resource group Static is created.
XAIEFAL: INFO: Resource group Generic is created.
[VART] Found snapshot for IP VE2802_NPU_IP_O00_A128_M3 IP on kernel 1.
[VART] Allocated config area in DDR:    Addr = [    0x880000000,  0x50000000000,  0x60000000000 ]       Size = [   0xe2a091,   0xab1501,   0xe2a151]
[VART] Allocated tmp area in DDR:       Addr = [    0x880e2c000,  0x50000ab3000,  0x60000e2c000 ]       Size = [    0x93c01,    0x93c01,    0x62801]
[VART] Parsing snapshot /run/media/mmcblk0p1/snapshot.VE2802_NPU_IP_O00_A128_M3.resnet50.TF//
[========================= 100% =========================]
NPU only mode set. Skipping node resnet50_embd_export_2_cpu_subgraph_call.
Warning: VART ML is already connected using XRT. Ignoring this call.
[VART] Found snapshot for IP VE2802_NPU_IP_O16_A080_M3 IP on kernel 0.
[VART] Allocated config area in DDR:    Addr = [    0x880f57000,  0x50000bde000,  0x60000ef4000 ]       Size = [   0xe2a091,   0xaaea01,   0xe2a151]
[VART] Allocated tmp area in DDR:       Addr = [    0x881d83000,  0x5000168e000,  0x60001d20000 ]       Size = [    0x62801,    0x62801,    0x31401]
[VART] Parsing snapshot /run/media/mmcblk0p1/snapshot.VE2802_NPU_IP_O16_A080_M3.resnet50.TF//
[========================= 100% =========================]
NPU only mode set. Skipping node resnet50_embd_export_2_cpu_subgraph_call.
Loading images: 100%
Loading images: 100%
[VART]  Running 2 models          219.08 imgs/s. (1 images)
resnet50 Image 0 (0:0) ILSVRC2012_val_00000001.JPEG
resnet50    GOLD - n03982430 pool table, billiard table, snooker table - 1.00000000
resnet50    PRED - n03982430 pool table, billiard table, snooker table - 0.99953282
resnet50    PRED - n03942813 ping-pong ball - 0.00015839
resnet50    PRED - n04336792 stretcher - 0.00007482
resnet50    PRED - n03376595 folding chair - 0.00005827
resnet50    PRED - n02797295 barrow, garden cart, lawn cart, wheelbarrow - 0.00003534
resnet50
resnet50 Image 0 (0:0) ILSVRC2012_val_00000001.JPEG
resnet50    GOLD - n03982430 pool table, billiard table, snooker table - 1.00000000
resnet50    PRED - n03982430 pool table, billiard table, snooker table - 0.99953282
resnet50    PRED - n03942813 ping-pong ball - 0.00015839
resnet50    PRED - n04336792 stretcher - 0.00007482
resnet50    PRED - n03376595 folding chair - 0.00005827
resnet50    PRED - n02797295 barrow, garden cart, lawn cart, wheelbarrow - 0.00003534
resnet50

============================================================
Accuracy Summary:
[AMD] [resnet50 TEST top1] 100.00% passed.
[AMD] [resnet50 TEST top5] 100.00% passed.
[AMD] [resnet50 ALL TESTS] 100.00% passed.
[AMD] [resnet50 TEST top1] 100.00% passed.
[AMD] [resnet50 TEST top5] 100.00% passed.
[AMD] [resnet50 ALL TESTS] 100.00% passed.
[AMD] Running 2 models 219.08 imgs/s (1 images)
root@amd-edf:/home/amd-edf#

You can also execute the snapshots on multiple NPUs by using X+ML application. Refer to the Multi-instance Support with x_plus_ml_app in VART X APIs Application Developer Guide for more details.

Contents