Project Background

This project developed an AI compilation and deployment toolchain for an upgraded SIMD vision chip supporting both 2D and 3D imaging (see the [chip paper], [chip paper PDF], [packaging paper], and [packaging paper PDF]). Before this toolchain was available, each network had to be implemented manually in the processor’s custom assembly language—an inefficient and error-prone workflow.

Building on the earlier toolchain for a custom systolic-array processor, I refactored both front-end scheduling and back-end code generation for the SIMD architecture:

• Front-End Abstraction and Memory Scheduling:
  • Operator decoupling: Separated Pooling and Resize from the Conv–BN–ReLU fusion pattern used by the earlier systolic-array compiler and implemented them as independent vector-core operators.
  • Conflict-free address mapping: Added static allocation of activations, parameters, and outputs under the VPDM constraint that simultaneous reads and writes must target different memory groups.
• Back-End Generation and Operator Library:
  • Tensor layout transformation: Reordered weights and biases at compile time to match the processor’s row-interleaved physical layout.
  • Assembly operator library: Implemented optimized templates for the supported reconfigurable PE-group (PEG) modes.
  • Automated generation: Bound parsed graph nodes and allocated addresses to the appropriate templates, emitting binary parameter images and assembly programs for complete-network inference.

Technical Approach Overview

• Designed instruction-level mappings of convolution and matrix multiplication for the SIMD vector-core microarchitecture.
• Developed an operator-template-driven compiler back end covering graph parsing, address allocation, tensor layout transformation, and assembly generation.

Personal Contributions

• Mapped core deep-learning operators to the SIMD instruction set, accounting for the multilevel register hierarchy and scalar–vector dual issue.

• Refactored the compiler infrastructure and built an assembly-template-based code generator that eliminated manual per-model assembly programming.

• Compiled and deployed a lightweight CenterNet detector (788.45 kB parameter footprint; 350.63 MOPs), achieving 6.36 ms measured end-to-end on-chip latency.

• Compiled and deployed the SiamVC tracker (437.06 kB parameter footprint; 234.98 MOPs), with 0.64 ms template initialization and 3.13 ms search-branch tracking latency.

• Compiled and deployed an EfficientViT classifier (1.405 MB parameter footprint; 171.824 MOPs), achieving 4.73 ms measured end-to-end on-chip latency.

Detailed Technical Implementation

💡 Tip: Click on the corresponding module to expand and read.

1. Hardware Platform: Custom SIMD Vision Chip Microarchitecture

Relative to the first-generation design, the new SIMD chip incorporates several process and architectural enhancements:

• Higher compute density: The 28 nm implementation operates at up to 600 MHz, while custom instructions double the density of AI MAC units in the vector core, providing 2.2 TOPS peak throughput.
• Ring-connected PE array: The first and last processing elements are physically connected, enabling data rotation across the array boundary.
• Multi-chip scalability: High-speed SERDES interfaces support low-latency chip-to-chip communication and distributed execution of larger models.

1.1. Vector Core Architecture

The vector core contains scalar and vector execution units. The vector unit comprises 256 PEs; each PE can access the local registers of its two neighboring PEs. The array supports six PE-group (PEG) configurations: PEG256×1, PEG128×2, PEG64×4, PEG32×8, PEG16×16, and PEG8×32. Figure 1 shows the pipeline, and Figures 2 and 3 illustrate representative PEG modes.

Figure 1. Schematic diagram of the vector core's 7-stage pipeline (IF-IL-ID-EX1-EX2-MA-WB)

Figure 2. PEG256 wide vector mode

Figure 3. PEG8×32 fine-grained parallel mode

The scalar and vector units share the first two pipeline stages. During instruction line-up (IL), instructions are dispatched by type through a dual-issue mechanism, allowing scalar and vector instructions to execute in parallel.

1.2. Memory Hierarchy and Access Interfaces

The vector core combines the Vision Processor Data Memory (VPDM) with several classes of register file and a variable-granularity load/store interface.

Register-file classes:

• Scalar register file (SRF): 32 32-bit registers in the scalar unit, used for branch conditions, address pointers, and global parameters.
• Global register file (GRF): 16 32-bit registers per PEG, typically holding weights shared across the PEs in that group.
• Local register file (LRF): 16 16-bit registers per PE, typically holding activation data.
• AI register file (ARF): Eight 32-bit accumulators per PE for partial sums generated by aimac instructions.

As shown in Figure 4, the 4 MB VPDM comprises two SRAM groups. Each group contains 16 banks, with each bank organized as 8192×128 bits. The port-selection logic permits concurrent reads and writes only when they address different groups, so the compiler must allocate operands to avoid same-group access conflicts.

Figure 4. Schematic diagram of the VPDM architecture

Together with PEG reconfiguration, the variable-granularity interface supports broadcast, unicast, and multicast transfers at several widths, as illustrated in Figures 5 and 6.

Figure 5. Load: Multi-granularity data loading from VPDM to registers

Figure 6. Store: Write-back mechanisms from registers to VPDM

2. Instruction-Level Mapping of Core Operators (Conv / MatMul)

2.1. Convolution Mapping

Convolution is spatially mapped onto the SIMD array: feature-map columns correspond to PE indices (for example, column 3 maps to PE3), rows correspond to LRF indices within each PE, and weights are broadcast through the PEG’s GRF. Figure 7 shows the resulting loop hierarchy.

Figure 7. Convolution loop hierarchy

The mapping has three principal characteristics:

• Array-aligned tiling: Output channels and rows are tiled to match the PE-array dimensions and the number of channels processed by each PE.
• Spatial loop unrolling: Output columns map across PEs and output rows across LRF entries, exposing data-level parallelism in the inner loop.
• Output-stationary dataflow: Partial sums remain in the accumulator registers until all contributing input channels have been processed, after which the completed outputs are written back.

The inner loop uses the custom aimac42ex instruction and neighboring-PE register accesses to compute four output rows for two output channels in parallel (Figure 8).

Figure 8. Convolution layer inner-loop mapping

One 3×3 convolution step is expressed as nine consecutive aimac42ex instructions:

aimac42ex lr1, lr3, gr1(lhw), d(l1) //Step=1
aimac42ex lr1, lr3, gr1(hhw)        //Step=2
aimac42ex lr1, lr3, gr2(lhw), d(h1) //Step=3
aimac42ex lr2, lr4, gr2(hhw), d(l1) //Step=4
aimac42ex lr2, lr4, gr3(lhw)        //Step=5
aimac42ex lr2, lr4, gr3(hhw), d(h1) //Step=6
aimac42ex lr3, lr5, gr4(lhw), d(l1) //Step=7
aimac42ex lr3, lr5, gr4(hhw)        //Step=8
aimac42ex lr3, lr5, gr5(lhw), d(h1) //Step=9

The surrounding loops iterate over input channels (loop 1), output rows (loop 2), and output channels (loop 3), repeatedly invoking the inner kernel until the complete output feature map has been generated.

Utilization optimization: For narrow feature maps, the vector unit can be divided into multiple PEGs that process channels in parallel. A 64×64 input, for example, can use the PEG64×4 configuration shown in Figure 9.

Row-interleaved layout: Multi-PEG execution stores a logical feature map $(W,H,C)$ physically as $(W,#PEG,H,C/#PEG)$. The compiler applies the corresponding parameter transformations: weights from $(K,K,C_{in},C_{out})$ to $(2,#PEG,K,K,C_{in},C_{out}/T_{co})$, and biases from $(C_{out})$ to $(2,#PEG,C_{out}/T_{co})$.

Figure 9. Inter-PEG channel parallelism and row-interleaved data layout

2.2. Matrix Multiplication Mapping

Matrix multiplication is implemented as a sequence of vector–scalar outer-product accumulations using aimulg and aiadd2, as shown in Figure 10.

Figure 10. Outer-product matrix multiplication through vector–scalar accumulation

Scalars from matrix A are placed in the PEG’s GRF, while vectors from matrix B are distributed across PE-local LRFs. Figure 11 shows the corresponding loop structure.

Figure 11. Matrix-multiplication loop hierarchy

The inner loop performs multiply–accumulate operations with aimulg and aiadd2, as illustrated in Figure 12.

Figure 12. Matrix-multiplication inner-loop mapping

The example in Figure 10 requires 12 consecutive multiply–accumulate steps:

aimulg  air1,lr1,gr1(lhw);        // Step=1, 
aiadd2; aimulg  air1,lr2,gr1(hhw) // Step=2,  
aiadd2; aimulg  air1,lr3,gr2(lhw) // Step=3,  
aiadd2; aimulg  air1,lr4,gr2(hhw) // Step=4, Row 1 completed
aimulg  air1,lr1,gr3(lhw);        // Step=5, 
aiadd2; aimulg  air1,lr2,gr3(hhw) // Step=6,  
aiadd2; aimulg  air1,lr3,gr4(lhw) // Step=7,  
aiadd2; aimulg  air1,lr4,gr4(hhw) // Step=8, Row 2 completed
aimulg  air1,lr1,gr5(lhw);        // Step=9, 
aiadd2; aimulg  air1,lr2,gr5(hhw); // Step=10, 
aiadd2; aimulg  air1,lr3,gr6(lhw); // Step=11, 
aiadd2; aimulg  air1,lr4,gr6(hhw); // Step=12, Row 3 completed

The outer loops iterate over output rows (loop 1) and reduction-dimension rows (loop 2), repeatedly invoking the vector–scalar inner kernel to construct the complete output matrix.

Utilization optimization: When matrix B has relatively few columns, multiple PEGs can process different output rows concurrently. For $A(32,128)\times B(128,64)=C(32,64)$, the vector unit can use PEG64×4.

Compilation, Deployment, and Correctness Verification

After implementing the compiler front end (operator fusion and address allocation) and back end (tensor transformation, binary generation, assembly templates, and code generation), I validated the flow on object detection, single-object tracking, and image classification models.

1. Summary of Comprehensive Performance Metrics

2. Compilation, Deployment, and Simulation Verification of the CenterNet Object Detection Model

A lightweight variant of the anchor-free CenterNet detector was developed (Figure 13). With a parameter footprint of 788.45 kB and a workload of 350.63 MOPs, the QAT model achieved 97.75% ship-detection mAP on HRSC2016.

Figure 13. CenterNet object-detection architecture

Operator fusion reduced the graph to 108 layers, and the compiler emitted 8,548 lines of assembly. The allocation map contained no overlapping address ranges (Figure 14), and VCS RTL simulation matched the MATLAB reference.

Figure 14. Spatiotemporal address-allocation map for CenterNet

3. Compilation, Deployment, and Simulation Verification of the SiamVC Target Tracking Model

Building on the SiamVC tracker developed in my undergraduate thesis, I produced the lightweight network shown in Figure 15. The model occupies 437.06 kB, requires 234.98 MOPs, and achieves 58.5% Success and 76.7% Precision on OTB100 after QAT.

Figure 15. SiamVC single-object tracking architecture

Graph optimization and operator fusion reduced the model to 130 layers, producing 10,218 lines of assembly. The allocation map contained no overlaps (Figure 16), and VCS RTL simulation matched the MATLAB reference.

Figure 16. Spatiotemporal address-allocation map for SiamVC

4. Compilation, Deployment, and Simulation Verification of the EfficientViT Image Classification Model

A lightweight classifier was developed from EfficientViT-B0 (Figure 17). With a 1.405 MB parameter footprint and 171.824 MOP workload, the QAT model achieved 80.68% Top-1 and 94.28% Top-5 accuracy on ImageNet100.

Figure 17. EfficientViT image classification model architecture

Operator fusion reduced the model to 184 layers, and compilation generated 14,410 lines of assembly. The allocation map contained no overlapping address ranges (Figure 18), and VCS RTL simulation matched the MATLAB reference.

Figure 18. Spatiotemporal address allocation map for the compiled classification model

Reflections and Summary

Personal Evaluation: From Coarse-Grained Scheduling to Fine-Grained Programming

This toolchain highlighted the programming-model difference between systolic arrays and SIMD processors. The first-generation NNA used in my undergraduate thesis and the later CNN–Transformer NNA were controlled through coarse-grained configuration words. By contrast, the SIMD vector core requires explicit instruction-level scheduling of each computational step. This fine-grained model supports a broader range of nonlinear operators and conventional vision algorithms, but it does not inherit the systolic array’s implicit data reuse. Sustained utilization therefore depends strongly on compiler scheduling, register allocation, and pipeline-aware code generation.

Bottleneck Analysis: The Memory Access Predicament of Single-Threaded SIMD

Board-level measurements exposed a clear bottleneck. Although the SIMD processor provides 2.2 TOPS peak throughput compared with 1.4 TOPS for the systolic-array design, CenterNet ran at fewer than 160 FPS on the SIMD processor versus 375 FPS on the systolic array.

Fine-grained SIMD execution made the workload memory-bound. Because the core provides no hardware context switching, instruction sequences frequently stall on Load/Store operations, lowering effective arithmetic intensity and leaving execution units idle.

Proposed Solutions:

• Software: dual-issue scheduling and register double buffering: Reorder assembly to exploit scalar–vector dual issue and preload the next iteration’s data into an alternate register set while the current iteration computes, thereby hiding part of the memory-access latency.
• Hardware: transition from single-threaded SIMD toward SIMT: A future design could add hardware multithreading so that the scheduler switches to a ready thread when the active thread stalls on memory, improving overlap between data movement and computation.