Project Background

Many AI accelerators are integrated as loosely coupled heterogeneous cores or memory-mapped peripherals. The host processor and accelerator then rely on a system bus or DMA engine for explicit data movement and task dispatch, introducing driver, synchronization, and interrupt overhead. Repeated movement through the memory hierarchy also increases latency and energy, which is particularly problematic in ultra-low-power edge systems.

The open and extensible RISC-V instruction-set architecture enables a tightly coupled alternative. Custom instructions allow a parallel compute array to be attached directly to the processor’s execution path, reducing host–accelerator control overhead and presenting the accelerator through the conventional instruction-stream programming model.

This independently developed research prototype explored a full-stack hardware–software architecture for energy-efficient AI MCUs.

Technical Approach Overview

• Hardware Layer: Defined custom RISC-V matrix-multiplication instructions and integrated a weight-stationary (WS) systolic-array accelerator through the NICE interface.
• Model Layer: Designed a compact MLP for MNIST under the available memory budget and applied QAT for INT8 deployment.
• Software Layer: Built an inline-assembly operator library and C-level runtime for tiled matrix operations and end-to-end inference.

Personal Contributions

• Defined five RISC-V matrix-multiplication instructions in the custom opcode space for low-latency control of the systolic array.

• Deployed a compact MLP with a 55.84 kB parameter footprint and 0.109 MOP workload, achieving 97.26% MNIST accuracy after QAT.

• Completed end-to-end FPGA validation; relative to scalar C execution, the custom-instruction implementation reduced the dynamic instruction count by approximately 24,000× and achieved a 423× cycle-count speedup.

Detailed Technical Implementation

💡 Tip: Click on the corresponding module to expand and read.

1. RISC-V Custom Extension Instruction Set

The systolic-array accelerator in Figure 2 is connected to the open-source Hummingbird E203 core (Figure 1) through the NICE (Nuclei Instruction Co-unit Extension) coprocessor interface. NICE supports the RISC-V custom0–3 opcode spaces, providing the encoding and handshake mechanisms required for tight integration.

Figure 1. Microarchitectural block diagram of the E203 MCU system

Five custom instructions were designed around the weight-stationary (WS) dataflow, covering configuration, data loading, multiply–accumulate execution, requantization, and result write-back:

Instruction (Inst)	Register 1 (rs1)	Register 2 (rs2)	Functional Description (Function)
sparam	{ReLU_en(1bit), sft(5bit), Mul(16bit)}	-	Loads dequantization multiply-shift factors and ReLU activation configurations into the output processing unit.
lybuf	Bias_addr	-	Loads biases into the accumulation registers of the PE array.
lwbuf	Weight_addr	Column count of weight matrix	Loads weights into the internal registers of the PE array and fixes them stationary.
matmul	Input_addr	Column count of input matrix	Triggers diagonalized input matrix loading and systolic MAC operations, updating partial sums.
sybuf	Output_addr	Column count of output matrix	Triggers dequantization calculations of the partial sums and writes the output features back to main memory.

2. Hardware Microarchitecture Design of the Systolic Array

The accelerator uses the WS systolic-array architecture shown in Figure 2. Its principal components are the input, weight, and output buffers; a parameterizable processing-element array; and requantization and ReLU post-processing logic.

Figure 2. Coprocessor architecture of the systolic array based on WS dataflow

2.1. Dataflow Control and Diagonal Matrix Generation

To align operands with the wavefront execution of the array, I designed hardware that converts buffered matrix rows into a skewed input stream. Periodic shifts generate the diagonal timing pattern shown in Figures 3 and 4, ensuring that corresponding operands meet at each PE in the correct cycle.

Figure 3. Example of a 16×16 input matrix

Figure 4. Timing diagram of periodic data shifting and diagonal output

2.2. Bus-Level Data Packing Optimization

To utilize the E203’s 32-bit data bus efficiently, four INT8 activations or weights are packed into each transfer (Figure 5). This increases useful payload per transaction and reduces bus-protocol overhead.

Figure 5. Memory access data concatenation and packing mechanism

3. Lightweight MLP Design and Quantization Compression

To fit within the E203’s expanded 128 kB data tightly coupled memory (DTCM), I designed the five-layer multilayer perceptron (MLP) shown in Figure 6 for MNIST classification.

Figure 6. Topology of the lightweight MLP network

The model occupies 55.84 kB and requires 0.109 MOPs per inference. Accuracy decreased only from 97.30% for the floating-point model to 97.26% after QAT, satisfying the deployment target.

4. Inference Driver and Compute Scheduling Implementation

The software stack is organized into three layers: a low-level inline-assembly interface to the custom instructions, tiled matrix operators that map large workloads to the physical array, and a network-level runtime that sequences the complete inference graph.

• Inline Assembly Encapsulation: Instruction interfaces were constructed using C-language inline assembly within NucleiStudio (Figure 7).
• Tile-Based Block Scheduling: When operator dimensions exceed the physical scale of the systolic array hardware (e.g., 32×32), the software automatically performs tile partitioning and accumulation scheduling (Figure 8).
• Network Inference Evaluation: The main function schedules layer-by-layer inference in topological order and integrates a hardware timer for precise profiling of performance and instruction counts (Figures 9–11).

Figure 7. Example of an inline assembly function

Figure 8. Tile-level scheduling logic for fully-connected layers

Figure 9. End-to-end inference and performance profiling workflow

Input and weight matrices are stored in the DTCM as two-dimensional arrays, with examples illustrated in Figure 10 and Figure 11, respectively.

Figure 10. Example of an input array

Figure 11. Example of a weight array

FPGA Verification and Performance Evaluation

The tightly coupled system was implemented on a Xilinx Artix-7 A7-100T FPGA with a 32×32 PE array running at 33 MHz (Figure 12). The test workload contained 32 MNIST images at 28×28 resolution (Figure 13), and inference results and performance counters were reported over UART (Figure 14).

1. Comprehensive Performance Metrics

At a 32×32 array scale, a benchmark comparison was conducted between the “custom instruction set-based coprocessor acceleration scheme” and a “pure scalar software execution scheme (C-language)”:

2. FPGA Board-Level Physical Test Results

Figure 12. Artix-7 A7-100t FPGA evaluation and verification platform

Figure 13. Test-set image array data

Figure 14. Serial output demonstrating inference results and clock cycle metrics

Reflections and Summary

Personal Evaluation: From Loose Coupling to Instruction-Level Tight Coupling

This project was my first implementation of a tightly coupled AI accelerator. Compared with a loosely coupled heterogeneous design, exposing the accelerator through RISC-V instructions provides a unified programming model: computation and control can be expressed in bare-metal C using a conventional compiler toolchain, with far less driver and software-abstraction overhead. The current prototype accelerates matrix multiplication only, however, and provides no native support for nonlinear operators such as SoftMax or LayerNorm.

Bottleneck Analysis: Bandwidth Constraints and Blocking Scheduling of the Coprocessor Interface

Performance analysis identified the bandwidth and blocking semantics of the Nuclei NICE interface as the principal limits on sustained array throughput.

A systolic array is most efficient when its pipeline remains filled with a continuous stream of multiply–accumulate operations. NICE executes these custom operations synchronously and blocks the in-order core, so software cannot issue the next tile’s load while the current tile is being computed. This explicit load–compute serialization periodically drains the array pipeline.

Proposed Solutions:

• Decoupled accelerator datapath: Replace NICE with a dedicated, higher-bandwidth accelerator path after the CPU EX stage. This provides greater scheduling freedom at the cost of substantially more invasive RTL changes.
• Queued command interface: Retain the coprocessor connection but add a command FIFO and hazard-management logic. Buffered load and compute commands could then overlap memory transfers with array execution and hide a significant fraction of the data-loading latency.

Overall, the project demonstrates an end-to-end AI MCU prototype and provides a basis for future work on pipelined nonlinear operators, decoupled execution, and out-of-order accelerator scheduling.