Wenyao Zhao

Wenyao Zhao

赵文尧

University of Chinese Academy of Sciences (UCAS), Institute of Semiconductors, CAS

Focus Areas

  • Edge AI inference chips
  • Algorithm-hardware co-design
  • AI compilers for custom processors
  • Near-sensor vision processors
Contact & profiles

Project Background

Transformers provide strong global context modeling for vision tasks, but edge deployment remains challenging:

  • Positional-encoding overhead: Because self-attention is permutation invariant, explicit positional information is often required, adding storage and nonlinear computation.
  • Compute and memory bottlenecks: Global interactions and wide feed-forward networks increase both operation count and parameter traffic, stressing the memory capacity, parallel throughput, and energy budget of edge devices.

Hybrid CNN–Transformer architectures combine the complementary strengths of the two model families. CNNs provide local receptive fields and translation-equivariant feature extraction, reducing reliance on explicit positional encoding, while Transformer blocks capture long-range spatial dependencies. Carefully combining these components can retain global modeling capacity with lower computational overhead.

This project evolved the first-generation vision processor used in my undergraduate thesis to support CNN–Transformer models efficiently while correcting limitations identified in the earlier architecture.

Technical Approach Overview

  • Model Design and Optimization: Deployed hybrid MobileViT and EfficientViT backbones with task-specific heads for classification, detection, and tracking, using quantization-aware training (QAT) for model compression.
  • Hardware Architecture Enhancements: Extended accumulator and effective-bias precision to INT32 to eliminate overflow and premature truncation in the first-generation chip. Redesigned the systolic array to support standard, pointwise, and depthwise convolution together with Strassen and transposed matrix multiplication. Added a reduction core that cooperates with the vector core to pipeline fixed-point LayerNorm and SoftMax.
  • Software Toolchain Refactoring: Implemented hardware-aware QAT with constrained quantization parameters for LayerNorm and SoftMax, and extended the compiler to model, fuse, and schedule the newly supported operators.

Personal Contributions

  • Developed hardware-constrained quantization schemes for SoftMax and LayerNorm, reducing numerical mismatch between software models and hardware execution.

  • Integrated MQBench-based QAT to address the accuracy degradation caused by post-training quantization in the legacy toolchain.

  • Refactored hardware modeling, graph parsing, operator fusion, and instruction generation to support end-to-end compilation and deployment.

Detailed Technical Implementation

💡 Tip: Click on the corresponding module to expand and read.

1. Algorithm Design and QAT Compression

MobileViT and EfficientViT were selected as representative deployment targets based on edge-side accuracy and computational requirements:

  • MobileViT combines inverted residual blocks with Transformer blocks and uses unfold/fold transformations between feature maps and token sequences, covering the principal operators found in hybrid architectures.
  • EfficientViT reduces expensive nonlinear operations and employs linear-complexity attention, making it a representative model for efficient edge inference.

The quantization workflow was rebuilt around MQBench, whose graph analysis and automatic fake-quantization insertion enabled QAT. This substantially reduced the accuracy loss observed with the legacy post-training quantization flow. Task-specific heads were attached to the backbones for end-to-end classification, detection, and tracking, as illustrated in Figure 1.


Figure 1. Implemented CNN-Transformer models and their inference examples under diverse tasks
2. Compilation and Deployment Toolchain

The legacy compiler and deployment flow was refactored to support CNN–Transformer graphs. Figure 2 shows the resulting architecture.

Figure 2
Figure 2. Next-generation compilation and deployment toolchain architecture

The toolchain accepts ONNX models and comprises three principal stages:

  • Hardware modeling and evaluation: Builds a behavioral model that generates the golden reference for RTL simulation.
  • Compilation and scheduling: Parses the computation graph and emits coordinated programs for the MPU, systolic-array core, vector core, and reduction core.
  • Verification and testing: Verifies the generated workload in RTL simulation before deploying it to the hardware platform for board-level evaluation.

Reflections and Summary

Personal Evaluation: From CNN to Transformer

Whereas the first-generation chip supported only CNNs, this revision extended the architecture and toolchain to CNN–Transformer workloads. The project spanned model selection, quantization, custom instructions, microarchitecture, and compiler development, providing end-to-end experience in hardware–software co-design for a domain-specific architecture.

Architectural Priorities: From Peak Compute to Memory and Communication

The project demonstrated diminishing returns from increasing arithmetic throughput in isolation: system bottlenecks increasingly arise from data movement, scheduling, and storage.

  • Memory wall: Unlike BatchNorm, LayerNorm is difficult to eliminate through operator fusion because its mean and variance depend on the current input. The resulting repeated and non-contiguous on-chip accesses can cost more than the arithmetic itself.
  • Communication bottleneck: The fixed-point exponential approximation used by SoftMax requires cooperation between the vector and reduction cores. Cross-core scheduling and synchronization therefore become material components of end-to-end latency.

These observations indicate that practical AI-chip performance is increasingly determined by the memory hierarchy and by energy-efficient orchestration of dataflow across heterogeneous units.