The target platform comprises an image sensor and a readout/processor die developed by our research group. The readout circuitry and processor are fabricated on the same die and integrated with the sensor through 3D stacking to form the complete vision SoC.

The processor uses a heterogeneous architecture with three compute engines:

• MPU: A microcontroller based on the MIPS instruction set, responsible for executing the global main function, scheduling data transfers, and controlling the activation/deactivation of the heterogeneous cores.
• Vector Processor (VP): A vector processor based on a custom instruction set, tasked with vector operations and data rearrangement.
• NN Processor (NNP): A custom-developed systolic array accelerator compatible with “Convolution-GEMM” operations ([Paper][pdf]), specifically designed to accelerate compute-intensive tasks within neural networks.

Model deployment must satisfy three primary hardware constraints:

• Operator support: The NNP supports only selected convolution configurations and pooling windows; it does not support activation functions other than ReLU or quantized matrix multiplication. Unsupported operators must therefore be avoided or reformulated.
• On-chip memory capacity: To prioritize energy-efficient on-chip execution, the chip has no external-memory interface. All parameters and intermediate feature maps must fit within the 4 MB global data memory (DM).
• Arithmetic precision: The processor implements an integer-only W8A8B16 datapath—INT8 weights and activations with INT16 biases. Its 24-bit accumulator is saturated and narrowed to INT16 before multiply–shift requantization, so the quantization model must reproduce these arithmetic semantics accurately.

2.1. SiamVC Network Architecture

Because the chip’s operator library supports only CNN-based models, SiamVC (Siamese Network for Vision Chip) adopts an efficient Siamese tracking architecture. As shown in Figure 1, the template and search branches share the backbone function $\phi$:

• Template Branch: Inputs a target image $z$ with a resolution of 128×128.
• Tracking Branch: Inputs a search region $x$ with a resolution of 256×256.

The two feature maps are combined by pointwise cross-correlation $f$ to produce the similarity map $s=f(\phi(z),\phi(x))$. This map is then processed by a foreground/background classification branch $g_{cls}$ and an ltrb bounding-box regression branch $g_{ltrb}$, producing $g_{cls}(s)$ and $g_{ltrb}(s)$.

SiamVC includes three hardware-oriented design choices:

• A streamlined MobileNetV2 backbone ensures that every operator is supported by the hardware.
• An anchor-free head based on depthwise separable convolution avoids complex post-processing such as non-maximum suppression (NMS), which maps poorly to the systolic array.
• Matrix multiplications in the cross-correlation stage are reformulated as NNP-supported 1×1 convolutions.

SiamVC occupies only 325 kB and requires 0.11 GOPS, satisfying the target platform’s deployment constraints.

2.2. Knowledge Distillation Based on Output Features

To recover accuracy lost through aggressive model compression and low-bit quantization, I introduced the output-feature distillation scheme shown in Figure 2. A larger teacher, SiamVC-T, supervises the student SiamVC-S through the mean-squared error (MSE) between their output features.

The total loss function is formulated as a weighted sum of the hard loss (Cross-Entropy and IoU losses) and the soft loss:

• Total Loss: $L_{total}=\alpha L_{hard}+(1-\alpha)L_{soft}$
• Hard Loss: $L_{hard}=L_{CE}+L_{IOU}$
  • Foreground-Background Classification Cross-Entropy: $L_{CE}(p,q)=-\sum p(x)\log (q(x))$
  • Coordinate Regression IoU Loss: $L_{IOU}(A,B)=1-IOU=1-\frac{A\cap B}{A\cup B}$
• Soft Loss: $L_{soft}=L_{MSE_{cls}}+L_{MSE_{loc}}$
  • Feature-Level MSE Loss: $L_{MSE}(G,Y)=\frac{1}{N}\sum\limits_{i=1}^{N}(G_{i}-Y_{i})^{2}$

Evaluation on OTB100 and VOT2019 (Figures 3 and 4) showed that:

• Distillation improved SiamVC-S by 2.1 percentage points in Success and 3.5 percentage points in Precision on OTB100, reaching 61.7% and 83.0%, respectively.
• On VOT2019, the model surpassed lightweight trackers such as LightTrack in accuracy while using fewer parameters and less computation.

2.3. Tracking Results Visualization

The actual tracking performance of various model versions is demonstrated below. Legend: Cyan represents the teacher network SiamVC-T, Orange represents the student network SiamVC-S, Red represents the distilled version SiamVC-S(KD), Pink represents the quantized + distilled version SiamVC-S(PTQ-KD), and Green denotes the Ground Truth bounding boxes.

The tracker remains robust under challenging conditions including camera shake (1), distractor objects (2), severe occlusion (3), scale variation (4), rotation (5, 6), low resolution (7), and deformation (8).

Deployment uses the custom vision-chip toolchain shown in Figure 5:

• Model quantization: Parses ONNX models, performs post-training quantization (PTQ), and emits operator descriptions and parameters in the chip-specific format.
• Algorithm-level verification: Models the integer-only datapath in MATLAB to estimate hardware accuracy and provide a golden reference for VCS simulation.
• Compiler front end: Performs graph transformations such as operator fusion, aggregates node metadata, and allocates memory.
• Compiler back end: Generates the C entry point for MPU scheduling and the configuration-register stream for the NNP array.
• (Optional) VP Assembly: For unsupported non-linear operators or specific data rearrangement logic, handwritten VP assembly programs are required.
• RTL Simulation Verification: Cross-compiles MCU/VP/NNP programs into binaries for VCS simulation, comparing the computational outcomes.
• Board-Level Testing: Following successful RTL simulation, the programs are flashed onto the vision chip test board for physical evaluation.