Project Background

As deep learning models continue to grow rapidly, deployment on resource-constrained edge devices is increasingly limited by compute capacity, memory footprint, and power consumption. Applying a single compression technique—such as pruning or quantization alone—often provides only limited gains before accuracy or hardware-efficiency constraints become dominant.

This project investigated and systematized joint compression paradigms that combine the complementary strengths of pruning, quantization, and knowledge distillation, with the objective of achieving a better trade-off between model compactness and predictive accuracy.

Technical Approach

• Joint Compression Strategy: Systematically analyzed the joint optimization paradigms of pruning, quantization, and knowledge distillation (KD).
  • Pruning: Streamlined model architecture to reduce the number of redundant parameters.
  • Quantization: Compressed the bit-width representations of model parameters and activations to lower storage and bandwidth overhead.
  • KD: Uses supervision from a teacher model to help a compact student recover accuracy lost through compression.

Personal Contributions

• Systematically summarized three mainstream joint compression paradigms: ‘Pruning + Quantization’, ‘Pruning + Distillation’, and ‘Quantization + Distillation’.

• To address the limited literature on jointly optimizing multiple compression techniques, I led the preparation of A Survey on Joint Optimization Methods for Neural Network Compression, published in CAAI Transactions on Intelligent Systems (CCF-C / Chinese core journal).

Reflections and Summary

Personal Evaluation: From Algorithmic Foundations to Architecture Research

This was the first research project that I independently led as an undergraduate. It gave me a rigorous foundation in neural networks and model-compression methods and, together with my background in integrated-circuit engineering, helped define my subsequent research focus on AI accelerators. The project concentrated on reproducing established pruning, quantization, and knowledge-distillation methods rather than proposing a new compression algorithm; its principal scholarly outcome was a structured survey of joint compression techniques. Nevertheless, the resulting theoretical foundation directly informed my later work on FPGA-based AI accelerators and model compression for custom processors.

The Evolution of Compression: From CNNs to Large Models

Pruning, quantization, and knowledge distillation remain important optimization techniques for model deployment and inference in the era of large models, while evolving into new forms of hardware–software co-design. Sparsification increasingly emphasizes regular, hardware-friendly computational structures; low-bit quantization has expanded from compressing models for edge devices to reducing the storage, bandwidth, and communication overhead of weights, activations, and caches; and knowledge distillation has evolved from aligning soft labels and intermediate features to leveraging stronger models to generate training data and reasoning traces. Although the specific techniques differ from those used in the CNN era, their fundamental objective remains the same: reducing the storage, computation, and deployment costs of models with minimal accuracy degradation.

Funding This project was funded by the Chinese Academy of Sciences Student Innovation Practice Training Program (Project No. 20224001168).