Project Background

In foggy or hazy environments, suspended particles attenuate scene radiance while atmospheric light is scattered toward the camera, reducing image contrast and visibility. This degradation can substantially impair downstream processing in applications such as urban surveillance and remote sensing. Deep neural networks have achieved strong image-dehazing performance through end-to-end feature learning, but their memory and computational requirements often preclude deployment on edge devices with stringent power and cost constraints.

This project adopted a hardware–software co-design methodology to build an energy-efficient real-time video dehazing system on the low-cost PYNQ-Z2 (Zynq-7020) platform. The network was simplified and quantized at the algorithm level, while a custom inter-layer-pipelined accelerator exploited FPGA spatial parallelism to meet the required throughput.

Technical Approach Overview

• Simplified the AOD-Net topology and applied quantization-aware training (QAT) to meet FPGA memory and compute constraints.
• Developed a streaming FPGA accelerator with inter-layer pipeline parallelism.
• Designed an on-chip line-buffering scheme to maximize feature-map reuse.
• Built an end-to-end PYNQ video pipeline: “USB Camera -> AOD-Net Accelerator -> HDMI Display”.

Personal Contributions

• Simplified AOD-Net and implemented TensorFlow-based QAT, reducing the parameter footprint to 1.61 kB while retaining strong reconstruction quality (SSIM: 0.872; PSNR: 22.79 dB).

• Designed the accelerator IP architecture and contributed to the RTL implementation of its core logic, including the processing-element array and line-buffer modules.

• Independently integrated the complete PYNQ dehazing system, including the Vivado block design and host-side software drivers.

Detailed Technical Implementation

💡 Tip: Click on the respective modules to expand and read.

1. Model Compression: Simplification and Quantization of AOD-Net

The target XC7Z020 device provides limited on-chip BRAM. The original AOD-Net contains several branches whose long-lived intermediate feature maps would incur unacceptable on-chip or off-chip storage overhead. I therefore applied two model-level optimizations:

• Architecture simplification: Removed the Concat paths with the longest feature lifetimes and largest memory footprint.
• INT8 QAT: Applied quantization-aware training to convert the floating-point model to INT8 inference.

The resulting compact architecture is shown in Figure 1. Its parameters occupy only 1.61 kB, while evaluation on RESIDE yields an SSIM of 0.872 and a PSNR of 22.79 dB.

Figure 1. Lightweight AOD-Net Architecture

2. Hardware Microarchitecture Design: Inter-Layer Pipelined Parallel Accelerator IP

To maximize throughput, the accelerator adopts the inter-layer pipeline shown in Figure 2. Each convolutional layer is implemented as a dedicated compute stage; once an upstream stage produces valid data, the downstream stage can begin processing immediately. This streaming schedule minimizes idle time between layers.

Figure 2. Overall Architecture of the Hardware Accelerator IP

The compute array combines an input-stationary (IS) dataflow with loop unrolling. Input activations remain in the input buffer while multiply–accumulate units iterate over the weight register file; an adder tree reduces the resulting partial sums before they are written to the output buffer (Figure 3). The PE datapath then applies multiplication, shifting, and ReLU to the accumulated INT32 values before requantizing them to INT8 (Figure 4).

A line-buffering scheme further increases spatial reuse (Figure 5). When the convolution window advances vertically, only the newly exposed image row is loaded; hardware interleaving preserves the required window layout and avoids reloading previously buffered pixels.

Figure 3. Input Stationary Dataflow Scheduling Demonstration

Figure 4. Processing Element (PE) Microarchitecture

Figure 5. Data Reuse Mechanism Based on Line Buffer

3. System-Level Integration: PYNQ-Based End-to-End Video Stream

Using PYNQ to expose programmable-logic (PL) resources to Linux user space, I implemented a complete acquisition–inference–display pipeline (Figure 6) and integrated it in the Vivado block design shown in Figure 7:

• The custom AOD-Net engine was packaged as an AXI4-Stream IP, with DMA managing data transfers.
• The image reconstruction stage derived from the atmospheric scattering model (ASM) was also implemented as a DMA-connected hardware IP, offloading post-processing from the CPU.

Figure 6. End-to-End Video Processing System Architecture

Figure 7. Vivado Block Design Wiring Diagram

Deployment and Comprehensive Performance Evaluation

At a 113 MHz operating frequency and an input resolution of 640×480, the finalized AOD-Net hardware reported in the paper achieved a pure-network latency of 26 ms, corresponding to 38.3 FPS. This metric excludes image acquisition, preprocessing, post-processing data preparation, dequantization, and HDMI output on the PYNQ software side.

1. Comprehensive Performance Metrics

2. Dual-System Hardware Demonstration

To demonstrate the dehazing effect directly, I constructed the dual-board comparison setup shown in Figure 8 using two PYNQ-Z2 boards, two displays, a USB camera, an HDMI splitter, and a host PC. The first board captures the camera stream and produces HDMI output. The splitter sends one copy directly to the left display and a second copy to the other PYNQ board for dehazing; the processed stream is displayed on the right.

Figure 8. Dual-Screen Comparison Demonstration System for Dehazing Performance

Figures 9 and 10 show the measured system output. The edge platform processes dynamic hazy scenes smoothly and improves both contrast and visual clarity.

Reflections and Summary

Personal Evaluation: From Theory to Practice

This project was my first complete hardware–software co-design effort, spanning model compression, custom datapath design, RTL implementation, and system integration. Combining a compact network with an inter-layer hardware pipeline enabled deployment on a severely resource-constrained Zynq device.

It also exposed the limitations of fully specialized hardware. Because the datapath and compute array were tailored to one network, a substantial model revision could require a complete IP redesign. This lack of programmability motivated my later work on domain-specific architectures (DSAs) and programmable AI processors.

AI Systems: The Constraints of Amdahl’s Law

As summarized in my National IC Innovation Competition Project, the overall performance of an AI inference system is bound by Amdahl’s Law, and this project was no exception.

A separate single-frame call-chain profile was retained from an earlier PYNQ/Jupyter development stage. The program recorded timestamps around image acquisition, preprocessing, hardware calls, post-processing, and output to identify the distribution of latency in the software prototype, as summarized in Table 1.

Table 1. Breakdown of End-to-End Inference Latency

The program recorded approximately 1.38 ms for the run_kernel_0() call interval and 8.76 ms for the run_kernel_1() call interval. The complete single-frame path—including camera capture, Python preprocessing, hardware calls, post-processing data preparation, dequantization, and HDMI output—took approximately 405.95 ms. Data preprocessing and post-processing data preparation, at 176.43 ms and 155.09 ms respectively, dominated the latency of this software prototype.

Bottleneck Analysis and Solutions: The dominant overhead arose in the Python-based PYNQ software path. Image preprocessing, format conversion, and data assembly through PIL and OpenCV executed serially on the ARM core, masking the performance of the FPGA kernels. PYNQ greatly simplified application development, but this software convenience came at a substantial end-to-end latency cost.

To thoroughly break this deadlock, the system architecture must be refactored:

1. Move the application path to native code: Reimplement preprocessing in C++ and OpenCV, using multithreading and ARM NEON where appropriate.
2. Construct a PL-only video pipeline: Offload cropping, resizing, and data-layout conversion to the FPGA and use VDMA to eliminate CPU-mediated frame copies.

This project further cemented my understanding that achieving system-level optimization requires rigorous focus on data movement and protocol translation.