Accelerating Image Super-Resolution Networks with Pixel-Level Classification

ECCV 2024

Jinho Jeong1, Jinwoo Kim1, Younghyun Jo2, Seon Joo Kim1
1Yonsei University, 2Samsung Advanced Institute of Technology
Paper arXiv Code
Teaser1

Abstract

In recent times, the need for effective super-resolution (SR) techniques has surged, especially for large-scale images ranging 2K to 8K resolutions. For DNN-based SISR, decomposing images into overlapping patches is typically necessary due to computational constraints. In such patch-decomposing scheme, one can allocate computational resources differently based on each patch's difficulty to further improve efficiency while maintaining SR performance. However, this approach has a limitation: computational resources is uniformly allocated within a patch, leading to lower efficiency when the patch contain pixels with varying levels of restoration difficulty. To address the issue, we propose the Pixel-level Classifier for Single Image Super-Resolution (PCSR), a novel method designed to distribute computational resources adaptively at the pixel level. A PCSR model comprises a backbone, a pixel-level classifier, and a set of pixel-level upsamplers with varying capacities. The pixel-level classifier assigns each pixel to an appropriate upsampler based on its restoration difficulty, thereby optimizing computational resource usage. Our method allows for performance and computational cost balance during inference without re-training. Our experiments demonstrate PCSR's advantage over existing patch-distributing methods in PSNR-FLOP trade-offs across different backbone models and benchmarks.

Teaser2

Overall Framework

Overview

Overview of PCSR. The architecture of the proposed PCSR model when the number of classes M is 2. We denote q as a single query pixel in the HR space and xq for its coordinate. Pixel-level probabilities obtained from the classifier are used to allocate each query pixel to a suitably-sized upsampler for the prediction of its RGB value.

Comparison with Previous Methods

Main Quantitative Results

The quantitative comparison of the previous patch-level methods and our pixel-level method PCSR on the large image SR benchmarks: Test2K, Test4K, Test8K, and Urban100 with ×4 SR. The lowest FLOPs values are highlighted in bold.

Main Qualitative Results

Qualitative results of previous methods and our method with X4 SR.

Citation

@misc{jeong2024acceleratingimagesuperresolutionnetworks,
      title={Accelerating Image Super-Resolution Networks with Pixel-Level Classification}, 
      author={Jinho Jeong and Jinwoo Kim and Younghyun Jo and Seon Joo Kim},
      year={2024},
      eprint={2407.21448},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.21448},
}