Citation: Min, S.-J.; Jo, Y.-S.; Kang,
S.-J. Super-Resolving Methodology
for Noisy Unpaired Datasets. Sensors
2022, 22, 8003. https://doi.org/
10.3390/s22208003
Academic Editor: Jianbo Yu
Received: 3 September 2022
Accepted: 18 October 2022
Published: 20 October 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Super-Resolving Methodology for Noisy Unpaired Datasets
Sung-Jun Min
†
, Young-Su Jo
†
and Suk-Ju Kang *
Department of Electronic Engineering, Sogang University, Seoul 04107, Korea
* Correspondence: sjkang@sogang.ac.kr
† These authors contributed equally to this work.
Abstract:
Although it is possible to acquire high-resolution and low-resolution paired datasets, their
use in directly supervised learning is impractical in real-world applications. In the present work, we
focus on a practical methodology for image acquisition in real-world conditions. The main method of
noise reduction involves averaging multiple noisy input images into a single image with reduced
noise; we also consider unpaired datasets that contain misalignments between the high-resolution
and low-resolution images. The results show that when more images are used for average denoising,
better performance is achieved in the super-resolution task. Quantitatively, for a fixed noise level
with a variance of 60, the proposed method of using 16 images for average denoising shows better
performance than using 4 images for average denoising; it shows 0.68 and 0.0279 higher performance
for the peak signal-to-noise ratio and structural similarity index map metrics, as well as 0.0071 and
1.5553 better performance for the learned perceptual image patch similarity and natural image quality
evaluator metrics, respectively.
Keywords: super resolution, unpaired dataset, average denoising
1. Introduction
In visual inspection tasks involving semiconductor images, resolution is an important
parameter that determines the structural information or defects. In such tasks, images
are often acquired at lower resolutions owing to the limited resources of sophisticated
hardware, and inspection is not possible because of lack of detail and texture. Therefore,
the super-resolution (SR) technique is frequently used for restoring low-resolution (LR)
to high-resolution (HR) images to enhance details and structural information in practical
visual inspection tasks.
Generally, image restoration (IR) aims to restore high-quality images from degraded
images, but it is an ill-posed problem for an observed image and can be modeled as follows:
y = DHx + n, (1)
where x is the original input image, D is the down-sampling operator, H is a blur kernel,
and n is additive noise. SR tasks constitute a field of IR that has been actively researched
and aims to restore HR images containing missing high-frequency details from LR images.
Recently, deep-learning-based SR methods have shown improved performance compared
to existing methods from a practical point of view.
However, most existing SR methods assume that both the input and reconstructed
images are noise-free and aligned perfectly. Unfortunately, in practical cases, noise is
inevitably included during image acquisition, making this assumption invalid for real-
world applications and rendering these SR tasks more difficult. This is because it is difficult
to prevent noise amplification during up-sampling, which often leads to loss of information
and emergence of artifacts. Further, alignment problems occur when acquiring HR–LR
image pairs. As observed in [
1
,
2
], when an HR–LR pair is directly photographed and the
alignments do not match, scale-invariant feature transform (SIFT) and random sample
Sensors 2022, 22, 8003. https://doi.org/10.3390/s22208003 https://www.mdpi.com/journal/sensors
