Review — LIU4K: A Comprehensive Benchmark for Single Image Compression Artifact Reduction (Codec Filtering)

LIU4K, Large-Scale Ideal Ultra high-definition 4K, A New 4K Image Dataset is Proposed

Example images sampled from LIU4K. (a) Training set. (b) Testing set.

In this story, A Comprehensive Benchmark for Single Image Compression Artifact Reduction, (LIU4K), by Peking University, and University of Science and Technology of China, is reviewed. In this paper:

• LIU4K: Large-Scale Ideal Ultra high-definition 4K, a new dataset, is proposed.
• A summary of state-of-the-art single image filtering approaches are also reviewed and evaluated using both full reference and no reference metric for objective and subjective quality measurement respectively.

This is a paper in 2020 TIP where TIP has a high impact factor of 9.34. (Sik-Ho Tsang @ Medium)

Outline

1. LIU4K: Large-Scale Ideal Ultra high-definition 4K
2. Summary of State-Of-The-Art Single Image Filtering Approaches
3. Benchmarking

1. LIU4K: Large-Scale Ideal Ultra high-definition 4K

Dataset Summary

1.1. High-resolution definition

• Compared to previous datasets, the resolution of the images in LIU4K is 2848×4288, offering abundant materials for testing and evaluating the performance on 4K/8K display devices.

1.2. Large-scale.

• The training, testing, and validation images include 1,500, 200, and 80 4K images, which is much more than in previous datasets.

1.3. Diversified and complex signals

The statistical comparison of different testing sets

• LIU4K achieves the best results in terms of entropy-driven non-reference metrics (ENTROPY), which demonstrates its signal diversity and complexity.

1.4. High visual quality

• LIU4K wins in general purpose non-reference metrics (ENIQA and NIQE).

2. Summary of State-Of-The-Art Single Image Filtering Approaches

Milestones in the history of compressed image restoration methods

• There are four categories: filter-based methods, probabilistic prior-based methods, deep learning-based JPEG artifacts removal methods (starting from 2015 ARCNN), and deep learning-based loop filter methods (starting from 2017 VRCNN).
• Most previous works are based on these two standards: JPEG and HEVC.
• Deep learning approaches are more likely to focus in this story.

The technical improvement pathway for deep learning-based compression artifacts removal and codec loop filters

2.1. Deep Learning-Based JPEG Artifacts Removal

• The works fall into two main streams: better network architectures (2–1) and better utilization of DCT domain information (2–2).

• Better network architectures: ARCNN, DnCNN, DRN, RED-Net, MemNet, etc.

• Better utilization of DCT domain information: D³, DMCNN, MWCNN (Wavelet), etc.
• Besides network improvement, some works try to embed traditional priors or constraints into deep networks.
• Below shows PSNR that improved across the years.

Recent evolution of deep learning-based JPEG artifacts removal

2.2. Deep Learning-Based Loop Filter Methods (HEVC)

• Deep-learning based loop filters focus more on handling the degradation caused by variable-size partitions and utilizing side information from codecs (1), e.g.: VRCNN, RHCNN, and MLSDRN.
• There are still a lot of works summarized in the paper.

This paper summarizes and introduces a lot of approaches, it is very good for those who starts to work on the filtering research.

3. Benchmarking

• For JPEG artifact reduction, the models are trained on the training of LIU4K.
• For loop filters, the models are trained on the training sets of both BSD500 and LIU4K. During the training phase, 80 additional 4K images are used as LIU4K validation set.
• ARCNN is modified/enhanced for better convergence.

Objective Evaluation

• DMCNN obtains better performance in terms of PSNR and SSIM.
• But it cannot obtain good results for NIQE and BRISQUE, which are no-reference metrics.

MOS Scores

• Mean Opinion Scores (MOS) are also obtained. The scores are given by 40 participants.
• It is observed that, DMCNN, MemNet, and PRN achieve overall superior visual quality than other methods.

Examples of restored results for a compressed image utilizing HM from LIU4K (QF = 10).

• It is observed that DMCNN achieves the overall best visual quality.
• The summarization and experiments are in very details. I’m just here to introduce there is a paper summarizing the JPEG and HEVC filtering works.

I believe there will be research works making use of this new LIU4K dataset in the coming future.

Reference

[2020 TIP] [LIU4K]
A Comprehensive Benchmark for Single Image Compression Artifact Reduction

Codec Filtering

JPEG [ARCNN] [RED-Net] [DnCNN] [Li ICME’17] [MemNet] [MWCNN] [CAR-DRN] [LIU4K]
JPEG-HDR [Han VCIP’20]
HEVC [Lin DCC’16] [IFCNN] [VRCNN] [DCAD] [MMS-net] [DRN] [Lee ICCE’18] [DS-CNN] [CNNF] [RHCNN] [VRCNN-ext] [S-CNN & C-CNN] [MLSDRN] [ARTN] [Double-Input CNN] [CNNIF & CNNMC] [B-DRRN] [Residual-VRN] [Liu PCS’19] [DIA_Net] [RRCNN] [QE-CNN] [MRRN] [Jia TIP’19] [EDCNN] [VRCNN-BN] [MACNN] [Yue VCIP’20] [SEFCNN] [LIU4K]
3D-HEVC [RSVE+POST]
AVS3 [Lin PCS’19] [CNNLF]
VVC [AResNet] [Lu CVPRW’19] [Wang APSIPA ASC’19] [ADCNN] [PRN] [DRCNN] [Zhang ICME’20] [MGNLF] [RCAN+PRN+] [Nasiri VCIP’20]

My Other Previous Paper Readings