Reading: RCAN+PRN+ — Ensemble Learning of Neural Networks (VVC Filtering)
Various Types of Model Ensemble, Bring Additional Performance Gain.
In this story, Compression Artifact Removal With Ensemble Learning of Neural Networks (RCAN+PRN+), is presented. I just call this paper, RCAN+PRN+, since it uses these two models for ensemble. I read this because I work on video coding research. In this paper:
- RCAN and PRN+ are ensembled for removing the compression artifact.
- There are QP-Driven Ensemble, MSE-Driven Ensemble, Image-Set-Driven Ensemble, and Ensemble of Different Architectures.
- This post processing filter is applied to DLVC decoder with 2 deep tools disabled, which has similar performance compared to VTM-7.0. (Also, the DLVC has the QTMT partitioning, thus, I just treat it as VVC filtering.)
This is a paper in 2020 CVPR Workshop (CVPRW). (Sik-Ho Tsang @ Medium)
1. What is Ensemble Learning?
- In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.
- Two kinds of ensemble learning are introduced: Majority Vote and Soft Voting.
1.1. Majority Vote
- One of the ensembles is majority vote.
- The class that the most classifiers predicted are the final predicted results.
1.2. Soft Voting
- Another one is soft voting.
- If the classifier outputs the probability of the predicted class, we can average the result to get a more accurate prediction.
- (Please feel free to read the reference: https://www.kaggle.com/
1.3. Example in CNN — LeNet
- RCAN is originally used for super resolution.
- The upsampling part is removed and used as post-processing filter.
- PRN is originally designed for in-loop filters.
- The capacity is further improved by deepening the network to twice the original depth.
- Also, the network is trained using RGB space.
3. Experimental Results
- CLIC Training set and DIV2K are used for training.
- Evaluation is performed on CLIC 2020 validation set.
- QP range is in [36,38] using DLVC without CNNLF.
- A set of images are selected with average bit-per-pixel less than 0.15 for training.
3.2. QP-Driven Ensemble
- With 3 PRN+ trained using QP 32, 37, 42 respectively, and then ensembled during evaluation, the highest PSNR and MS-SSIM are obtained.
3.3. MSE-Driven Ensemble
- The images are divided into 3 subsets according to three levels of MSE, i.e. Low (L), Median (M) and High (H), then used for training the RCAN model.
- Full (F): All images are used for training.
- With H+M+F RCAN models ensembled, the highest PSNR and MS-SSIM are obtained.
3.4. Image-Set-Driven Ensemble
- First train a model with the mixed dataset (M).
- Then fine-tune the model to produce two different models, each tuned on either CLIC (C) or DIV2K (D) dataset.
- Using Mixed+C+D (All), for ensembling, the highest PSNR and MS-SSIM are obtained.
3.5. Ensemble of Different Architectures
- For two model A and B, A+B refers to averaging pixels while A/B (N) means conducting block-level selection with the block size N.
- Block-level selection introduces a flag to select the model.
- P and R stand for PRN+ and RCAN, respectively, while P-All and R-HMF correspond to the best ensemble results in section 2.4 and section 2.3.
- P-All / R-HMF (96) obtains the best result.
- Finally, different patch sizes are tried.
- By using block-wise ensemble with streamed flags, P-All / R-HMF (48), further PSNR improvement is achieved with slight amount of increase in bitrate.
[2020 CVPRW] [RCAN+PRN+]
Compression Artifact Removal with Ensemble Learning of Neural Networks
JPEG [ARCNN] [RED-Net] [DnCNN] [Li ICME’17] [MemNet] [MWCNN]
HEVC [Lin DCC’16] [IFCNN] [VRCNN] [DCAD] [MMS-net] [DRN] [Lee ICCE’18] [DS-CNN] [CNNF] [RHCNN] [VRCNN-ext] [S-CNN & C-CNN] [MLSDRN] [ARTN] [Double-Input CNN] [CNNIF & CNNMC] [B-DRRN] [Residual-VRN] [Liu PCS’19] [DIA_Net] [RRCNN] [QE-CNN] [Jia TIP’19] [EDCNN] [VRCNN-BN] [MACNN]
AVS3 [Lin PCS’19] [CNNLF]
VVC [AResNet] [Lu CVPRW’19] [Wang APSIPA ASC’19] [ADCNN] [PRN] [DRCNN] [Zhang ICME’20] [MGNLF] [RCAN+PRN+]