Reading: CNF — Context-wise Network Fusion Fusing Multiple CNNs (Super Resolution)
In this story, Image Super Resolution Based on Fusing Multiple Convolution Neural Networks (CNF), by Samsung Semiconductor Inc., is briefly presented. In this paper:
- By fusing multiple CNNs, state-of-the-art accuracy is achieved.
This is a paper in 2017 CVPRW with over 30 citations. (Sik-Ho Tsang @ Medium)
- Pixel-Wise network Fusion (PWF)
- Progressive Network Fusion (PNF)
- Proposed Context-wise Network Fusion (CNF)
- Experimental Results
1. Pixel-Wise network Fusion (PWF)
- Let x denote a LR image and y denote a HR image, and we have M individual CNNs, Sj. For each individual CNN, we got:
- With PWF, the fused output is the pixel-wise weighted sum of the outputs of individual networks, as shown above. We have:
- The PWF might work well for some cases, such as image classification and object detection.
- However, in the super resolution, since the output is a feature map with relatively complicate characteristics, pixel-wise fusion will not fit well.
2. Progressive Network Fusion (PNF)
- For PWF, we progressively organize the individual networks, e.g., using the output of the previous network as the input of the next network. It is inspired by cascade refining the SR output.
- An advantage of the PNF is that it motivates us for a ‘preview’ stage in the super resolution system, e.g., use an efficient network to roughly SR the LR image to a coarse HR image, and then apply a complicate network to refine the coarse HR image to the final output.
- Such SR preview will be very useful in the real SR application.
3. Context-wise Network Fusion (CNF)
3.1. The Idea of CNF
- For the proposed CNF, the weights of the fusion layers could be learned by fine-tuning the whole network.
- In the fine-tuning, the weights of the individual networks could be either frozen or not.
- The above CNF in (b) fine-tunes both the fusion layers and the individual network.
3.2. Deep SRCNN as Individual Network
4. Experimental Results
4.1. Comparison with Different Schemes
- CNF obtains higher PSNR and SSIM, outperforms PNF, PWF, and individual SRCNN networks.
4.2. CNF Variants
- CNF with deep SRCNN of layers 11, 13, 15 with no freezing of the individual networks, obtain the highest PSNR and SSIM.
4.3. SOTA Comparison
This is the 5th story in this month.
[2017 CVPRW] [CNF]
Image Super Resolution Based on Fusing Multiple Convolution Neural Networks
[SRCNN] [FSRCNN] [VDSR] [ESPCN] [RED-Net] [DnCNN] [DRCN] [DRRN] [LapSRN & MS-LapSRN] [MemNet] [IRCNN] [WDRN / WavResNet] [MWCNN] [SRDenseNet] [SRGAN & SRResNet] [SelNet] [CNF] [EDSR & MDSR] [MDesNet] [RDN] [SRMD & SRMDNF] [DBPN & D-DBPN] [RCAN] [ESRGAN] [SR+STN]