Reading: CNF — Context-wise Network Fusion Fusing Multiple CNNs (Super Resolution)

Fusing Multiple Deeper SRCNN, Outperforms A+, SRCNN, VDSR and DRCN.

In this story, Image Super Resolution Based on Fusing Multiple Convolution Neural Networks (CNF), by Samsung Semiconductor Inc., is briefly presented. In this paper:

By fusing multiple CNNs, state-of-the-art accuracy is achieved.

This is a paper in 2017 CVPRW with over 30 citations. (Sik-Ho Tsang @ Medium)

Outline

Pixel-Wise network Fusion (PWF)
Progressive Network Fusion (PNF)
Proposed Context-wise Network Fusion (CNF)
Experimental Results

1. Pixel-Wise network Fusion (PWF)

Let x denote a LR image and y denote a HR image, and we have M individual CNNs, Sj. For each individual CNN, we got:

With PWF, the fused output is the pixel-wise weighted sum of the outputs of individual networks, as shown above. We have:

The PWF might work well for some cases, such as image classification and object detection.
However, in the super resolution, since the output is a feature map with relatively complicate characteristics, pixel-wise fusion will not fit well.

2. Progressive Network Fusion (PNF)

For PWF, we progressively organize the individual networks, e.g., using the output of the previous network as the input of the next network. It is inspired by cascade refining the SR output.
An advantage of the PNF is that it motivates us for a ‘preview’ stage in the super resolution system, e.g., use an efficient network to roughly SR the LR image to a coarse HR image, and then apply a complicate network to refine the coarse HR image to the final output.
Such SR preview will be very useful in the real SR application.

3. Context-wise Network Fusion (CNF)

3.1. The Idea of CNF

For the proposed CNF, the weights of the fusion layers could be learned by fine-tuning the whole network.
In the fine-tuning, the weights of the individual networks could be either frozen or not.
The above CNF in (b) fine-tunes both the fusion layers and the individual network.

3.2. Deep SRCNN as Individual Network

A 3-layer SRCNN as the baseline following the SRCNN 9–5–5 structure.
To go deeper, there is also 5-layer SRCNN 9–5–3–3–5.
Then, two 3 × 3 layers are added each time, which results in SRCNNs from 3 layers to 15 layers.