Review: DMRNet / DFN-MR — Merge-and-Run Mappings (Image Classification)

Outline

  1. From Residual Block to Merge-and-Run Block
  2. Analyses of Merge-and-Run Block
  3. Experimental Results

1. From Residual Block to Merge-and-Run Block

(a) Residual Block, (b) Vanilla-Assembly Block, (c) Merge-and-Run Block (Dotted Circle: Average Operation, Solid Circle: Sum Operation)
The network formed by (a) Residual Block, (b) Vanilla-Assembly Block, © Merge-and-Run Block (Dotted Circle: Average Operation, Solid Circle: Sum Operation)

1.1. Residual Block (ResNet)

  • The above equation is the very familiar equation of a Residual Block from ResNet. With xt as input from t-th residual block, H(t) is the output of convolutional path. By adding both, we got xt+1.

1.2. Vanilla-Assembly Block (DVANet)

  • Before talking about Merge-and-Run block, there is also the Vanilla-Assembly Block.
  • It is a ResNeXt-like block but with only 2 convolutional paths. (If interest, please read my review about ResNeXt.)

1.3. Merge-and-Run Block (DMRNet)

  • A Merge-and-Run Block is formed by assembling two residual branches in parallel with a merge-and-run mapping:
  • Merge: Average the inputs of two residual branches.
  • Run: And add the average to the output of each residual branch as the input of the subsequent residual branch.

2. Analyses of Merge-and-Run Block

2.1. Information Flow Improvement

  • The above equation can be written in matrix form:
  • With
  • It can be:
  • This shows that during the forward flow there are quick paths directly sending the input and the intermediate outputs to the later block.
  • A similar conclusion can be drawn for gradient back-propagation.
  • Thus, merge-and-run mappings can improve both forward and backward information flow.

2.2. Shorter Paths

Comparing the distributions of the path lengths for three networks.
  • All the three networks are mixtures of paths, where a path is defined as a sequence of connected residual branches, identity mappings, and possibly other layers (e.g., the first convolution layer, the FC layer) from the input to the output.
  • The proposed network are distributed in the range of lower lengths, potentially performs better.

2.3. DVANet and DMRNet are Wider

  • For Vanilla-Assembly Block (DVANet) in matrix form:
  • There are two parallel residual branches.
  • Hence, Merge-and-Run Block (DMRNet) is also wider.
  • But in DMRNet, two residual branches are not independent as there is a merge-and-run mapping.

3. Experimental Results

3.1. Merge-and-Run Mapping

Comparison between merge-and-run mappings and identity mappings.
  • With Merge-and-Run mapping, it consistently performs better than the networks without Merge-and-Run mapping.

3.2. Comparison with Wide ResNet

Average Classification Error from 5 Runs on CIFAR-10, CIFAR-100, SVHN
  • DMRNet performs the best on CIFAR-10.
  • And the superiority of DVANets over ResNets stems from the less long paths and greater width.
  • On CIFAR-100 and SVHN, when the network is deep enough, DMRNet performs the best.
  • But when the network is not deep enough, ResNet and Wide-ResNet are better. Authors believe the paths in the DVANet and DMRNet are not very long and too many short paths lower down the performance for networks in such a shallow network.

3.3. Combination with ResNeXt

Average Classification Error from 5 Runs on CIFAR-10, CIFAR-100
  • In ResNeXt, it can support K>2 convolutional paths.
  • By using Merge-and-Run Mapping on ResNeXt, it becomes DMRNeXt and outperforms ResNeXt which shows the efficiency of Merge-and-Run Mapping.

3.4. Combination with Xception

Average Classification Error from 5 Runs on CIFAR-10, CIFAR-100
  • Here, DMRNets contain two Xception blocks.
  • Again, it outperforms Xception which shows the efficiency of Merge-and-Run Mapping.

3.5. Comparison with State-of-the-art Approaches

Classification error comparison with state-of-the-arts

References

My Previous Reviews

--

--

PhD, Researcher. I share what I learn. :) Reads: https://bit.ly/33TDhxG, LinkedIn: https://www.linkedin.com/in/sh-tsang/, Twitter: https://twitter.com/SHTsang3

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store