Using VGGNet+Conv-LSTM, Outperforms BTBCRL, DeFusionNet & DHDE

Image for post
Image for post
An example of defocus blur detection
  • Multiscale convolutional features from same image with different sizes are extracted.
  • Conv-LSTMs are used to integrate the fused features gradually from top-to-bottom layers, and to generate multiscale blur estimations.

Outline

  1. Multiscale Feature Extraction Sub-Network (MsFEN)
  2. Multiscale Blur Detection Sub-Network (MsBEN)
  3. Experimental Results

1. Multiscale Feature Extraction Sub-Network (MsFEN)


ResNet + FCN-2s, Outperforms U-Net, SegNet & FCN-2s(VGG16)

Image for post
Image for post
First Row: Images, Second Row: Ground Truth Masks
  • Blur Detection Net, BDNet, is proposed to well integrate global image-level context and cross-layer context information by combining ResNet and Fully Convolutional Network (FCN).

Outline

  1. Datasets
  2. BDNet: Network Architecture
  3. Experimental Results

1. Datasets

  • There are three datasets used in the experiments called BlurRaw, BlurDB1 and BlurDB2.


Extension of DeFusionNet (CVPR’19). Outperforms DHDE, DBM, DMENet, BTBNet, and DeFusionNet (CVPR’19)

  • This paper is the extension of DefusionNet in CVPR’19. Since it is an extension, I will mainly describe the new stuffs in this paper.
  • An enhanced feature fusing and refining module (FFRM) is proposed where Channel Attention Module (CAM) and Feature Adaptation Module (FAM) is used.
  • A more challenging new dataset is also proposed. …


U-Net-Like Architecture. Outperforms Park CVPR’17 and DBM

Image for post
Image for post
Example of a challenging case in blur detection
  • A deep encoder-decoder U-Net-like network with long residual skip-connections and multi-scale reconstruction loss to exploit high-level contextual features as well as low-level structural features.
  • A synthetic dataset is constructed that consists of complex scenes with both motion and defocus blur.

Outline

  1. Synthetic Dataset for Blur Classification
  2. Blur Detection and Classification using DCNN
  3. Experimental Results

1. Synthetic Dataset for Blur Classification


Image for post
Image for post
Some challenging cases for defocus blur detection
  • A fully convolutional network is used to extract multi-scale deep features. These features from different layers are fused as shallow features and semantic features.
  • The feature fusing and refining are carried out in a recurrent manner. Finally, the output of each layer at the last recurrent step is fused to obtain the final defocus blur map.


Using Domain Adaptation, Dataset with Synthetic Blurring is used, Outperforms Park CVPR’17

  • A novel depth-of-field (DOF) dataset, SYNDOF, is produced where each image is synthetically blurred with a ground-truth depth map.
  • As the feature characteristics of images in SYNDOF can differ from those of real defocused photos, domain adaptation is used to transfer the features of real defocused photos into those of synthetically blurred ones.
  • DMENet consists of four subnetworks: blur estimation, domain adaptation, content preservation, and sharpness calibration networks. The subnetworks are connected to each other and jointly trained.


Ensemble Network With Enhancing Diversity, Outperforms BTBNet & Park CVPR’17 / DHCF

  • A novel learning strategy by breaking DBD problem into multiple smaller defocus blur detectors and thus estimate errors can cancel out each other.
  • Cross-negative and self-negative correlations and an error function are designed to enhance ensemble diversity and balance individual accuracy.

Outline

  1. Problem Formulation
  2. Single Detector Network (SENet)
  3. Multi-detector Ensemble Network (MENet)
  4. Proposed Cross-Ensemble Network (CENet)
  5. CENet: Network Architecture & Training & Testing
  6. Experimental…


From CAM, Grad-CAM, to Grad-CAM++. Better Visual Explanations Than Grad-CAM

Image for post
Image for post
All dogs (Multiple objects) are better visualized (1st and 2nd Rows) and the entire region of the class is localized (3rd and 4th Rows) in the Grad-CAM++ and Guided Grad-CAM++ saliency maps while Grad-CAM heatmaps only exhibit partial coverage.
  • Grad-CAM++, built on Grad-CAM, provides better visual explanations of CNN model predictions, in terms of better object localization as well as explaining occurrences of multiple object instances in a single image.
  • A weighted combination of the positive partial derivatives of the last convolutional layer feature maps with respect to a specific class score as weights is used to generate a visual explanation for the corresponding class label.


Using a Pyramid M-Shaped Deep Neural Network, Outperforms Park CVPR’17 & Zeng TIP’19, etc.

Image for post
Image for post
Examples of (a) globally blurred image, (b) © partially motion-blurred images, and (d) partially defocused image.
  • Handcrafted features can hardly differentiate a blurred region from a sharp but homogeneous region.
  • Another challenge is that blur metrics based on handcrafted features are difficult to detect the pseudo-sharp backgrounds.
  • A novel multi-input multi-loss encoder-decoder network (M-shaped) is proposed to learn rich hierarchical representations related to blur.
  • Blur degree is susceptible to scales, a pyramid ensemble model (PM-Net) consisting of different scales of M-shaped subnets and a unified fusion layer, is constructed. …


CNN + PCA for Feature Learning, Iterative Refinement Using Tanh

Image for post
Image for post
(a) Input, (b) SOTA [37], (c) Proposed, (d) Ground Truth
  • The ConvNets automatically learn the most locally relevant features.
  • By extracting convolution kernels from the trained neural network structures and processing it with principal component analysis (PCA), the local sharpness metric is automatically obtained by reshaping the principal component vector.
  • An effective iterative updating mechanism is proposed to refine the defocus blur detection result from coarse to fine by hyperbolic tangent function.

About

Sik-Ho Tsang

PhD, Researcher. I share what I've learnt and done. :) My LinkedIn: https://www.linkedin.com/in/sh-tsang/, My Paper Reading List: https://bit.ly/33TDhxG

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store