Brief Review — Rectified Linear Units Improve Restricted Boltzmann Machines

Rectified Linear Unit (ReLU) Introduced

Rectified Linear Units Improve Restricted Boltzmann Machines
ReLU
, by University of Toronto
2010 ICML, Over 17000 Citations (Sik-Ho Tsang @ Medium)
Activation Function, Restricted Boltzmann Machine, Image Classification, Face Recognition

  • Rectified Linear Unit (ReLU) is introduced, which outperforms Sigmoid.
  • This is a paper from Hinton’s research group.

Outline

  1. Rectified Linear Unit (ReLU)
  2. Image Classification Results
  3. Face Recognition Results

1. Rectified Linear Unit (ReLU)

  • More precisely, Noisy ReLU is proposed to replace the logistic sigmoid function needs to be used many times to get the probabilities required for sampling an integer value correctly:
  • where N(0, V) is Gaussian noise with zero mean and variance V.

2. Image Classification Results

Network architecture used for the Jittered-Cluttered NORB classification task
  • Two hidden layers of NReLUs as RBMs, are greedily pretrained. (For RBM, please read Autoencoder.)
  • The class label is represented as a K-dimensional binary vector with 1-of-K activation, where K is the number of classes.
  • The classifier computes the probability of the K classes from the second layer hidden activities h2 using the softmax function.
Test error rates for classifiers with 4000 hidden units trained on 32×32×2 Jittered-Cluttered NORB images
  • Pre-training helps improve the performance of both unit types.
  • But NReLUs without pre-training are better than binary units with pre-training.
Test error rates for classifiers with two hidden layers (4000 units in the first, 2000 in the second), trained on 32×32×2 Jittered-Cluttered NORB images
  • Pre-training both layers gives further improvement for NReLUs but not for binary units.

3. Face Recognition Results

Siamese network used for the Labeled Faces in the Wild task
  • The feature extractor FW contains one hidden layer of NReLUs pre-trained as an RBM. (For RBM, please read Autoencoder.)
  • Cosine distance is used to check whether the faces are the same.
Accuracy on the LFW task for various models trained on 32×32 colour images
  • Models using NReLUs are more accurate.

This paper and AlexNet are often cited when ReLU is used. Classic!

Reference

[2010 ICML] [ReLU]
Rectified Linear Units Improve Restricted Boltzmann Machines

Image Classification

1989 2010 [ReLU] … 2022 [ConvNeXt] [PVTv2]

Face Recognition

2005 [Chopra CVPR’05] 2010 [ReLU] 2014 [DeepFace] [DeepID2] [CASIANet] 2015 [FaceNet] 2016 [N-pair-mc Loss]

My Other Previous Paper Readings

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store