Sharing — Deep Learning for AI, By Yoshua Bengio, Yann LeCun, and Geoffrey Hinton

2018 Turing Award, 2021 Communications of the ACM

2018 ACM A.M. Turing Award
Yann LeCun

1. The Rise of Deep Learning

  1. The software platform such as Caffe, TensorFlow, and PyTorch.
  2. The invention of deep network such as ResNet.
  3. Unsupervised learning using pretraining (Denoising Autoencoder).
  4. The mysterious success of rectified linear units (ReLU).
  5. The breakthrough in speech and object recognition (Deep Relief Network).
  6. The dramatic victory by AlexNet in the 2012 ImageNet competition.
Geoffrey Hinton

2. Recent Advances

  1. Soft attention and the Transformer architecture.
  2. Unsupervised and self-supervised learning in NLP (e.g. BERT).
  3. Contrastive learning (e.g. SimCLR) and Non contrastive learning (e.g. SwAV & BYOL) in computer vision.
  4. Variational auto-encoders (VAE).
Yoshua Bengio

3. The Future of Deep Learning

  • Right now, many test cases expected to come from the same distribution as the training examples. Unfortunately, this is not the case in real world.
  • Humans can generalize in a way that is different and more powerful than ordinary iid generalization: we can correctly interpret novel combinations of existing concepts, even if those combinations are extremely unlikely under our training distribution.
  • Evidence from neuroscience suggests that groups of nearby neurons (forming what is called a hyper-column) are tightly connected and might represent a kind of higher-level vector-valued unit able to send not just a scalar quantity but rather a set of coordinated values.
  • Current deep learning is most successful at perception tasks and generally what are called system 1 tasks. Using deep learning for system 2 tasks that require a deliberate sequence of steps is an exciting area that is still in its infancy.
  • There are still many other questions raised at the end of the article. Please feel free to read the article directly.
  • Finally, below is the video associated with the article, but the materials mentioned in the article and in the video are very different.
July 2021 CACM: Deep Learning for AI: https://www.youtube.com/watch?v=at0OXoUmEj4

Reference

Legends

My Other Previous Paper Readings

--

--

PhD, Researcher. I share what I learn. :) Linktree: https://linktr.ee/shtsang for Twitter, LinkedIn, etc.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store