# Lesson 3

From Deep Learning Course Wiki

## Contents

## Lesson resources

- Lesson 3 video and associated Lesson 3 Video Timeline
- Lesson Notes
- Lesson 3 notebook
- MNIST walk-through notebook

## Assignments

- Read through the following notebooks carefully:
- lesson1.ipynb
- lesson2.ipynb
- sgd-intro.ipynb
- convolution-intro.ipynb
- lesson3.ipynb

- Ask at least
**one question**on the forums. Possible types of questions include:- What is the purpose of <something>?
- Why do we write <something> in python?
- Why is the output of <something> equal to <something>, instead of <something else>?
- How do I fix this error?
- Could I also use <some technique> for <something>?
- Why aren't I getting a better position on the leaderboard for <competition> using <some process>?

- Be sure that you can independently replicate the steps shown in each lesson notebook so far
- We suggest following the guidance: How to use the Provided Notebooks

- Get a result in the top 50% of Dogs v Cats, if you haven't already
- Get a result in the top 50% of State Farm
- Be sure that you have created a validation set that gives similar results to submitting to Kaggle
- Think about which layers of the pre-trained model are likely to be the most useful

## CNN review

Today we reviewed all the key components of a convolutional neural network. Here are some resources you can use to help you if you are unclear on any piece:

### Matrix Product (dense layers)

- Matrix Multiplication
- Khan Academy - Good introduction to the basics
- 3 Blue 1 Brown videos - Excellent videos with lots of visualizations and explains the intuition behind matrix products.

### Convolutions (and Max-Pooling)

- Image kernels explained visually - fantastic interactive visualizations of convolutions
- Understanding convolutions - a deep dive into convolutions, looking not just at image processing applications, but a more general approach
- Jeremy's convolution notebook motivates and builds convolutions from scratch, as well as introducing max-pooling
- This blog post is a nice introductory overview to CNNs, including convolutions and max-pooling.

### Activations

- This documentation from the Stanford Course has an overview of commonly used activation functions and their pros vs. cons.

### Stochastic Gradient Descent (SGD)

- Gradient Descent
- Jeremy's jupyter notebook walking through using stochastic gradient descent for linear regression, with animation.
- Detailed tutorials explaining and implementing in Python gradient descent and stochastic gradient descent.
- An animation of gradient descent (not stochastic).

#### Backpropagation (Chain Rule)

- Chris Olah's tutorial on backpropagation, which may resonate with those with CS backgrounds.
- Khan Academy lesson on the Chain Rule.

### Putting it all together

- Stanford CNN class - some of the clearest technical writing we've seen on any topic
- Chapter 6 of Michael Nielsen's book