# Lesson 3

From Deep Learning Course Wiki

## Contents

## Lesson resources

- Lesson 3 video and associated Lesson 3 Video Timeline
- Lesson Notes
- Lesson 3 notebook
- MNIST walk-through notebook

## Assignments

- Read through the following notebooks carefully:
- lesson1.ipynb
- lesson2.ipynb
- sgd-intro.ipynb
- convolution-intro.ipynb
- lesson3.ipynb

- Ask at least
**one question**on the forums. Possible types of questions include:- What is the purpose of <something>?
- Why do we write <something> in python?
- Why is the output of <something> equal to <something>, instead of <something else>?
- How do I fix this error?
- Could I also use <some technique> for <something>?
- Why aren't I getting a better position on the leaderboard for <competition> using <some process>?

- Be sure that you can independently replicate the steps shown in each lesson notebook so far
- We suggest following the guidance: How to use the Provided Notebooks

- Get a result in the top 50% of Dogs v Cats, if you haven't already
- Get a result in the top 50% of State Farm
- Be sure that you have created a validation set that gives similar results to submitting to Kaggle
- Think about which layers of the pre-trained model are likely to be the most useful

## CNN review

Today we reviewed all the key components of a convolutional neural network. Here are some resources you can use to help you if you are unclear on any piece:

### Matrix Product (dense layers)

- Matrix Multiplication
- Khan Academy - Good introduction to the basics
- Matrix Multiplication - A different perspective - For a visually understanding the rule for Matrix Multiplication and the multiplication output.
- 3 Blue 1 Brown videos - Excellent videos with lots of visualizations and explains the intuition behind matrix products.

### Convolutions (and Max-Pooling)

- Image kernels explained visually - fantastic interactive visualizations of convolutions
- Understanding convolutions - a deep dive into convolutions, looking not just at image processing applications, but a more general approach
- Jeremy's convolution notebook motivates and builds convolutions from scratch, as well as introducing max-pooling
- This blog post is a nice introductory overview to CNNs, including convolutions and max-pooling.

### Activations

- This documentation from the Stanford Course has an overview of commonly used activation functions and their pros vs. cons.

### Stochastic Gradient Descent (SGD)

- Gradient Descent
- Jeremy's jupyter notebook walking through using stochastic gradient descent for linear regression, with animation.
- Detailed tutorials explaining and implementing in Python gradient descent and stochastic gradient descent.
- An animation of gradient descent (not stochastic).

#### Backpropagation (Chain Rule)

- Chris Olah's tutorial on backpropagation, which may resonate with those with CS backgrounds.
- Khan Academy lesson on the Chain Rule.

### Putting it all together

- Stanford CNN class - some of the clearest technical writing we've seen on any topic
- Chapter 6 of Michael Nielsen's book

## Additional Resources

- This week in Episode 006 of the Startup Data Science podcast Apurva, Alex, and Edderic exchange views on their understanding of the difference between Covolution and Correlation and make sure they agree on the meaning of Max Pooling. Episode 007 talks about Dropout in detail. Apurva offers some tips to stay motivated to learn Deep Learning and Edderic announces revamping his PC workstation for deep learning (bye-bye Amazon!)