1. Predictions in Neural Networks
In the past we have looked at both logistic regression and binary classification. There, we would collect some data, and then try to predict 1 of 2 possible labels. For example, if we were dealing with an e-commerce stie, we could collect time spent on site and number of pages viewed, and then try to predict whether someone is going to buy something on the site.
2. Neural Network Training
In the previous post we went over the architecture of a Neural Network, and how data passes from input to output, yielding probabilistic predictions. However, the problem was that all of our weights were random and thus, our predictions were not accurate!
3. Backpropagation Calculations
At this point I am going to take an aside to go through a thorough backpropagation walk through. For this walk through, we are going to start by looking at a single logistic neuron, then an overview of backpropagation, and then look at a specific example with numbers!
4. E-commerce Project
We are now going to look at training logistic regression with softmax. In my other walk throughs on logistic regression, we weren't looking at multiclass classification (just binary) so we had just been using the sigmoid function, and not softmax. This will give us the chance to see how logistic regression performs compared to a neural network. Remember, for logistic regression our architecture looks like:
5. Practical Issues
Up until this point we have focused mainly on theory and the two main functions of a neural networks:
6. Facial Recognition Project
We are now going to go through an example project concerning facial recognition.
7. TensorFlow
We are now going to go through a quick TensorFlow example. You may be wondering: "What is TensorFlow?" So here is a quick synopsis from the https://www.tensorflow.org/ website:
8. Modern Deep Learning: Introduction
As we get ready to dive into Modern Deep Learning in Python, a quick review may help to get us started. Note that this is meant to only be a review to material already learned, so if you have not gone through my Deep Learning in Python walkthrough, I highly recommend you do so first.
9. Gradient Descent: Full vs. Batch vs. Stochastic
Let's take a moment to talk about the different ways that we can perform gradient descent, and their particular advantages and disadvantages. The three basic types we will go over are:
10. Momentum and Adaptive Learning Rates
We will now take a look at one of the most effective methods at improving plain gradient descent, called momentum. This can be thought of as the 80% factor to improve your learning procedure.
11. Hyperparameter Optimization
We have just introduced a lot of new hyperparameters, and this is only going to get worse as our neural networks get more and more complex. This leads us in to a topic called hyperparameter optimization. So, what are the hyperparameters that we have learned about so far?
12. Weight Initialization
In this section we will discuss a very popular topic in modern deep learning, weight initialization. You may have wondered in the past why we divide the square root of our weight by the dimensionality?
13. Theano Basics
Going from regular numpy to pandas is a pretty straightforward leap, however, going to theano is not and there are certain things that do not happen as you would expect. So we can now talk about variables.
14. Tensor Flow Basics
We are now going to cover TensorFlow Basics. It is going to introduce basic variables and functions and expressions, and show you how you can optimize a simple function.
15. Facial Expression Recognition - Theano
We are now going to go through the facial expression recognition project that we have worked on in the past, but we will use Theano as our framework of choice this time! We will be creating a neural network that has 2000 units in the first hidden layer, and 1000 units in the second hidden layer. We can start with our imports.
16. Facial Expression Recognition - TensorFlow
We are now going to go through the facial expression recognition project that we have worked on in the past, but we will use TensorFlow as our framework of choice this time! We will be creating a neural network that has 2000 units in the first hidden layer, and 1000 units in the second hidden layer. We can start with our imports.
17. Modern Regularization Techniques
We are now going to look at modern regularization techniques such as drop out. Dropout is a technique developed by Geoff Hinton and team. Up until we have worked with the following types of regularization:
18. Batch Normalization
At this point we should know that normalizing our data for machine learning models is typically a good idea. That is when we make all of the input features have mean of 0 and variance of 1.
1. Recurrent Neural Networks: Introduction
As with the notebooks related to Hidden Markov Models, Recurrent Neural Networks are all about learning sequences. But, where Markov Models are limited by the Markov Assumption, Recurrent Neural Networks are not. As a result, they are more expressive and powerful than anything we have seen and haven't made progress on in decades.
2. The Simple Recurrent Unit
We are now going to talk about the simple recurrent unit, also known as the Elman Unit. But before we do that, I want to quickly touch on sequences. This is going to look slightly different from the data that we are used to.
3. Word Embeddings & Poetry
This section is intended to serve as an introduction to many of the things that we will touch on in my NLP section pertaining to Deep learning applied to natural language processing.
4. Advanced RNN Units
We are now going to move into the realm of advanced RNN units. Up until this point, we have been dealing with the simple recurrent unit exclusively. However, implementing Rated units, Gated Recurrent Units, and Long Short-Term Memory will allow our models to become far more powerful.