Machine Learning | Quizlet

Gaussian Process What

A probabilistic model that defines a distribution over possible functions to predict outputs from inputs together with uncertainty summarized by mean and variance

Gaussian Process How

We start with inputs and outputs and a mean function and covariance or kernel function to predict outputs with an uncertainty value. Hyperparameters are learned by maximizing the log marginal likelihood

GP Kernel

Tells us how similar two inputs are. A common kernel is the RBF Radial Basis Function where values close together get a value closer to 1 and values far away approach 0

GP Gamma Value

A hyperparameter used to tune the distance to determine essentially what close is

GP Why

Beneficial if data is limited or uncertainty is useful or if experiments are expensive. Very good at small data sets but traditional GPs become computationally expensive as training points grow due to the covariance matrix

GP Pros

Mature theory and stochastic guarantees and uncertainty bounds which are very useful for control and model certification

GP Cons

Can be seen as a 1-layer ANN which leads to lower approximation capabilities

Applying GP Steps

1 Pick a Kernel then 2 Set Up Cost Function then 3 Optimize Kernel then 4 Do GP Math then 5 Predict Mean and Uncertainty

GP Cost Function Benefit

The GP cost function automatically prevents overfitting because of built-in complexity penalization based on Occams Razor

ANN Pros

Highly flexible and high approximation capability

ANN Cons

Immature theory regarding stochastic properties and function space and no reliable uncertainty bounds

Overfitting Definition

When a model becomes too confident and learns all the noise and random quirks of training data instead of true underlying patterns resulting in failure to generalize

Overfitting Symptoms

Training error keeps decreasing while validation or test error starts increasing

Main Dangers of Overfitting

Poor generalization to unseen data and sensitivity to noise and high instability and misleading metrics that give a false sense of security

Fixes for Overfitting

Get more data or data augmentation or early stopping or simplify the model or use dropout or L2 regularization or feature selection

L2 Regularization

Adds a penalty to the loss function for having overly large weights to restrict weight sizes. Typically only network weights are penalized and not biases

Dropout

A powerful regularization technique that randomly turns off a percentage of neurons during training to force the network to learn robust independent features

NARX What

Nonlinear AutoRegressive with eXogenous inputs. A nonlinear model where current output depends on a nonlinear function of past inputs and past measured noisy outputs

NARX Characteristics

The noise is part of the process dynamics. It is difficult to analyze but easy to estimate because you use actual measured previous outputs during training

NARX Driving Analogy

Driving while looking through the windshield because you check where you actually are in reality every second before making your next guess

NOE What

Nonlinear Output Error. Predicts output from past inputs and its own past noiseless or predicted outputs

NOE Characteristics

Noise is only on the measurements and not the underlying dynamics. It is difficult to analyze and difficult to estimate because errors compound over time

NOE Driving Analogy

Driving blindfolded in your mind where you press the gas pedal and try to imagine where the car is going without looking at the road

NARX vs NOE

They look very alike but the only difference is where the noise enters the system. NOE uses noiseless output for feedback

Nonlinear State Space

Output depends on the instantaneous inputs and unknown states. Difficult to estimate due to unmeasured states but multiple noise structures like process and output noise are possible

Cost Function Definition

A rule or math formula that measures how far off a computers guess is from the correct answer. High cost means wildly wrong and low cost means very accurate

Cost Function Basketball Analogy

Missing a shot by 3 feet gives a penalty score of 3 while making the shot gives a penalty score of 0

Cost Function Importance

Without it machine learning cannot happen because it provides the exact feedback needed for the computer to adjust its settings

Learning Loop

The guess is made then the cost function judges the error then the computer makes an adjustment and the process repeats until the cost drops

Squared L2 Norm

Takes the difference between the modeled and true output and squares the error then takes the mean which is the Mean Squared Error

Gradient Descent Definition

An optimization algorithm used to minimize a cost function by moving in the opposite direction of the gradient which is the direction of steepest increase

Gradient Descent Operational Steps

1 Model makes predictions then 2 Calculate cost then 3 Calculate gradient then 4 Move parameters in opposite direction of gradient then 5 Repeat

Gradient Descent Challenges

Can have difficulty with local minima or saddle points or poorly chosen learning rates

Batch Gradient Descent

Uses the entire training dataset to calculate the gradient before each update. Gives a stable gradient but is extremely slow for large datasets

Stochastic Gradient Descent SGD

Uses one random training example or tiny mini-batch to calculate gradient. Extremely fast and noise can help escape shallow local minima but path is noisy

Mini-Batch Gradient Descent

Uses a small batch of training examples to calculate gradient. Provides a compromise between batch GD and SGD that is computationally efficient

Backpropagation Definition

An algorithm used to calculate the gradient of the cost function with respect to every weight and bias using the chain rule

Backpropagation Purpose

It determines how much each parameter contributed to the final error or assigns blame but it does not update the weights itself

Backpropagation Cake Analogy

If the cake output is too salty backprop goes backwards through the assembly line to find the person responsible for the salt and places blame on them

Backpropagation Steps

1 Forward pass then 2 Calculate error then 3 Backward pass then 4 Gradient descent update

Forward Pass

Input data is passed through the neural network to produce a prediction

Backward Pass

Error is propagated backwards from output layer to input layer using the chain rule to calculate how much each weight affects the final cost

Reverse Mode Backpropagation

Reverse mode is king because you start at the one final defective product or cost score and trace backward to find blame for all suppliers in one single pass

Weights

Define the strength of connection between neurons and control the importance of an input feature by scaling it

Biases

Additional parameters that shift the activation function result up or down to help the model fit the data accurately

Activation Functions

Applied to the weighted sum of inputs at each neuron to introduce nonlinearity. Examples include linear and rectified linear units and sigmoids and radial basis

Why We Need Nonlinearity

Without non-linear activations a multi-layer network is just a simple linear model. Stacking 100 linear layers is mathematically identical to one single linear layer

Folding Paper Analogy

Drawing straight lines on a flat paper cannot make a circle but adding non-linear activation is like folding the paper to make 3D origami shapes

1-Step Ahead Prediction

Predicting the next output using previous inputs and true measured outputs from the real system. Used to evaluate accuracy and measurement feedback prevents error accumulation

1-Step Ahead Simulation

Predicting future outputs where previous predicted outputs are fed back into the model. Harder because errors accumulate over time

Prediction vs Simulation Question

Prediction asks if my model can predict the next point given true history while Simulation asks if my model can generate the future by itself

Estimation Dataset

The dataset specifically used to estimate the parameters of the model

Validation Dataset

Independent dataset used to validate model structure and tune hyperparameters like number of neurons or training iterations

Test Dataset

Independent dataset used for an unbiased final evaluation of model quality that was never involved in any design decisions

Time Series Splitting

Random assignment is not used for time series because we need some length of data to be able to start simulating the model

Early Stopping

Halting the optimization process when the validation cost does not decrease for a set number of iterations to find the sweet spot between bias and variance

Barron's Theorem

Proves that a neural networks error bound is O 1/n and does not care how many inputs you have unlike polynomials which suffer from the curse of dimensionality

Deep Neural Networks

A neural network with an input layer and multiple hidden layers and an output layer

Recurrent Neural Network RNN

A network for sequential data that uses information from previous time steps as a form of memory so it does not completely forget the past

Q-1 Operator

The backward shift operator where Q-1 applied on a signal at time t gives the value at time t minus 1

RNN Hidden State Formula

New memory equals current information plus previous memory which combines the current input with the previous hidden state

LSTM Gates

Includes input gate to decide when to update memory and output gate to decide when to output memory and forget gate to decide when to erase memory

Convolutional Neural Network CNN

Specializes in pattern recognition where filters perform the recognition tasks

Vanishing Gradient Problem

When gradient information becomes tiny when backpropagating through deep networks which prevents effective weight training

Exploding Gradient Problem

When the gradient explodes because the derivative times the weights is greater than 1

Vanishing Gradient Solutions

Changing activation functions to ReLU or data normalization or better initialization or faster hardware

ReLU Advantage

The gradient propagates well because the slope is 1 for all positive values

Data Normalization

Always normalize inputs and outputs to zero-mean and unit variance to improve conditioning and lower risk of vanishing gradients

Data Augmentation

Using system symmetries and transformations that do not impact behavior to create more training data

Steepest Descent Steps

1 Make initial guess with small random values then 2 Compute gradients then 3 Parameter update then 4 Check stopping criterion then 5 Stop

Stopping Criterion

Includes gradient threshold or cost threshold or early stopping

Epsilon Greedy Approach

A strategy to balance exploration and exploitation by picking the random action with a small probability epsilon

Value Function

A formula that estimates how good it is for an agent to be in a given state based on expected future rewards

Q Function

A reinforcement learning algorithm that assigns values to actions based on state without requiring a model of the environment

Bellman Equation

States that the total value of a state equals the immediate reward plus the discounted value of the next state

Optimal Policy

A policy where the value function is larger than or equal to any other policy for all states. Can be retrieved by selecting the action that maximizes the Q function

Q Learning

Off-policy temporal difference control that directly learns the optimal Q function independently of the policy being followed

Q Learning Steps

1 Take Bellman optimality equation then 2 Turn into iterative update then 3 Use transition sample at each step instead of a model

Temporal Difference Learning

Model-free method that learns directly from raw experience and updates knowledge on every timestep by bootstrapping from current estimates

Sarsa

On-policy temporal difference control that learns the value of the policy being carried out including exploration steps

Discounted Return

Concept where future rewards are multiplied by a discount factor over time

Model Free vs Model Based

Model free uses only transition data while model based uses a known transition model P for dynamic programming

Deep Q Network DQN

Combines traditional Q-learning with deep neural networks to approximate future reward values for complex inputs like raw pixels

Policy Gradient

Directly learns the parameters of a policy to ensure maximum rewards. Good for continuous action spaces and stochastic policies

Actor-Critic

The actor tries to learn a policy while the critic learns the value function and minimizes prediction error of the current estimate

Proximal Policy Optimization PPO

Increases stability using a clip function to restrict how much a policy can change during an update to prevent destructive updates

PILCO

An efficient on-policy episodic model-learning approach

RKHS Concept

Reproducing Kernel Hilbert Space is a space of functions where the evaluation of a function at any point can be represented as an inner product with a kernel

MAP Estimation

Maximum A Posteriori is a Bayesian method that estimates a quantity by incorporating a prior distribution to find the most probable value

Noise Robustness

Adding noise to hidden layers increases robustness but introduces bias and links with L2 regularization

MISO vs MIMO

GP extensions to Multiple-Input Multiple-Output cases are possible but Multiple-Input Single-Output is often used

Estimator Definition

A mapping from measured data to a value. If data contains random variables then the estimator is also a random variable

Mean Squared Cost Decomposition

Can be decomposed into contributions from the square of the bias error and the variance error

Automatic Differentiation

Includes forward versus reverse automatic differentiation for gradient calculation

ANN cons theory

Immature theory specifically regarding stochastic properties and function space and representer concept

MDP

Short for Markov Decision Process mentioned in reinforcement learning context

Input layer

The layer that receives raw data into the network

Hidden layers

Layers that perform mathematical computations and feature extraction

Output layer

The layer that generates the final prediction or decision

The candidate gate GRU

In a Gated Recurrent Unit (GRU), (or candidate hidden state) creates a proposed new memory for the current time step.