Midterm Review (Condensed)

Quiz 1 Content

Bias - a parameter that basically acts as an intercept; way to represent prior knowledge
- Can bias estimators any way we want
MLE - Maximum Likelihood Estimation, way of choosing parameters so that the observed data is as likely as possible under the model
- Estimates will increase in variance as observed data increases
- Will converge to true values as size of dataset increases to total population
Naive Bayes - classification based on Bayes theorem
- Assumes feature independence - once you know the class, each feature has equal probability
Noise in dataset - If noise is independently and identically distributed (iid), then increasing the size of the dataset will improve test-set accuracy
- More data helps the model learn the underlying signal
Pseudo Observations - estimates that could occur in the real world that would match probability
- More data = more sure - should match the probability estimates
- Less data = less sure
Gradient descent - optimization algorithm that helps a machine learning model learn by repeatedly adjusting parameters to reduce error
- Trying to minimize the Loss function, aka error
- Uses gradient to do so
- Learning rate - determines how large each update to the parameters is
  - too small = takes too long
  - Too large = overshoots minimum
  - Generally, want to start quick and then slow down as we go, or have an appropriate learning rate
    - Adaptive learning rate
Cross validation - Retraining on different divisions of one dataset to gather statistics on generalization
- Different from batching - process training data in different batches to calculate loss and gradients, then update weights batch by batch; used to perform training updates efficiently and create a generalized model
Regularization - technique to prevent overfitting by assigning a penalty when weights grow too large; regularization techniques try to get weights towards zero
- L1 - Penalty is based on the sum of the absolute weights; encourages some weights to become exactly zero.
- L2 - Penalty is based on the sum of squared weights; shrinks weights towards zero without actually hitting zero
  - works well when many features contribute a little bit to the prediction

Quiz 2 content

Nonlinear activation functions - function applied to a nodes output that allows a neural network to learn complex patterns and relationships in data
- Allow a nonlinear, complex response to data and increase complexity of decision boundaries
He Initialization - method of choosing the initial weights of a neural network so signals and gradients stay at reasonable scales as they pass through many layers
- Samples weights from a normal distribution
- Prevents vanishing or exploding gradients in shallow networks
- Still useful in deeper networks, since signals and gradients are passing through many layers and have even more opportunity to vanish or blow up
Shattered gradient problem - situation where gradients become irregular and uncorrelated through backpropagation through many layers
- Makes network harder to train; lose meaningful structure through nearby inputs
- Occurs when small changes in early layers lead to responses at the ouput that resemble noise; no strong direction to gradient causes SGD to fail
CNN vs LN:
- Linear network - one layer of a neural network, connects neurons to inputs without exploiting spatial structure
  - Arbitrarily wide linear network with ReLU activations has decision boundaries of arbitrary complexity - enough neurons and nonlinear activation functions can produce very complex decision boundaries
  - Number of parameters = (input * output) + output
  - Number of weights = input * output
  - Number of bias = output
- Convolutional Neural Network - neural network designed to find patterns in structured data, especially images
  - Uses small matrices called filters that slide across the input
  - Number of weights = Filter height x filter width x input channels x output channels
  - Number of biases = Number of output channels
  - Total num parameters = weights + biases
  - Receptive field:
    - $r_{new}=r_{old}+\left(k-1\right)j_{old}$
      $r_0=1,j_0=1$
      $j_{new}=s\left(j_{old}\right)$
- CNN’s over LN’s because:
  - Spatial invariance resists overfitting to inconsistent features
  - CN’s can reach the same results with far fewer parameters
  - Each filter get’s a training signal from multiple positions across an input image
Residual Networks: neural networks that use skip connections to make it easier to train very deep networks
- Network learns a change to the input and adds it back to the original input
- Make deep neural networks easier to train - improved gradient flow, easier learning, greater depth
- There is little to no direction between early layers and output layers before resnets
Emission Matrix:
- Present probabilities based on different situations - i.e. fair die vs. unfair die + roll numbers
transition matrix:
- Probability of going from one situation to another, i.e. fair die to fair die, fair die to unfair die, etc.

Midterm Review (Condensed)

Quiz 1 Content

Quiz 2 content

Other Midterm Content