ISYE 6501 - Intro to Data Analytics

Midterm 2

Overfitting

Model fits noise; low training error, high validation error

Underfitting

Model too simple; high training and validation error

Bias

Systematic error from overly simple model

Variance

Model sensitivity to data; high variance = overfitting

Forward Selection

Start with no variables, add significant ones (p ≤ threshold)

Backward Elimination

Start with all variables, remove insignificant ones (p > threshold)

Stepwise Regression

Combination of forward and backward selection

Lasso Regression

Sets some coefficients exactly to 0 (feature selection)

Ridge Regression

Shrinks coefficients to reduce variance (no variable removal)

Elastic Net

Combination of Lasso and Ridge penalties

Regularization Effect

Decreasing regularization → lower bias, higher variance

A/B Testing

Compare two alternatives using binary outcomes

Balanced Design

Each option tested equally often

Factorial Design

Test combinations of multiple factors

Fractional Factorial Design

Subset of combinations; faster but may confound effects

Exploration

Testing options to gain information

Exploitation

Using best-known option for immediate reward

Multi-Armed Bandit

Balances exploration and exploitation

Blocking

Control unwanted variation by grouping similar units

Bernoulli Distribution

Single trial with success probability p

Binomial Distribution

Number of successes in n independent trials

Geometric Distribution

Number of failures before first success

Poisson Distribution

Number of events in a fixed interval

Exponential Distribution

Time between events

Weibull Distribution

Time to failure (flexible hazard rate)

Poisson vs Exponential

Poisson = counts; Exponential = time between events

Memoryless Property

Future independent of past (Exponential, Geometric only)

Q-Q Plot

Straight line = data matches distribution; curve = mismatch

Deterministic Simulation

No randomness; same input → same output

Stochastic Simulation

Includes randomness

Discrete-Event Simulation

Changes occur at discrete times

Simulation Replications

Run multiple times to capture variability

Simulation Quality

Depends on accuracy of input data

Markov Chain

Future depends only on current state (memoryless)

Transition Probability (Pij)

Probability of moving from state i to j

Missing Data First Step

Check for patterns in missingness

Missing Data Options

Remove, indicator variable, or impute

Mean Imputation

Replace missing with mean; introduces bias

Regression Imputation

Predict missing values; risk of overfitting

Perturbation

Add noise to imputed values to restore variability

Imputation Risk

Using same data twice can cause overfitting

Optimization Variables

Decisions to be made

Optimization Constraints

Restrictions on variables

Objective Function

Measure to maximize or minimize

Feasible Solution

Satisfies all constraints

Optimal Solution

Best feasible solution

Convex Optimization

Guarantees global optimum

Non-Convex Optimization

May have multiple local optima

Integer Programming

Variables restricted to integers; harder to solve

Optimization Difficulty Order

Linear < Convex < Integer < Non-convex

Heuristic Algorithm

Fast, good solution but not guaranteed optimal

Nonparametric Tests

Do not assume distribution; use ranks

McNemar Test

Paired categorical (Yes/No)

Wilcoxon Signed-Rank Test

Paired numeric data

Mann-Whitney Test

Independent numeric samples

Parametric vs Nonparametric

Parametric uses means; nonparametric uses ranks

Bayes Theorem

P(A|B)=P(B|A)P(A)/P(B)

Empirical Bayes

Combines individual and population data

Graph Nodes

Entities in network

Graph Edges

Connections between nodes

Clique

Fully connected group of nodes

Community

Tightly connected group

Louvain Algorithm

Detects communities by maximizing modularity

Modularity

Measures strength of network clustering

Neural Network Layers

Input, hidden, output layers

Deep Learning

Neural networks with many hidden layers

Descriptive Model

Describes data

Predictive Model

Predicts future outcomes

Prescriptive Model

Recommends decisions

Game Theory

Study of strategic decision-making

Equilibrium

No player benefits from changing strategy

Zero-Sum Game

One gain equals another’s loss

Non-Zero-Sum Game

Total payoff can vary

Survival Model

Predicts time until event occurs

Cox Proportional Hazards Model

h(t)=h0(t)e^(βx)

Hazard Rate

Risk of event occurring at time t

Censored Data

Incomplete information about event timing

Gradient Boosting

Sequentially builds models to correct errors

Cross Validation

Used to estimate model performance

Validation Set

Used to compare models

Test Set

Used to estimate final performance

Training Set

Used to fit model

Better Model Selection

Lower validation error + fewer variables preferred

SVM Margin

Wider margin → better generalization

Overfitting SVM

Narrow margin, complex boundary

k-Nearest Neighbors

Classifies based on nearby points

k-Means Clustering

Groups data into clusters based on distance

Principal Component Analysis

Reduces dimensionality via variance

Logistic Regression

Used for classification (binary outcomes)

Linear Regression

Used for predicting numeric response

Time Series Models

Use temporal data (ARIMA, Exponential Smoothing)

Attribute Data Models

Use feature-based data

Confusion Matrix

Shows classification performance

Threshold (Logistic)

Cutoff for classification decision

Lower Threshold

More positives, fewer false negatives

Higher Threshold

Fewer positives, more false negatives