ML, AI, DS, Algorithms, Interview Prep

Vector

Ordered collection of numbers representing features or directions.

Matrix

Rectangular array of values used for linear transformations and batched computations.

Dot product

Sum of pairwise products; measures alignment and computes linear model scores.

Eigenvector and eigenvalue

A direction unchanged by a matrix transformation except for scaling; the scale is its eigenvalue.

Gradient

Vector of partial derivatives pointing toward steepest local increase of a function.

Chain rule

Computes derivatives of compositions; foundational to backpropagation.

Bayes' theorem

P(A|B)=P(B|A)P(A)/P(B); updates prior beliefs using evidence.

Conditional probability

Probability of A given B: P(A∩B)/P(B), when P(B)>0.

Expected value

Probability-weighted average of possible outcomes.

Variance

Expected squared deviation from the mean; measures dispersion.

Covariance

Measures how two variables vary together; scale-dependent.

Correlation

Normalized covariance, typically between -1 and 1.

Central limit theorem

Under suitable conditions, normalized sample means approach a normal distribution as sample size grows.

Confidence interval

Interval procedure that captures the fixed population parameter at a specified long-run frequency.

p-value

Probability, assuming the null hypothesis and model, of results at least as extreme as observed.

Type I vs Type II error

Type I: reject a true null; Type II: fail to reject a false null.

Maximum likelihood estimation

Choose parameters maximizing the probability or density of observed data.

Gradient descent

Iteratively update parameters opposite the gradient: θ←θ−η∇L.

Learning rate

Step size of parameter updates; too high can diverge, too low can train slowly.

Feature scaling

Rescale numeric features to comparable magnitudes, often important for SVMs and gradient-based models.

Standardization

Transform using (x−training mean)/training standard deviation; fit statistics on training data only.

One-hot encoding

Represent categories with indicator columns; handle unseen categories explicitly.

Data leakage

Information unavailable at prediction time contaminates training or evaluation and inflates measured performance.

Train/validation/test split

Train fits parameters, validation selects settings, test estimates final generalization.

Stratified sampling

Preserve approximate class proportions across splits.

Cross-validation

Repeatedly fit and evaluate across folds to estimate performance and tune models.

Missing data mechanisms

MCAR: unrelated to values; MAR: explained by observed data; MNAR: depends on unobserved/missing values.

Class imbalance

Unequal label frequencies; use suitable metrics, stratification, weighting, and threshold selection.

Supervised learning

Learn from labeled input-output examples for classification or regression.

Unsupervised learning

Find patterns or structure in unlabeled data.

Regression vs classification

Regression predicts continuous values; classification predicts categories.

Linear regression

Predicts continuous outcomes as weighted feature sums, often minimizing squared error.

Logistic regression

Linear log-odds model using sigmoid for binary class probabilities.

L1 regularization

Penalizes sum of absolute coefficients; can yield sparse weights.

L2 regularization

Penalizes squared coefficient magnitudes; shrinks weights.

Bias-variance tradeoff

Too-simple models underfit; overly flexible models may overfit; regularization and data affect both.

Decision tree

Recursively partitions feature space using split criteria such as Gini impurity.

Random forest

Bagged ensemble of randomized trees that reduces variance relative to a single tree.

Gradient boosting

Sequentially adds weak learners to improve an objective using gradient information.

XGBoost

Optimized gradient-boosted tree library with regularization and efficient split finding.

Support vector machine

Classifier maximizing margin; kernels enable nonlinear decision boundaries.

RBF kernel

Similarity exp(−γ||x−x'||²); enables nonlinear SVM boundaries.

k-nearest neighbors

Predict using labels or values of nearby training samples; sensitive to distance scaling.

Naive Bayes

Probabilistic classifier assuming conditional independence of features given class.

K-means

Partitions observations into k clusters by minimizing within-cluster squared distances.

DBSCAN

Density-based clustering that identifies dense groups and marks sparse points as noise.

PCA

Finds orthogonal directions of maximal variance for dimensionality reduction.

t-SNE vs UMAP

Nonlinear visualization techniques; local neighborhood structure matters, and plot distances need caution.

ROC AUC

Probability a randomly selected positive scores higher than a randomly selected negative, with tie adjustment.

Precision

TP/(TP+FP): proportion of predicted positives that are correct.

Recall

TP/(TP+FN): proportion of actual positives detected.

F1 score

Harmonic mean of precision and recall: 2PR/(P+R).

PR AUC

Summarizes precision-recall tradeoff; often informative for rare positive classes.

Calibration

Agreement between predicted probabilities and observed outcome frequencies.

Threshold tuning

Choose decision cutoff using business costs and validation data; ranking AUC alone does not select a threshold.

Confusion matrix

Counts true positives, false positives, true negatives, and false negatives.

Hyperparameter tuning

Search settings such as depth, regularization, or learning rate without using the test set.

Early stopping

Stop training when validation performance stops improving to limit overfitting.

Stacking ensemble

Train a meta-model on out-of-fold predictions of base models to avoid leakage.

Permutation importance

Measure performance degradation after shuffling a feature; correlated features complicate interpretation.

SHAP

Shapley-based feature attribution method; explanations are model-relative, not automatically causal.

Perceptron

Single neuron computing an activation of a weighted sum plus bias.

MLP

Feedforward neural network with one or more hidden layers.

ReLU

Activation max(0,x), common in deep networks.

Sigmoid

Maps real values to (0,1), often used for binary output probabilities.

Softmax

Converts logits into normalized class probabilities.

Cross-entropy loss

Penalizes low predicted probability for the true class.

Backpropagation

Efficiently computes parameter gradients through a computation graph via chain rule.

Adam optimizer

Adaptive first- and second-moment gradient optimizer.

Batch normalization

Normalizes intermediate activations using batch statistics and learned scale/shift.

Layer normalization

Normalizes across features within each example; common in transformers.

Dropout

Randomly masks activations during training to regularize.

CNN

Neural network using shared convolutional filters to exploit spatial structure.

RNN vs LSTM

RNN processes sequences recurrently; LSTM gates help retain long-range information.

Attention

Computes weighted combinations of values based on query-key similarity.

Self-attention

Tokens attend to other tokens in the same sequence.

Transformer

Architecture built around attention, feedforward blocks, residual connections, and normalization.

Positional encoding

Provides token-order information to otherwise order-agnostic attention operations.

Tokenization

Splits text into model input units such as subwords; token counts differ from word counts.

Embedding

Dense vector representation of an item, token, or passage.

Cosine similarity

Dot product divided by vector norms; compares vector direction.

Pretraining

Learn broad representations from large-scale data before downstream adaptation.

Fine-tuning

Update pretrained model weights on task-specific data.

LoRA

Parameter-efficient adaptation using trainable low-rank updates to selected weights.

Instruction tuning

Train a model to follow prompt-response instructions.

RLHF

Use human preference signals and reinforcement-learning-style optimization to align outputs.

DPO

Direct preference optimization of a language model using preference pairs without an explicit reward-model RL loop.

RAG

Retrieve external documents and condition generation on retrieved evidence.

Vector database

Stores embeddings and supports similarity or approximate-nearest-neighbor search.

Chunking

Split documents into retrievable pieces; size and overlap affect recall and context quality.

Hybrid retrieval

Combine lexical matching such as BM25 with semantic vector retrieval.

Reranker

Reorders retrieved candidates with a more expensive relevance model.

Hallucination

Model output that is unsupported or false despite sounding plausible.

Prompt injection

Untrusted content attempts to override system or developer instructions; enforce trust boundaries.

Tool calling

Model selects structured tool actions; application validates permissions, arguments, and outputs.

AI agent

System that plans and invokes tools across steps toward a goal, with guardrails and stopping criteria.

Context window

Maximum amount of input/output token context supported by a model invocation.

Temperature

Sampling control affecting output randomness; does not guarantee factuality.

Top-p sampling

Samples from smallest token set whose cumulative probability reaches p.

Quantization

Represent weights/activations at lower precision to reduce memory and often accelerate inference.