Comprehensive Guide to Practical Statistics for Data Scientists

Taxonomic Classification and Structural Characteristics of Data

Data identifies as either structured or unstructured. Unstructured formats comprise images made of pixels with $RGB$ values, text sequences, or clickstreams. Structured data is primarily categorized into numeric and categorical types. Numeric data consists of continuous values, such as wind speed, or discrete values like counts. Categorical data uses a fixed set of values, including special binary cases ($0$ or $1$) and ordinal data which maintains a specific rank, such as ratings from $1$ to $5$. In software, identifying data as categorical rather than simple text optimizes storage, improves computational performance, and signals how statistical models should process the information.

Rectangular data structures are the standard framework for analysis. These two-dimensional matrices consist of rows, known as records, observations, or samples, and columns, known as features, predictors, or attributes. The outcome or target is the specific feature being predicted. Non-rectangular structures include time series for successive measurements over time, spatial data for mapping coordinates, and graph structures to represent social or physical networks. In statistics, the term graph typically refers to visualizations, whereas in computer science, it refers to the underlying relational data structure.

Estimates of Central Tendency and Variability

Location metrics determine the typical value of a feature. The mean is the sum of values divided by the total count $n$. For a sample, it is denoted as $\bar{x}$. Variations include the trimmed mean, which omits a specific percentage of extreme values from both ends to reduce sensitivity to outliers, and the weighted mean, where each value $x_i$ is multiplied by a weight $w_i$. The median is the middle value of a sorted list, providing a robust estimate resistant to extreme outliers. While statisticians often refer to these as estimates to account for uncertainty, data scientists frequently term them metrics to reflect business objectives.

Variability measures the dispersion of data. The range is the difference between the maximum and minimum values, though it is highly sensitive to outliers. Variance ($s^2$) is the average of squared deviations from the mean, often using $n - 1$ as the denominator for unbiased population estimation. The standard deviation ($s$) is the square root of the variance, expressed on the same scale as the data. Modern statistics often prefers the Median Absolute Deviation ($MAD$) as a robust measurement, calculated as $median(|x_i - m|)$. Percentiles represent the value below which a certain percentage of data falls; the Interquartile Range ($IQR$) is the difference between the $75th$ and $25th$ percentiles.

Exploratory Visualization of Distributions and Relationships

Visual tools reveal the distribution and interaction of variables. A boxplot summarizes quartiles and identifies outliers, typically defined as points beyond $1.5 \times IQR$ from the box edges. Histograms display frequency tables via equal-width bins on the $x$-axis and counts on the $y$-axis. Density plots provide a smoothed version of histograms using kernel density estimation. In statistical theory, location and variability are the first and second moments; skewness (asymmetry) and kurtosis (extreme value propensity) are the third and fourth moments, often identified visually rather than through pure metrics.

For categorical data, bar charts display proportions or frequencies across discrete categories, while pie charts are generally less favored by experts. The mode is the most frequent category. Expected value is a weighted mean used when categories are mapped to numeric values, often involving probabilities ($EV = \sum P(x) \times x$). In a business context, this is fundamental for capital budgeting, such as calculating the potential value of webinar attendees based on sign-up rates like $5\%$ or $15\%$.

Correlation analysis examines the association between two variables. Pearson's correlation coefficient ($r$) ranges from $-1.0$ (perfect negative) to $+1.0$ (perfect positive), where $0$ indicates no linear relationship. Scatterplots are the standard bivariate visualization. For large data sets, hexagonal binning or contour plots avoid over-density by grouping records. Multivariate analysis also utilizes contingency tables for categorical counts and violin plots, which combine boxplots with density estimates to show distribution nuances.

Sampling Theory and Resulting Distributions

Sampling is the process of selecting a subset from a population. Random sampling ensures every member has an equal chance of selection to avoid sample bias, such as the historic failure of the 1936 Literary Digest poll. Stratified sampling divides populations into groups to ensure representativeness. Selection bias can occur through data snooping or the vast search effect, where repeated modeling leads to illusory patterns. Regression to the mean is a phenomenon where extreme observations are naturally followed by more central ones due to the waning of luck or random error.

Measuring the variability of a statistic across multiple samples leads to the sampling distribution. The Central Limit Theorem states that as sample size $n$ increases, the sampling distribution of the mean will become normally distributed, even if the underlying population is not. The Standard Error ($SE$) is the standard deviation of this sampling distribution, estimated as $SE = \frac{s}{\sqrt{n}}$. To estimate uncertainty without relying on theoretical assumptions, the bootstrap method involves repeatedly resampling the observed data with replacement. Confidence intervals provide a range around an estimate that covers the central percentage of the bootstrap distribution, such as $90\%$ or $95\%$.

Key probability distributions provide mathematical archetypes for data. The Normal (Gaussian) distribution is defined by its bell shape; the standard normal has a mean of $0$ and a standard deviation of $1$, where values are expressed as $z$-scores. The Student's t-distribution serves as a similar reference but with thicker tails to account for smaller samples. Binary outcomes follow a Binomial distribution. Random events occurring at a constant rate are modeled by the Poisson distribution ($\lambda$), while the time between these events follows an Exponential distribution. When the event rate changes over time, as in mechanical failure, the Weibull distribution is applied.

Statistical Experiments and Significance Testing

A/B testing compares a treatment group to a control group to establish superiority, typically using a single predetermined test statistic like conversion rate. Blinding prevents subject or investigator awareness from biasing the results. Hypothesis testing starts with a null hypothesis ($H_0$), assuming any observed effect is due to chance. The p-value is the probability that a chance model would produce results as extreme as those observed. If this value is below a threshold, known as alpha ($\alpha$), typically set at $0.05$, the result is deemed statistically significant. Testing can be one-way (directional) or two-way (testing for any difference).

Permutation tests are a resampling approach to significance, involving the repeated shuffling and reallotting of group data to build a null distribution of the difference. Analysis of Variance ($ANOVA$) extends this to multiple groups, using the F-statistic to compare variance between groups to variance within groups. For categorical count data, the Chi-Square test ($\chi^2$) measures the departure of observed counts from those expected under the null hypothesis of independence. Fisher's Exact test is used for very small counts where the Chi-Square approximation fails.

Multi-arm bandit algorithms optimize experiments in real-time, shifting resources toward more successful treatments during the test. The epsilon-greedy algorithm is a simple version where a small percentage ($\epsilon$) of subjects are randomly assigned while the majority receives the best-performing option. Calculating the required sample size for an experiment involves effect size, power (probability of detecting an effect), and the significance level. Higher power and smaller effect sizes require larger samples.

Linear Regression and Predictive Modeling

Simple linear regression models the relationship between a response ($Y$) and a predictor ($X$) with the equation $Y = b_0 + b_1X + e$. The intercept ($b_0$) and slope ($b_1$) are estimated by minimizing the residual sum of squares ($RSS$), a method called Ordinary Least Squares ($OLS$). Multiple linear regression incorporates multiple predictors. Model performance is evaluated using Root Mean Squared Error ($RMSE$), measuring average prediction error, and R-squared ($R^2$), the proportion of variance explained.

Model selection aims for parsimony through methods like stepwise regression, which adds or drops variables to optimize the Akaike Information Criterion ($AIC$). Penalized regression, such as Ridge or Lasso, adds constraints to coefficients to prevent overfitting. Factor variables are incorporated into regression via dummy coding, typically using $P - 1$ binary variables for $P$ levels to avoid multicollinearity. When predictors are highly correlated, individual coefficients become unstable and difficult to interpret.

Regression diagnostics include the analysis of residuals to check for heteroskedasticity (non-constant variance) or non-normality. Influential values are identified using Cook's distance or leverage (hat-values). For nonlinear relationships, polynomial regression adds squared or cubed terms, while splines use piecewise polynomials connected at knots to create smooth curves. Generalized Additive Models ($GAM$) automate this spline selection.

Advanced Classification Techniques

Classification predicts categorical outcomes or propensity scores. Naive Bayes assumes independent predictors and uses conditional probabilities ($P(X|Y)$) to estimate the posterior probability of a class ($P(Y|X)$). Linear Discriminant Analysis ($LDA$) uses a covariance matrix to maximize group separation. Logistic regression is a generalized linear model ($GLM$) that maps a linear combination of predictors to a probability between $0$ and $1$ using the logit (log-odds) link function: $logit(p) = \ln(\frac{p}{1-p})$.

Model evaluation uses the confusion matrix to calculate accuracy, precision (correctly predicted positives), recall/sensitivity (proportion of actual positives identified), and specificity (proportion of actual negatives identified). The ROC curve plots sensitivity against specificity across various cutoffs; the Area Under the Curve ($AUC$) summarizes the overall effectiveness, where $1.0$ is a perfect classifier and $0.5$ is random. For imbalanced data where 1s are rare, models are optimized using undersampling, oversampling, or synthetic data generation ($SMOTE$).

Statistical Machine Learning and Unsupervised Techniques

K-Nearest Neighbors ($KNN$) identifies the $K$ most similar records to a target to determine its classification or average. This requires standardization of variables ($z$-scores) so that scale does not dominate distance metrics like Euclidean or Manhattan distance. Tree models use recursive partitioning to divide data based on split values that minimize impurity using Gini or Entropy measures. Trees are often pruned back to avoid overfitting.

Ensemble methods combine many models for higher accuracy. Bagging (Bootstrap Aggregating) fits models to bootstrap samples. Random Forests add variable sampling at each split to create diverse trees and provide variable importance scores. Boosting, such as $XGBoost$, fits a sequence of models where each corrects the errors of the previous one. Boosting is highly prone to overfitting and requires careful tuning of hyperparameters like $eta$ and $subsample$, often via cross-validation.

Unsupervised learning extracts patterns without labeled outcomes. Principal Components Analysis ($PCA$) reduces dimensionality by creating linear combinations of predictors that explain the most variance, visualized via screeplots. K-Means clustering divides data into $K$ groups by minimizing the distance to cluster centroids, with the elbow method used to select the optimal $K$. Hierarchical clustering builds a dendrogram by iteratively merging similar records or clusters using linkage methods like average or complete. Model-based clustering assumes data layers are mixtures of multivariate normal distributions and uses the Bayesian Information Criteria ($BIC$) to optimize the model.