Statistical Learning, Linear Models, and Non-Linear Decision Theory

Terminology and Notation in Statistical Learning

• Supervised Learning Fundamentals:

  • Supervised learning is a paradigm where a predictive model is trained using observed inputs and desired outputs. The central objective is to use input variables $X$ to predict target output variables $Y$.

  • Inputs: Also referred to as features, predictors, or independent variables.

  • Outputs: Also referred to as responses or dependent variables.

  • Variable Types:

  • In most practical statistical learning scenarios, inputs are quantitative (i.e., continuous or real-valued numbers $\boldsymbol{x} \in \mathbb{R}^p$).

  • Outputs can be quantitative (e.g., in regression problems where $Y \in \mathbb{R}$) or qualitative/categorical (e.g., in classification problems where output $G \in \mathcal{G}$).

  • Target Encoding:

  • Qualitative output variables are typically encoded into numeric representations for computational convenience. For example, binary outcomes such as "success" versus "failure" are mapped to $1$ and $0$, respectively.

  • Numeric code values representing categories are formally termed targets or labels.

• Comprehensive Mathematical Notation Guide:

  • $X$: The generic input variable representing a vector of features.

  • $X_j$: The $j$-th component variable or feature of $X$

  • $x_i$: The $i$-th observed instance of $X$, structured as a $p$-vector.

  • $x_{ij}$: The specific value of feature $j$ within the $i$-th observed sample vector $x_i$

  • $Y$: Quantitative output variable ($Y \in \mathbb{R}$).

  • $G$: Qualitative or categorical output variable ($G \in \mathcal{G}$).

  • $\mathbf{X}$: An $N \times p$ feature matrix containing $N$ observed sample vectors $x_i$ for $i = 1, \dots, N$

  • $\mathbf{x}_j$: An $N$-vector column within matrix $\mathbf{X}$ comprising all $N$ observed measurements for feature variable $X_j$

  • $x_i^T$: The $i$-th row of matrix $\mathbf{X}$ (written as a row vector, since all standalone vectors default to column vectors).

  • $\hat{Y}$: The model's predicted quantitative output value ($\hat{Y} \in \mathbb{R}$).

  • $\hat{G}$: The model's predicted categorical class label ($\hat{G} \in \mathcal{G}$).

• Indexing and Boldface Conventions:

  • Index $i$ is strictly used for rows (sample observations ranging from $1$ to $N$).

  • Index $j$ is strictly used for columns (input features ranging from $1$ to $p$).

  • Boldface Rule: Bold typography is reserved exclusively for matrices (e.g., $\mathbf{X}$) and $N$-vectors representing entire feature columns (e.g., $\mathbf{x}_j$). Boldface is not applied to individual observation $p$-vectors (such as $x_i$).

  • Training Set Representation: A dataset comprising $N$ observations used for training is expressed as $\mathcal{T} = \{(x_i, y_i)\}_{i=1}^N$

Linear Regression Models

• Formulation of the Linear Model:

  • Linear regression models form a foundational baseline for statistical learning.

• Two-Feature Model Formulation:

  • Given an input dataset with $p = 2$ features $X = (X_1, X_2)$, the model estimates scalar coefficients $\hat{\beta} = (\hat{\beta}_0, \hat{\beta}_1, \hat{\beta}_2)$ to predict values:       $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_{i1} + \hat{\beta}_2 x_{i2}, \quad \text{where } \hat{\beta}_j, x_{ij}, \hat{y}_i \in \mathbb{R}$

• Generalization to $p$ Features:

  • For an arbitrary number of features $p$, the prediction formula for observation $i$ is:       $\hat{y}_i = \hat{\beta}_0 + \sum_{j=1}^p \hat{\beta}_j x_{ij}, \quad \text{for } i = 1, \dots, N$

• Generic Variable Form:

  • Dropping the specific observation index $i$ yields the generic equation over continuous input spaces:       $\hat{Y} = \hat{\beta}_0 + \sum_{j=1}^p X_j \hat{\beta}_j, \quad \text{where } \hat{\beta}_j, X_j, \hat{Y} \in \mathbb{R}$

  • Vectorized Representation and Bias Vector Augmentation:

• Using explicit vector inner products, the equation simplifies to:     $\hat{Y} = X^T \hat{\beta} + \hat{\beta}_0, \quad \text{where } X, \hat{\beta} \in \mathbb{R}^p, \text{ and } \hat{\beta}_0, \hat{Y} \in \mathbb{R}$

• Constant Augmentation Trick:

  • By appending a constant term $1$ to the feature vector $X$ (making $X \in \mathbb{R}^{p+1}$), the intercept $\hat{\beta}_0$ is seamlessly folded into the coefficient vector $\hat{\beta}$:       $\hat{Y} = X^T \hat{\beta}$

  • This compact vector form simplifies underlying mathematics and mirrors exact computational implementations.

• Python Computational Implementation:

  • Numerical linear algebra packages execute matrix operations efficiently via matrix multiplication operators or dot products:

import numpy as np

# X: shape (p,), beta_hat: shape (p,)
Y_hat = X.T @ beta_hat  # returns scalar

import numpy as np

# X: shape (p,), beta_hat: shape (p,)
Y_hat = np.dot(X.T, beta_hat)  # returns scalar

• Extension to Multiple Outputs ($K$-Vector Response):

  • When predicting $K$ distinct targets simultaneously ($\hat{Y} \in \mathbb{R}^K$), the model expands to:     $\hat{Y} = X^T \hat{\beta}, \quad \text{where } \hat{\beta} \in \mathbb{R}^{p \times K}, X \in \mathbb{R}^p, \hat{Y} \in \mathbb{R}^K$

  • Each individual output variable within $\hat{Y}$ requires its own dedicated set of $p$ parameter coefficients. Otherwise, identical predictions would be generated across all $K$ outputs.

Geometric Interpretation and Fitting of Linear Models

• Geometric Interpretations:

  • One Feature ($p=1$, $\hat{Y} \in \mathbb{R}$):

  • The system $X^T \hat{\beta} = [x \quad 1] [\hat{b} \quad \hat{b}_0]^T = \hat{Y}$ defines a straight line in two-dimensional space $(X_1, Y)$.

  • Two Features ($p=2$, $\hat{Y} \in \mathbb{R}$):

  • The system forms a flat plane residing in three-dimensional space $(X_1, X_2, Y)$.

  • General Case ($p$ Features, $\hat{Y} \in \mathbb{R}$):

  • The model produces a $p$-dimensional hyperplane embedded within a $(p+1)$-dimensional coordinate space.

  • Multiple Target Outputs ($\hat{Y} \in \mathbb{R}^K$):

  • The overall space remains $(p+1)$-dimensional, containing $K$ distinct hyperplanes. Each input instance $x_i$ maps to $K$ independent values across these planes.

• Least Squares Fitting on Training Data:

  • Given a design matrix $\mathbf{X} \in \mathbb{R}^{N \times p}$ representing $N$ training observations, the prediction vector across all training instances is $\mathbf{\hat{Y}} = \mathbf{X} \hat{\beta}$.

• The objective is finding optimal parameters $\hat{\beta}$ making predicted outputs $\mathbf{\hat{Y}}$ as close to observed targets $\mathbf{Y}$ as possible.

• Residual Sum of Squares (RSS):     $RSS(\beta) = \sum_{i=1}^N (y_i - x_i^T \beta)^2$

• The estimation process minimizing RSS is termed the method of least squares.

  • Analytical Derivation of the Least Squares Parameter Vector:

• Express RSS in matrix-vector notation:      $RSS(\beta) = \sum_{i=1}^N (y_i - x_i^T \beta)^2 = \sum_{i=1}^N (y_i - \hat{y}_i)^2 = (\mathbf{y} - \mathbf{\hat{y}})^T (\mathbf{y} - \mathbf{\hat{y}}) = (\mathbf{y} - \mathbf{X}\beta)^T (\mathbf{y} - \mathbf{X}\beta)$

• Differentiate $RSS(\beta)$ with respect to vector $\beta$ and set derivative equal to zero:      $\frac{dRSS}{d\beta} = \mathbf{X}^T (\mathbf{y} - \mathbf{X}\beta) = 0$

• Solve for $\hat{\beta}$:      $\mathbf{X}^T \mathbf{y} - \mathbf{X}^T \mathbf{X} \beta = 0 \implies \mathbf{X}^T \mathbf{X} \beta = \mathbf{X}^T \mathbf{y} \implies \hat{\beta} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$

  • Classification via Linear Regression Thresholding:

• Consider a dataset of 100 sample points with two feature dimensions $X = (X_1, X_2)$ and a binary qualitative outcome $G \in \{\text{Blue}, \text{Orange}\}$, encoded numerically as Blue = 0 and Orange = 1.

• Fitting a linear regression model yields continuous estimates $\hat{Y}$, which are transformed into categorical predictions $\hat{G}$ using a threshold decision rule at $0.5$:     $\hat{G} = \begin{cases} \text{Orange} & \text{if } \hat{Y} > 0.5 \\ \text{Blue} & \text{if } \hat{Y} \le 0.5 \end{cases}$

• The resulting decision boundary in feature space is linear, defined precisely by the line $x^T \hat{\beta} = 0.5$

Non-Linear Models: $k$-Nearest Neighbor ($k\text{NN}$) Methods

• Core Principles:

  • Nearest-neighbor methods estimate output $\hat{Y}$ at target point $x$ by averaging response values of training observations located closest to $x$ in input feature space.

  • Formal Definition of $k\text{NN}$:     $\hat{Y}(x) = \frac{1}{k} \sum_{x_i \in N_k(x)} y_i$     where $N_k(x)$ specifies the neighborhood surrounding point $x$, defined by the $k$ nearest training instances $x_i$

• Classification Decision Rule:

  • Given threshold $\theta$ (typically $\theta = 0.5$ for binary $0/1$ outcomes):     $\hat{g}(x) = \begin{cases} \text{Orange} & \text{if } \hat{y}(x) > \theta \\ \text{Blue} & \text{if } \hat{y}(x) \le \theta \end{cases}$

• Distance Metric:

  • Closeness is evaluated using Euclidean distance standard metric:     $\operatorname{EuD}(x_1, x_2) = \|x_1 - x_2\|_2 = \left( \sum_{j=1}^p (x_{1j} - x_{2j})^2 \right)^{1/2}$

• Behavior Under Different Values of $k$:

  • $15\text{-NN}$ Classifier:

  • Determines class membership via majority vote across the 15 nearest neighboring points.

  • Generates a smooth, non-linear decision boundary.

• $1\text{-NN}$ Classifier:

  • Sets prediction $\hat{Y}$ equal to the exact label $y_\ell$ of the single nearest point $x_\ell$ to $x$

  • Creates a highly complex, non-linear decision boundary containing enclosed decision islands.

  • Training Error Rate: Exactly $0\%$ misclassifications on training data. Because every training sample instance $x_i$ is its own closest neighbor ($x_i \in N_1(x_i)$), the model evaluates $\hat{y}(x_i) = y_i$, yielding zero prediction errors during training.

Comparison: Linear vs. Non-Linear ($k\text{NN}$) Models

• Model Flexibility, Complexity, and Trade-offs:

  • Linear Model Properties:

  • Makes higher misclassifications on non-linearly structured training data.

  • Highly rigid parameterization. Represented compactly by only $p+1$ scalar values (parameters $\beta \in \mathbb{R}^{p+1}$).

  • Low model complexity yields strong stability and low variance when evaluating unseen test data.

  • $1\text{-NN}$ Model Properties:

  • Perfectly classifies training data ($0\% error rate).\n - Controlled by a single global hyperparameter k \in \mathbb{N}$, but structurally relies on calculating local neighborhood averages across$\frac{N}{k} local regions.\n - Extremely fragile and highly sensitive to training data noise (high variance), leading to severe overfitting on new data points.\n\n- **Synthetic Data Benchmark Setup**:\n - Synthetic simulation demonstrating non-linear cluster overlap:\n - Total sample size: 200 observations (100 Blue, 100 Orange).\n - Blue class centers: 10 mean vectors m_k$generated from bivariate Gaussian distribution$\mathcal{N}\left( (1,0)^T, \mathbf{I} \right).\n - Orange class centers: 10 mean vectors m_k generated from bivariate Gaussian distribution $ NORTH \mathcal{N}\left( (0,1)^T, \mathbf{I} \right).

  • Individual sample generation: Select cluster mean $m_k$ uniformly at random with probability $0.1$, then sample data point from $\mathcal{N}\left( m_k, \frac{\mathbf{I}}{5} \right)$.

  • This generative process produces overlapping multi-modal clusters that linear decision boundaries cannot separate cleanly.

Statistical Decision Theory Framework

• Theoretical Framework Principles:

  • Let $X \in \mathbb{R}^p$ denote a real-valued random input vector, and $Y \in \mathbb{R}$ represent a real-valued target random variable, governed by joint probability distribution $\operatorname{Pr}(X, Y)$.

  • The goal is finding a prediction function $f(X)$ mapping inputs to predicted targets $Y$

  • Loss Function:

  • Quantifies penalization incurred by prediction errors. Using squared error loss:       $L(Y, f(X)) = (Y - f(X))^2$

• Mathematical Foundations of Expectation:

  • Expectation of continuous variable $X$: $E(X) = \int_x x \operatorname{Pr}(x) dx = \int_x x \operatorname{Pr}(dx)$

  • Expectation of function $g(X, Y)$:

  • Discrete: $E(g(X, Y)) = \sum_x \sum_y g(x, y) \operatorname{Pr}(x, y)$

  • Continuous: $E(g(X, Y)) = \int_x \int_y g(x, y) \operatorname{Pr}(x, y) dx dy$

  • In decision theory, the evaluated function is squared loss $g(X, Y) = (Y - f(X))^2$

• Expected Prediction Error (EPE):

  • Objective criterion for selecting optimal function $f$:     $EPE(f) = E\left[ (Y - f(X))^2 \right] = \int_x \int_y (y - f(x))^2 \operatorname{Pr}(x, y) dx dy = \int (y - f(x))^2 \operatorname{Pr}(dx, dy)$

  • Applying conditional probability factorization $\operatorname{Pr}(X, Y) = \operatorname{Pr}(Y|X) \operatorname{Pr}(X)$, EPE rewrites as:     $EPE(f) = E_X E_{Y|X} \left[ (Y - f(X))^2 \mid X \right]$

  • The inner expectation $E_{Y|X}$ calculates expected squared error over all outcomes conditional on a specific input $X$. The outer expectation $E_X$ averages these errors across input space $\mathbb{R}^p$

• Pointwise Minimization and the Regression Function:

  • To minimize total $EPE(f)$, it suffices to minimize expected error conditional on every individual input value $X = x$:     $f(x) = \operatorname{argmin}_c E_{Y|X} \left[ (Y - c)^2 \mid X = x \right]$     where $c$ represents a fixed predicted scalar value.

  • The condition $X = x$ does not isolate a single training instance; rather, it refers to all possible population instances sharing feature values $x$. For a given $x$, target $Y$ remains a random variable.

  • The theoretical minimizer $c$ of expected squared error $E(Y - c)^2$ is the conditional expectation $c = E(Y)$. Thus, the optimal theoretical target solution is:     $f(x) = E_{Y|X}(Y \mid X = x)$     which is formally termed the regression function.

• Conceptual Application (Real Estate Evaluation):

  • Consider predicting home values $Y$ using features $X_1$ (bedroom count), $X_2$ (square footage), and $X_3$ (neighborhood quality rating).

  • The regression function $f(x) = E_{Y|X}(Y \mid X = x)$ does not represent the average price of a single home, nor does it mean average price across the full housing market.

  • It equals the population average price across all homes possessing identical features $x = (x_1, x_2, x_3)$.

• Theoretical Target vs. Empirical Reality:

  • In practical applications, the true joint probability distribution $\operatorname{Pr}(X, Y)$ is unknown.

  • Continuous input features mean exact feature matches $X = x$ appear at most once in finite datasets. Evaluating empirical expectations over a single observation yields no smoothing capability.

  • Parametric and non-parametric models represent practical computational attempts to estimate $\hat{f}(x) \approx E(Y \mid X = x)$ using finite data.

• Model Alignment with Decision Theory:

  • Linear Regression: Assumes $f(x)$ is approximated by a global linear structure $f(x) \approx x^T \beta$. Solves $\beta$ by substituting this structure into the theoretical EPE equation.

  • $k\text{-Nearest Neighbors}$: Relaxes conditioning at an exact point $X = x$ to conditioning over a localized neighborhood region $N_k(x)$, replacing theoretical population expectations with sample arithmetic averages:     $\hat{f}(x) = \operatorname{Ave}\left( y_i \mid x_i \in N_k(x) \right)$     Assumes $f(x)$ is approximated locally by a constant function.

• Asymptotic Convergence of $k\text{NN}$:

  • As sample size $N \to \infty$ and neighborhood size $k \to \infty$ such that $\frac{k}{N} \to 0$:     $\hat{f}(x) \xrightarrow{\text{convrg}} E(Y \mid X = x)$

  • Under asymptotic infinite sample conditions, $k\text{NN}$ operates as a universal function approximator.

Decision Theory for Categorical Outputs (Classification & Bayes Classifier)

• Decision Theory Formulation for Classification:

  • Let target class labels belong to finite categorical set $\mathcal{G} = \{G_1, G_2, \dots, G_K\}$ with set size $|\mathcal{G}| = K$. Prediction model $\hat{G}(X)$ outputs labels in $\mathcal{G}$.

  • Loss Matrix $\mathbf{L}$:

  • Costs are specified by a $K \times K$ matrix $\mathbf{L} \in \mathbb{R}_{\ge 0}^{K \times K}$, where entry $L(k, \ell)$ defines the loss incurred by classifying an observation belonging to true class $G_k$ as predicted class $G_\ell$

  • Correct classifications along the matrix diagonal incur zero cost ($L(k, k) = 0$).

• 0-1 Loss Function:

  • The standard classification loss penalizes every misclassification with a unit cost of 1:     $L[G_k, \hat{G}(X)] = \begin{cases} 1 & \text{if } G_k \neq \hat{G}(X) \\ 0 & \text{if } G_k = \hat{G}(X) \end{cases}$

• Categorical Expected Prediction Error:

  • Integrating misclassification loss over joint distribution $\operatorname{Pr}(G, X)$ yields:     $EPE = E\left[ L(G, \hat{G}(X)) \right] = E_X \sum_{k=1}^K L\left[ G_k, \hat{G}(X) \right] \operatorname{Pr}(G_k \mid X)$

  • Pointwise minimization at specific feature vector $X = x$ requires:     $\hat{G}(x) = \operatorname{argmin}_{g \in \mathcal{G}} \sum_{k=1}^K L[G_k, g] \operatorname{Pr}(G_k \mid X = x)$

• Derivation of the Bayes Classifier Under 0-1 Loss:

  1. Substituting 0-1 loss simplifies the summation:      $\sum_{k=1}^K L[G_k, g] \operatorname{Pr}(G_k \mid X = x) = \sum_{k \neq g} \operatorname{Pr}(G_k \mid X = x) = 1 - \operatorname{Pr}(g \mid X = x)$

  2. Minimizing misclassification cost simplifies directly to maximizing class probability:      \hat{G}(x) = \operatorname{argmin}_{g \in \mathcal{G}} \left[ 1 - \operatorname{Pr}(g \mid X = x) \right] = \operatorname{argmax}_{g \in \mathcal{G}} \operatorname{Pr}(g \n\mid X = x)

  • Bayes Optimal Decision Rule:     $\hat{G}(x) = G_k \quad \text{if } \operatorname{Pr}(G_k \mid X = x) = \max_{g \in \mathcal{G}} \operatorname{Pr}(g \mid X = x)$

  • The strategy assigning observations to the most probable class conditional on $X = x$ is termed the Bayes classifier. The minimal error rate achieved by this optimal boundary is the Bayes rate.

  • $k\text{NN}$ classification directly approximates the Bayes classifier by replacing exact conditional class probabilities $\operatorname{Pr}(G_k \mid X = x)$ with empirical training class frequencies inside local neighborhood $N_k(x)$.

The Curse of Dimensionality in Local Methods

• Breakdown of Local Methods in High Dimensions:

  • Although asymptotic convergence $\hat{f}(x) \to E(Y \mid X = x)$ holds theoretically as $N, k \to \infty$, high feature dimensionality ($p \gg 1$) degrades performance drastically. This phenomenon is known as the curse of dimensionality.

• Geometric Distortion 1: Loss of Neighborhood Locality:

  • Assume $p$ input variables are uniformly distributed within a unit hypercube $[0, 1]^p$.

  • Consider a sub-cube neighborhood capturing a fractional proportion $r$ of total sample volume. The required edge length $e_p(r)$ of this sub-cube is:     $e_p(r) = r^{1/p}$

  • Evaluating required edge lengths to capture a small $10\%$ volume fraction ($r = 0.1$) across dimensions:

  • $p = 1 \implies e_1(0.1) = 0.10$ ($10\% of feature range)\n - p = 2 \implies e_2(0.1) = 0.32\n - p = 3 \implies e_3(0.1) = 0.46 ($46\% of feature range)

  • $p = 4 \implies e_4(0.1) = 0.56$

  • $p = 5 \implies e_5(0.1) = 0.63$

  • $p = 6 \implies e_6(0.1) = 0.68$

  • $p = 10 \implies e_{10}(0.1) = 0.80$ ($80\% of feature range)\n - **Implication**: To collect even 10\%$of data instances in 10-dimensional space, the neighborhood must span$80\% of the full coordinate range along every input dimension. Consequently, neighborhoods cease being local, invalidating local constant assumptions.\n\n\n\n\n\n- **Geometric Distortion 2: Vanishing Inscribed Spherical Volume**:\n - Consider a hypersphere S_p$inscribed within a unit hypercube$C_p$across$p dimensions.\n - The ratio of hypersphere volume to hypercube volume approaches zero rapidly as dimension increases:\n    \lim_{p \to \infty} \frac{\text{Vol}(S_p)}{\text{Vol}(C_p)} = 0\n - **Implication**: As feature dimensions grow, uniformly distributed sample mass concentrates almost entirely in the corners of the bounding hypercube. Data points become sparse and isolated near domain boundaries. Nearest neighbor estimation converts from local interpolation to unreliable boundary extrapolation.\n\n\n\n# Supervised Learning as Function Approximation\n\n- **Additive Error Statistical Models**:\n - Quantitative response data are modeled assuming an underlying statistical structure:\n    Y = f(X) + \varepsilon\n    where systematic function f(X) = E(Y \mid X)$is perturbed by random stochastic error$\varepsilon$satisfying$E(\varepsilon) = 0\n - Unexplained variance \varepsilon reflects unobserved input features or inherent stochastic system variability.\n - Non-deterministic systems allow multiple distinct target outputs Y$to occur for an identical feature instance$X = x\n\n- **Categorical Response Surface Modeling**:\n - Additive error formulations cannot represent qualitative targets (e.g., adding numerical noise to categorical text labels is undefined).\n - Instead, categorical responses are modeled by approximating conditional class probability distributions p(x) = \operatorname{Pr}(G \mid X = x).\n - For binary outcome coding (0/1$), conditional probabilities map back to expected conditional target values$p(x) = E(Y \mid X = x).\n\n\n\n- **Supervised Function Approximation Framework**:\n - Learning algorithm parameterize approximators f_\theta(x)$using parameter vector$\theta.\n - **Linear Basis Expansion Models**:\n - A flexible class of approximators expands functions as weighted sums of basis functions h_k(x):\n      f_\theta(x) = \sum_{k=1}^K h_k(x) \theta_k\n - Basis choice dictates model architecture:\n - h_k(x) = x_k \implies Standard Linear Model\n - h_k(x) = x_k^m \implies Polynomial Surface Model\n - h_k(x) = \operatorname{ReLU}(w_k^T x + b_k) \implies Rectified Linear Neural Network\n - h_k(x) = \frac{1}{1 + \exp(-x^T \beta_k)} \implies Sigmoidal Neural Network\n\n\n\n- **Maximum Likelihood Estimation (MLE)**:\n - Parameters \theta$are estimated by maximizing log-likelihood over$N independent samples:\n    L(\theta) = \sum_{i=1}^N \log \operatorname{Pr}\theta(y_i)\n - Log-probabilities \log \operatorname{Pr}\theta(y_i)$lie on$(-\infty, 0]$. Maximizing$L(\theta) selects parameters under which observed training data exhibits maximum occurrence probability.\n\n\n\n- **Equivalence of Least Squares and Maximum Likelihood**:\n - Assume additive error follows independent Gaussian noise \varepsilon \sim \mathcal{N}(0, \sigma^2), giving conditional likelihood:\n    \operatorname{Pr}(Y \mid X, \theta) = \mathcal{N}\left( f_\theta(X), \sigma^2 \right)\n - The conditional log-likelihood equation expands as:\n    L(\theta) = C - \frac{1}{2\sigma^2} \sum_{i=1}^N \left( y_i - f_\theta(x_i) \right)^2\n - Maximizing log-likelihood L(\theta)$with respect to$\theta$is mathematically equivalent to minimizing Residual Sum of Squares$RSS(\theta) = \sum_{i=1}^N (y_i - f_\theta(x_i))^2$$