1/78
79 cards from the 5 completed Linear Regression items. Deep-ML grouping: 01 The Geometry of Least Squares (39 cards); 02 Implementation 1: The Normal Equation (14); 04 The Gradient Step (26). Every question starts with its group number and name. Only completed lessons/problems are covered; group 03 and unfinished projects/labs are not included. Includes plain-language definitions, worked examples, and coding requirements. Advanced cards are labeled for later study.
01 · The Geometry of Least Squares — What does it mean for two real vectors to be orthogonal?
Answer: Their dot product is zero. Explanation: For nonzero vectors, this means their directions meet at 90°. This is the perpendicularity needed for orthogonal projection.
01 · The Geometry of Least Squares — Are u = (1, 2) and v = (4, −2) orthogonal?
Answer: Yes. Explanation: Their dot product is 1×4 + 2×(−2) = 0.
01 · The Geometry of Least Squares — What is the orthogonal projection of a vector onto a line through the origin?
Answer: The point on that line closest to the vector's endpoint. Explanation: Imagine dropping a perpendicular from the endpoint to the line: the foot of that perpendicular is the projection.
01 · The Geometry of Least Squares — In a line-projection problem, what does the nonzero vector L describe?
Answer: The direction of the line, not a single allowed endpoint. Explanation: The line contains every scalar multiple of L and passes through the origin. For L = (1, 0), it is the horizontal axis.
01 · The Geometry of Least Squares — How do you project v onto the line in the nonzero direction u?
Answer: p = ((u · v)/(u · u))u. Explanation: Compute two dot products, divide them to get one scalar weight, then multiply u by that weight. p is the projection vector.
01 · The Geometry of Least Squares — Why divide by u · u in the projection formula?
Answer: To account for the length of the line's direction vector. Explanation: u · u is its squared length. This makes the projection depend on the line, rather than on how long you drew its direction arrow.
01 · The Geometry of Least Squares — Why must the direction vector in the projection formula be nonzero?
Answer: Otherwise u · u is zero and the formula divides by zero. Explanation: The zero vector does not specify a unique line direction. The problem does not specify an error return for this case.
01 · The Geometry of Least Squares — For v = (2, 2) and u = (1, 1), what scalar multiplies u in the projection formula?
Answer: The coefficient is 2. Explanation: (u · v)/(u · u) = (2 + 2)/(1 + 1) = 4/2. The projection is then 2u = (2, 2).
01 · The Geometry of Least Squares — Project v = (3, 4) onto u = (1, 0).
Answer: p = (3, 0). Explanation: The scalar weight is (3×1 + 4×0)/(1² + 0²) = 3. Multiplying (1, 0) by 3 gives the projection.
01 · The Geometry of Least Squares — If p is v's orthogonal projection onto direction u, what is special about v − p?
Answer: It is perpendicular to u. Explanation: The leftover vector has dot product zero with u. For v = (3, 4), u = (1, 0), p = (3, 0), the leftover is (0, 4).
01 · The Geometry of Least Squares — Does changing the direction vector from u to 2u change the projected point?
Answer: No: they describe the same line. Explanation: The scalar coefficient adjusts for the changed length. The projection vector stays the same.
01 · The Geometry of Least Squares — If v is already on the projection line, what is its projection?
Answer: v itself. Explanation: The closest point is its own endpoint, so the leftover v − p is zero.
01 · The Geometry of Least Squares — What does orthonormal mean for a set of vectors?
Answer: Each has length 1, and different vectors are perpendicular. Explanation: Ortho refers to perpendicularity; normal refers here to unit length.
01 · The Geometry of Least Squares — For a square orthogonal matrix Q, what is QᵀQ?
Answer: The identity matrix I. Explanation: Qᵀ means transpose. Each column has dot product 1 with itself and 0 with every other column.
01 · The Geometry of Least Squares — [Coding] What output does orthogonal_projection(v, L) require?
Answer: The projected vector as a Python list, rounded to three decimal places. Explanation: The inputs are two lists: v is projected onto the line in direction L. Example output: [3.0, 0.0].
01 · The Geometry of Least Squares — What is a training example in regression?
Answer: One observed case with input features and a target value. Explanation: For a house, features might be size and age; the target could be its known sale price.
01 · The Geometry of Least Squares — What does a feature mean in regression?
Answer: An input quantity used to make a prediction. Explanation: For example, house size. A row of the design matrix contains the inputs for one case.
01 · The Geometry of Least Squares — What is the target y in regression?
Answer: The observed values the model tries to predict. Explanation: For house prices, y contains known prices. It differs from the model's predicted values.
01 · The Geometry of Least Squares — What is the design matrix X?
Answer: The matrix of inputs, with one example per row and one model input per column. Explanation: A column of ones may be included for the intercept. The columns determine which prediction vectors the model can produce.
01 · The Geometry of Least Squares — What are the weights w in linear regression?
Answer: The coefficients that multiply the input columns. Explanation: For a one-feature model, prediction = intercept + slope×feature. The weights may also be called θ (theta).
01 · The Geometry of Least Squares — If X has n rows and d columns, how many coefficients are in w?
Answer: There are d coefficients. Explanation: One per input column. Xw produces n predictions, one per row. The counts include an intercept column if one is present.
01 · The Geometry of Least Squares — What does Xw represent?
Answer: The vector of model predictions. Explanation: Each prediction is a row of X dotted with w. A hat, as in ŷ, often marks predicted values: ŷ = Xw.
01 · The Geometry of Least Squares — What is a residual in regression using the observed-minus-predicted convention?
Answer: An observed value minus its prediction. Explanation: For all examples, r = y − Xw. Some formulas use the opposite sign; the squared errors are the same either way.
01 · The Geometry of Least Squares — What does the sum of squared errors (SSE) measure?
Answer: The sum of the squared differences between predictions and observations. Explanation: If residuals are (1, −2), SSE = 1² + (−2)² = 5. Squaring prevents positive and negative mistakes from canceling.
01 · The Geometry of Least Squares — What does least squares choose the weights to minimize?
Answer: The sum of squared errors. Explanation: Written ‖Xw − y‖₂². The notation means square each component of the error vector and add them.
01 · The Geometry of Least Squares — What does arg min mean in the least-squares formula?
Answer: The value of the weights that gives the smallest loss. Explanation: It returns the minimizing weights, not the minimum loss number. w* means a best-fitting weight vector.
01 · The Geometry of Least Squares — What is the column space of X, in plain language?
Answer: All vectors you can make by combining X's columns with scalar weights. Explanation: It is exactly the set of prediction vectors Xw available to the model.
01 · The Geometry of Least Squares — Geometrically, where does the least-squares prediction Xw* lie?
Answer: At the orthogonal projection of y onto X's column space. Explanation: The model chooses the available prediction vector closest to the observed target vector.
01 · The Geometry of Least Squares — At a least-squares optimum, how does the residual relate to every column of X?
Answer: It is perpendicular to each column. Explanation: Xᵀ(y − Xw*) = 0. Each entry of this equation is a column–residual dot product.
01 · The Geometry of Least Squares — What are the normal equations for least squares?
Answer: XᵀXw = Xᵀy. Explanation: They express that the residual is perpendicular to all input columns. The word normal here refers to perpendicularity, not a normal probability distribution.
01 · The Geometry of Least Squares — When is XᵀX invertible in ordinary linear least squares?
Answer: When X's columns are linearly independent. Explanation: This is full column rank: no input column can be made by combining the others.
01 · The Geometry of Least Squares — What closed-form weight formula applies when X has full column rank?
Answer: w* = (XᵀX)⁻¹Xᵀy. Explanation: ᵀ means transpose and ⁻¹ means matrix inverse. Closed form gives the answer through matrix operations instead of repeated gradient steps.
01 · The Geometry of Least Squares — For no-intercept data X = (1, 2, 3) as one column and y = (2, 4, 5), what slope minimizes squared error?
Answer: w* = 25/14, approximately 1.7857. Explanation: Xᵀy = 1×2 + 2×4 + 3×5 = 25. XᵀX = 1² + 2² + 3² = 14. Divide the first by the second.
01 · The Geometry of Least Squares — For X = [[1, 1], [1, 2], [1, 3]], what is XᵀX?
Answer: [[3, 6], [6, 14]]. Explanation: The first column is (1, 1, 1), the second is (1, 2, 3). Their self-dot-products are 3 and 14; their cross-dot-product is 6.
01 · The Geometry of Least Squares — For X = [[1, 1], [1, 2], [1, 3]] and y = (1, 2, 2), what is Xᵀy?
Answer: (5, 11), as a column vector. Explanation: Dot each input column with y: 1+2+2 = 5 and 1×1+2×2+3×2 = 11.
01 · The Geometry of Least Squares — For feature values (1, 2, 3) and targets (1, 2, 2), what best-fitting line is in the lesson's worked example?
Answer: prediction = 2/3 + (1/2)×feature. Explanation: The normal equations are 3w₀ + 6w₁ = 5 and 6w₀ + 14w₁ = 11. They give intercept w₀ = 2/3 and slope w₁ = 1/2.
01 · The Geometry of Least Squares — [Advanced] What does the pseudoinverse X⁺ do for least squares?
Answer: It provides weights w = X⁺y even when an ordinary inverse formula is unavailable. Explanation: For a tall full-column-rank X, X⁺ = (XᵀX)⁻¹Xᵀ. More generally it gives the least-squares solution with smallest weight-vector length.
01 · The Geometry of Least Squares — [Extension mentioned in this lesson] What does ridge regression add to XᵀX?
Answer: λI, with a positive penalty strength λ. Explanation: I is the identity matrix; λ (lambda) is a chosen positive number. Adding it to every coefficient's diagonal makes the system invertible and penalizes large weights.
01 · The Geometry of Least Squares — Why can ridge help when columns are dependent or nearly dependent?
Answer: It removes singularity for a positive penalty on all coefficients and reduces sensitivity. Explanation: Dependent columns contain redundant information. Nearly dependent columns can make weights very sensitive to small changes in the data.
02 · Implementation 1: The Normal Equation — What does linear regression model for one input feature?
Answer: A straight-line prediction: ŷ = intercept + slope×x. Explanation: ŷ is the prediction, while y is the actual observed target. x is the input feature.
02 · Implementation 1: The Normal Equation — What does a line's intercept mean?
Answer: Its predicted target when the input feature is zero. Explanation: For ŷ = 3 + 2x, the intercept is 3. Zero may lie outside the data you observed, so the intercept need not have a useful real-world interpretation.
02 · Implementation 1: The Normal Equation — What does a line's slope mean?
Answer: The change in its prediction for a one-unit increase in the feature. Explanation: For ŷ = 3 + 2x, the prediction rises by 2 when x rises by 1.
02 · Implementation 1: The Normal Equation — Why put a column of ones in X?
Answer: To let a coefficient act as the intercept. Explanation: For a row [1, x] and weights [w₀, w₁], their dot product is w₀ + w₁x.
02 · Implementation 1: The Normal Equation — [Coding] If X already contains an intercept column of ones, should you add another?
Answer: No. Explanation: Two identical ones-columns are redundant and make XᵀX singular. The examples already provide the intercept column.
02 · Implementation 1: The Normal Equation — [Coding] For a one-feature X whose columns are [ones, feature], what order are the coefficients returned in?
Answer: [intercept, slope]. Explanation: The coefficient order follows the column order. For ŷ = 1 + 2x, return approximately [1, 2].
02 · Implementation 1: The Normal Equation — For X = [[1, 1], [1, 2], [1, 3]] and y = [1, 2, 3], what coefficients fit exactly?
Answer: [0.0, 1.0]. Explanation: The line is ŷ = 0 + 1×x. The first coefficient is the intercept and the second the slope.
02 · Implementation 1: The Normal Equation — [Coding] What rounding does the normal-equation problem require?
Answer: Round the returned coefficients to four decimal places. Explanation: For example, 1/3 becomes 0.3333. The task explicitly accepts −0.0 from rounding a tiny negative number.
02 · Implementation 1: The Normal Equation — Does the normal-equation method need a learning rate?
Answer: No. Explanation: It computes a closed-form solution rather than choosing repeated step sizes.
02 · Implementation 1: The Normal Equation — Does the normal equation mathematically require feature scaling?
Answer: No. Explanation: It has no gradient steps to tune. Scaling can still help numerical conditioning when feature magnitudes differ greatly.
02 · Implementation 1: The Normal Equation — Why can the normal equation become costly with many features?
Answer: It builds and solves a system involving the feature-by-feature matrix XᵀX. Explanation: That matrix grows with the number of model input columns, not just the number of training rows.
02 · Implementation 1: The Normal Equation — [Coding] What does X.T @ X mean for a 2D NumPy array?
Answer: The transpose of X multiplied by X using matrix multiplication. Explanation: .T transposes the array; @ is matrix multiplication. X * X would multiply entries individually instead.
02 · Implementation 1: The Normal Equation — [Coding] Why might you solve (XᵀX)w = Xᵀy rather than explicitly build an inverse?
Answer: A direct solve generally uses less work and is more numerically reliable than forming the inverse. Explanation: The inverse formula explains the mathematics. For challenging data, QR or a least-squares routine can avoid the numerical problems of forming XᵀX.
02 · Implementation 1: The Normal Equation — What problem occurs if two feature columns are exact multiples of one another?
Answer: X is not full column rank, so XᵀX has no ordinary inverse. Explanation: Example: columns size and twice-size contain the same information. Do not assume the plain inverse formula works for every input matrix.
04 · The Gradient Step — What does gradient descent do in linear regression?
Answer: Repeatedly adjusts the weights to reduce prediction loss. Explanation: Loss is a number measuring how badly predictions match observations. Start with weights, calculate a direction, take a step, and repeat.
04 · The Gradient Step — [Prerequisite] What is a gradient?
Answer: A vector describing the direction of fastest local increase of the loss. Explanation: Its components say how the loss changes as each weight changes. Moving in the negative-gradient direction locally goes downhill.
04 · The Gradient Step — What does the learning rate α control?
Answer: How large each weight-update step is. Explanation: α is read alpha. The update is weights_new = weights_old − α×gradient.
04 · The Gradient Step — What can happen if the learning rate is too large?
Answer: The updates can overshoot and the loss can grow or diverge. Explanation: Diverge means the weights fail to approach a stable solution. More iterations do not fix an unstable step size.
04 · The Gradient Step — What can happen if the learning rate is too small?
Answer: Learning may be very slow. Explanation: The weights make only tiny changes per iteration. An iteration is one pass through the update procedure.
04 · The Gradient Step — [Coding] How should the weights start in Deep-ML's gradient-descent problem?
Answer: Initialize every weight to zero. Explanation: Use one coefficient per column of X, including the intercept column.
04 · The Gradient Step — What does batch gradient descent use for each update?
Answer: All training examples. Explanation: This completed exercise requires batch updates, not a randomly selected example or a smaller subset.
04 · The Gradient Step — [Coding] In this task, what do X's dimensions (m, n) count?
Answer: m training examples and n input columns, including the intercept column. Explanation: The target array y has shape (m,), and the weights have n components. Here n includes the ones-column.
04 · The Gradient Step — How are predictions calculated in one batch-gradient step?
Answer: ŷ = Xθ. Explanation: θ (theta) is the weight vector. Each row of X dotted with θ gives one predicted target.
04 · The Gradient Step — Which error sign does this gradient-descent implementation use?
Answer: Prediction minus observation: e = Xθ − y. Explanation: The least-squares geometry often defines the residual as y − Xθ instead. The signs differ, but squared losses match.
04 · The Gradient Step — What is ordinary mean squared error (MSE)?
Answer: The average of the squared prediction errors. Explanation: For errors (1, −2), MSE = (1² + (−2)²)/2 = 2.5. It uses 1/m times the sum of squared errors.
04 · The Gradient Step — Which scaled squared-error loss does this exercise use?
Answer: L(θ) = (1/(2m)) × sum of squared errors. Explanation: It is half of ordinary MSE. The factor 1/2 makes the gradient formula simpler without changing the minimizing weights.
04 · The Gradient Step — For errors (1, −2), what is the exercise's half-MSE loss?
Answer: The loss is 1.25. Explanation: (1² + (−2)²)/(2×2) = 5/4. There are m = 2 examples.
04 · The Gradient Step — [Algorithm] What is the batch gradient for the exercise's half-MSE loss?
Answer: g = (Xᵀe)/m, with e = Xθ − y. Explanation: Xᵀ combines each input column with the errors. Divide by m, the number of examples. The result has one gradient component per weight.
04 · The Gradient Step — [Algorithm] How are all the weights updated after computing g?
Answer: θ_new = θ_old − αg. Explanation: Subtract the learning rate times the gradient. Compute the full gradient from the old weights before changing them.
04 · The Gradient Step — Why does the half-MSE loss produce a gradient without a leading factor of 2?
Answer: The loss's 1/2 cancels the 2 from differentiating a square. Explanation: For ordinary MSE instead, the gradient is (2/m)Xᵀe. Keep the loss convention and gradient consistent.
04 · The Gradient Step — For half-MSE, what happens if you accidentally use the ordinary-MSE gradient with the same α?
Answer: Your effective update step doubles. Explanation: The gradient is twice as large. To get the same update with that gradient, use half the learning rate.
04 · The Gradient Step — [Coding] What should the gradient-descent function return?
Answer: The final learned coefficients as a NumPy array. Explanation: The inputs include NumPy arrays X and y, the learning rate, and the number of iterations. X already has a ones-column for the intercept.
04 · The Gradient Step — For targets [3, 5, 7] at feature values [1, 2, 3], which line fits exactly?
Answer: ŷ = 1 + 2x. Explanation: The coefficients are [1, 2] when X's columns are [ones, feature]. Numerical gradient descent approaches them rather than guaranteeing exact decimal equality.
04 · The Gradient Step — With zero initial weights, X = [[1, 1], [1, 2], [1, 3]], and y = [3, 5, 7], what is the first error vector?
Answer: e = (−3, −5, −7). Explanation: Initial predictions Xθ are all zero. Error = prediction − target.
04 · The Gradient Step — For that initial error e = (−3, −5, −7), what is the half-MSE gradient with X = [[1, 1], [1, 2], [1, 3]]?
Answer: g = (−5, −34/3). Explanation: The first column gives (−3−5−7)/3 = −5. The feature column gives (−3−10−21)/3 = −34/3.
04 · The Gradient Step — Starting at θ = (0, 0), with g = (−5, −34/3) and α = 0.1, what are the next weights?
Answer: θ_new = (0.5, 17/15), approximately (0.5, 1.1333). Explanation: Subtract 0.1g from zero. This is one step, not the final best-fitting line.
04 · The Gradient Step — Why might differently scaled features slow gradient descent?
Answer: One learning rate must cope with directions whose useful step sizes differ greatly. Explanation: Scaling features to similar ranges can make optimization easier. This is a practical aid, not a requirement to understand the update equation.
04 · The Gradient Step — What is one key tradeoff between the normal equation and gradient descent?
Answer: The normal equation solves a matrix system; gradient descent takes repeated updates with a learning rate. Explanation: The preferred method depends on data size, feature count, conditioning, and required accuracy. Neither is always fastest.
04 · The Gradient Step — [Advanced] What quantity limits a stable fixed learning rate for this quadratic loss?
Answer: The largest eigenvalue of H = XᵀX/m. Explanation: For positive curvature directions, the usual stability range is 0 < α < 2/λmax(H). An eigenvalue measures scaling along a special direction; larger curvature requires smaller steps.
04 · The Gradient Step — [Advanced] Is the playground's approximate learning-rate threshold 0.169 universal?
Answer: No: it is specific to that playground's X and loss scaling. Explanation: Changing the feature values or the loss scale changes XᵀX/m and therefore changes the stable step-size range.