Module 3

Module two covered machine learning algorithms based on memory.
Specific focus on k-nearest neighbors, which relies on memorizing all the data.
Introduction to learning cosine laws that describe data points.
Major task during this module is supervised regression, which involves predicting numerical labels for data points.

Supervised regression involves predicting a numerical label for each data point based on input features.
Example involves data points represented by two coordinates, where the second coordinate serves as a numerical label.

Hypothesis formulated: a concise equation exists to represent the data.
Insignificant variance possibly results from noise within the dataset.
Hypothesis leads to the proposal that data can be modeled as a linear equation:
- General equation format: $y = ax + b$
- Here, the goal is to summarize the data using only two parameters, a and b.

Learning only two parameters (a and b) while modeling data is called parametric learning.
This is in contrast to nonparametric learning methods, such as k-nearest neighbors, which do not involve learning parameters and are instead based on memorization of the entire dataset.

Modeling refers to the process of creating a general equation with parameters.
Many hypotheses exist within the scope of settings for a and b, representing different possible models.
The concept of a "model" emerges from our endeavor to summarize data using a limited number of parameters.

The task at hand is to find the best parameters (a and b) that yield the most accurate model.
The definition of "best" will be explored later in the module.

Linear regression is only one type of regression. More generally, one might seek to find a function $f(x)$ such that $y = f(x)$ .
This would expand the possible hypothesis space significantly, allowing for more complex relationships.

Initial hypothesis for the model may yield significant errors, comparing predicted labels against actual values.
Based on the errors produced, an iterative correction process occurs—nudging the model parameters to enhance accuracy.
Similarities drawn to human learning: initial missteps are corrected for improved performance over time, akin to a baby learning language.

Formulating error for a specific data point based on the model predictions:
- Prediction for input $xi$ is given by $ext{prediction} = A imes xi + B$ .
- Error is computed as $ext{Error}i = ext{predicted} - ext{actual} = (A imes xi + B) - y_i$ .

Aggregate measure of errors calculated as:
- $ext{MAE} = rac{1}{N} imes ext{sum of errors}$ .
Minimizing the MAE can lead to an optimal model representation.

Another loss function defined as the average of the squared discrepancies between predicted and actual values:
- $ext{MSE} = rac{1}{N} imes ext{sum of (predicted - actual)}^2$ .
Preference in practice often tilts towards minimizing the MSE due to its mathematical properties, particularly its convex nature leading to easier optimization.

Concept introduced: use of gradient descent for optimization in finding the parameters a and b.
Initial guess for a and b may not yield accurate predictions, thus corrections will minimize errors iteratively.
Gradient descent utilizes the derivative to adjust parameters slowly towards optimum values in multiple iterations.
Step size (learning rate) is crucial in determining how swiftly the algorithm converges to the optimal solution.

Taking derivatives to inform how functions change and their rates at specific points.
Derivative encapsulates both direction and rate of change for functions across their domain, summarizing how function values shift alongside the input.

Gradient descent extended from a single variable to two variables ($a$ and $b$).
Partial derivatives indicate how output will vary with respect to changes in $a$ or $b$.
Updates occur independently for both a and b, still optimizing for descent towards lower error.

Addressing higher-dimensional features. For example, in 2D:
- Equation of a hyperplane is presented as $y = w1 imes x1 + w2 imes x2 + b$ .
Using linear models effectively in N-dimensional space will still follow similar principles dictated by the loss function and optimization.

Learning rate as a hyperparameter influences how parameters are adjusted during descent—critical for model refinement quality.
Initial values for weights are necessary for activation and often assume semi-randomized values to initialize learning much like in statistical modeling.
Decision on the number of iterations and epochs or the shuffle of the dataset plays a role in the thoroughness of gradient descent.

Address concerns about randomness in algorithmic design to allow for reproducibility in results measurement over multiple trials.
Mean Squared Error serves as evaluation metrics while being aware of limitations in interpretation due to scale sensitivity.

Feature engineering can significantly improve effectiveness—understanding how manipulating data dimensions contributes to predictive power.
Quadratic features highlighted as a useful solution for predicting outputs not simply confined to linear frameworks—example case involves stock market trend analysis.
Collective example could feature building models utilizing quadratic or cubic terms while deriving from raw datasets to bolster predictive capabilities.

RANSAC process described for iteratively drawing models based on sample sets while excluding outliers:
- Randomly sample from data points to develop baseline fitting.
- Track inlier retention while discarding points classified as outliers based on error thresholds—retaining quality models.
Testing RANSAC’s application to regress against real-world data.

Polynomial regression and quadratic features discussed as a vital means to adapt linear regression towards more complex data distributions,
Final insights touch on the importance of usability and practicality of methods taught for deployment in real-life models and regression predictions.
Encouragement to explore different methods of regression, refine approaches to model fitting, and iteratively improve understanding through practice.