152 Machine Learning Semester 2

Machine learning vocabulary

Machine Learning

A field of artificial intelligence that uses statistical techniques to give computer systems the ability to learn from data.

Deep Learning

A subset of machine learning that uses neural networks with many layers to analyze various levels of data abstraction.

Keras

A high-level API for building and training neural networks, supporting multiple backend libraries like TensorFlow and PyTorch.

TensorFlow

A neural network library developed by Google for research purposes, known for its capabilities in deep learning and production use.

PyTorch

A neural network library developed by Meta, popular in research for its dynamic computation graph and debugging capabilities.

JAX

A machine learning framework that aims to follow NumPy's syntax and style, enabling high-performance numerical computing.

Scikit-learn

An industry-standard library for machine learning in Python, primarily focusing on classical machine learning algorithms.

Model.compile()

A Keras method used to configure the learning process by specifying optimizers, loss functions, and metrics.

Model.fit()

A Keras method that trains the model for a fixed number of epochs (iterations on a dataset).

Backends in Keras

Function sets or libraries like TensorFlow or PyTorch that Keras relies on to execute training and inference operations.

Sequential Model

A type of model in Keras where layers are defined in a linear stack, starting with the input layer and ending with the output layer.

Activation functions

Mathematical functions applied to the output of each layer in a neural network, determining the output based on the input.

Epochs

One complete presentation of the dataset to be estimated during training, often leading to adjusting the model's weights.

Batch size

The number of training examples utilized in one iteration of the training process.

Loss function

A function that measures the difference between the predicted values and the actual values during training.

Metrics

Quantitative measures used to evaluate the performance of a model during training and testing.

Training neural networks

The process of adjusting a neural network's weights based on training data to improve its predictions.

Linear regression

A statistical method for modeling the relationship between a dependent variable and one or more independent variables using a linear equation.

Least squares

A method for minimizing the differences between observed values and those predicted by a model.

Multilayer perceptron

A type of neural network consisting of multiple layers of nodes that are fully connected.

Forward propagation

The process of inputting data into a neural network and calculating the output based on current weights.

Backward propagation

The process of adjusting the weights in a neural network to minimize the loss function.

Batch size

The number of training instances used in one iteration to update the weights of a neural network.

Epoch

A full pass through the entire training dataset during training of a neural network.

Regularization

A technique used to prevent overfitting in predictive models by adding a penalty for larger weights.

Dropout

A regularization technique where a fraction of neurons are randomly ignored during training to reduce overfitting.

Dendrite

A branched extension of a neuron that receives signals from other neurons.

Node of Ranvier

A gap in the myelin sheath of a nerve, allowing for the rapid conduction of impulses.

Axon

A long, slender projection of a neuron that conducts electrical impulses away from the cell body.

Multilayer Perceptron

A type of neural network that consists of multiple layers of nodes, including an input layer, hidden layers, and an output layer.

Activation Function

A mathematical function that introduces non-linearity into the output of a node in a neural network, determining the activated output of that node.

ReLU Activation Function

An activation function defined as f(z) = max(0, z), commonly used for hidden layers in neural networks.

Sigmoid Activation Function

An activation function defined as f(z) = rac{1}{1 + e^{-z}}, often used for binary classification in neural networks.

Softmax Activation Function

An activation function used in the output layer for multi-class classification that converts logits to probabilities.

Weights in Neural Networks

Parameters that define the strength of the connection between nodes in a neural network, adjusted during the training process.

Feedforward Neural Network

A type of neural network where connections between the nodes do not form cycles, allowing data to flow in one direction.

Random Forests

An ensemble model that uses decision trees fitted to a subset of input features as the base models.

Ensemble Model

A model that combines multiple individual models to produce a better predictive performance.

Bootstrap Sample

A random sample of data points taken from a dataset with replacement, used in training models.

Feature Sampling

The selection of a random sample of features from the input features for training a model.

Decision Tree

A type of model that makes decisions based on asking a series of questions about the input features.

Prediction Aggregation

The process of combining predictions from multiple models to produce a final prediction, such as averaging or voting.

Feature Importance

A measure of the contribution of each input feature to the overall prediction of the model.

Correlated Input Features

Features that are related to each other, which can impact the performance of machine learning algorithms.

Gini Index

A metric used to measure the impurity or purity of a dataset; a lower Gini index indicates a more pure set.

Hyperparameters

Parameters that are set before the learning process begins, which can affect the performance of the model.

Data Cleaning

The process of addressing or fixing missing values, duplicate data, and incorrectly formatted data.

Data Integration

The process of combining data from different sources into a unified view.

Data Reduction

The process of reducing the dimensionality of the dataset, simplifying the data.

Data Transformation

The process of converting features into a format suitable for specific models or algorithms.

Imputation

A group of techniques used to replace missing values in a dataset with a reasonable estimate.

Univariate Imputation

Replaces missing values for a feature using only non-missing values for that same feature.

Multivariate Imputation

Provides more accurate imputations compared to univariate methods, especially when complex dependencies between features exist.

K-nearest Neighbors Imputation (KNN)

Uses the K most similar instances to a data point to impute the missing values, can handle numeric and categorical data.

Iterative Imputation

Uses regression to predict the missing values based on other features in the data.

SimpleImputation()

An initial imputation method used in iterative imputation to obtain estimates for missing values.

Stopping Criterion

Conditions that dictate when the iterative imputation process should stop, such as maximum iterations or specified tolerance.

Decision tree regressor

A decision tree used for regression tasks that divides input feature space using inequalities.

Leaf

A terminal node in a decision tree where the predicted value is computed from training instances.

Mean Squared Error (MSE)

A common measure of the average squared difference between predicted and actual values.

Mean Absolute Error (MAE)

A metric that measures the average magnitude of errors between predicted and actual values, without considering their direction.

Friedman's Improvement (FI)

A measure used to evaluate the performance of a decision tree regression model, comparing the differences in predictions.

Cost Complexity Pruning

A technique used in decision trees to remove nodes that do not significantly improve the model's performance.

Error estimate

An assessment of the accuracy of a model based on its predictions compared to actual values.

Performance evaluation

The process of assessing how well a decision tree regressor performs using metrics like MSE and MAE.

Subtree

A section of a decision tree that can be analyzed separately.

Cost complexity formula

Rx(T) = R(T) + alpha * |T|, where R(T) is the error estimate of the tree and |T| is the number of leaves.

Bootstrapping

The process of generating simulated samples by repeatedly sampling instances with replacement from an existing sample.

Sampling with replacement

A sampling method where some instances are selected multiple times while others may not be selected at all.

Out-of-bag sample (OOB)

A subset of data points that are not selected for training in a bootstrap sample, typically containing about 37% of the original unique data points.

Cross-validation with bootstrap

A technique where bootstrap samples are used as training sets in each iteration, and corresponding out-of-bag samples are used as validation sets.

Population

The complete set of items or individuals that are being studied or sampled from, in this case, comprising 60000 instances.

Bootstrap sample

A subset created from the original dataset by randomly sampling with replacement, utilized in bootstrapping to estimate the distribution of a statistic.

Satisfaction

A measure of contentment, often categorized in surveys as satisfied or unsatisfied based on responses.

Survey

A method of gathering information from individuals, often used to assess opinions or satisfaction in a sample population.

Ensemble Model

A model that combines predictions from multiple models trained on bootstrap samples, resulting in a stronger aggregated model.

Base Model

An individual model within an ensemble, often a decision tree used for its computational efficiency and suitability for classification and regression.

Bootstrap Sample

A sample drawn with replacement from the original dataset, used in the bagging algorithm to train base models.

Bagging Algorithm

A method involving bootstrapping to generate samples, fitting base models on those samples, and aggregating their predictions to form an ensemble.

Overfitting

A modeling error that occurs when a model learns the noise in the training data instead of the actual signal, leading to poor performance on unseen data.

Underfitting

A modeling error that occurs when a model is too simple to capture the underlying pattern of the data, resulting in poor performance even on training data.

Estimators

In the context of bagging, the number of base models used in the ensemble, which typically ranges between 10 and 25 for optimal performance.

Bias-Variance Tradeoff

The balance between the error introduced by the bias and the error introduced by the variance in a model, influencing overall prediction accuracy.

Averaging

The method of averaging the predicted outputs of different models to form an ensemble prediction, commonly used for regression problems in bagging.

Decision Tree

A hierarchical structure made up of nodes and edges used for classification tasks.

Node

An object in a decision tree representing a question or decision point.

Edge

A directed link from a parent node to a child node.

Leaf

A node that has no outgoing edges to child nodes and signifies a final decision.

Root Node

The top node of a decision tree with no parent.

Depth of a Node

The number of edges that must be followed to reach that node from the root node.

Decision Tree Classifier (DTC)

A decision tree specifically built for classification tasks, where each leaf corresponds to a class.

Hyperplane

A decision boundary used to divide feature space in classification tasks.

Gini Impurity Measure

A measure of impurity in a node given by the formula: $Gini = P_0 (1-P_0) + P_1 (1-P_1)$.

Overfitting

A condition where a model learns the details of the training data too well, leading to poor generalization on unseen data.

Early Stopping

A technique used to prevent further growth of the tree based on node statistics, aimed at reducing overfitting.

Cost Complexity Pruning

A method to prevent overfitting by reducing the size of an already trained decision tree.

Probability Estimate in DTC

The proportion of instances from the predicted class in the resulting leaf, provided by the method $DTC.predict ext{-}proba(X)$.

Data Cleaning

The process of addressing and fixing missing values, duplicate data, and incorrectly formatted data.

Data Integration

The process of combining data from different sources into a unified view.

Data Reduction

The process of reducing the dimensionality of the dataset, simplifying the data.