Deep Learning and Neural Networks: Perceptrons, MLPs, and Optimization

Vocabulary practice flashcards covering artificial intelligence, machine learning, biological vs. artificial neurons, perceptron learning rules, activation functions, multilayer perceptrons (MLP), backpropagation calculations, gradient descent variants, and Keras implementations from TP1 and TP2.

Artificial Intelligence (AI)

The overarching field comprising all techniques that allow a machine to carry out tasks normally requiring human intelligence, including reasoning, planning, perception, and natural language, encompassing non-learning methods such as rule-based expert systems and search algorithms.

Machine Learning (ML)

A subfield of artificial intelligence where algorithms learn patterns from data by using an optimization algorithm to minimize the error between predictions and target values, rather than being explicitly programmed with manual rules.

Deep Learning (DL)

A specialized branch of machine learning that utilizes artificial neural networks with multiple hidden layers to automatically and progressively extract hierarchical, higher-level feature representations directly from raw inputs.

Classical Programming

A software paradigm where developers manually write exact rules to process input data into answers, in contrast to machine learning where data and answers are supplied so the algorithm can infer the underlying rules.

Biological Neuron

A specialized biological cell consisting of dendrites that receive incoming electrical signals, a cell body (perikaryon) that processes and sums them, an axon that transmits the outgoing signal, and synapses connecting to downstream cells.

Perceptron

An early and elementary artificial neuron model that calculates a weighted sum of inputs plus a bias, then applies an activation function to yield a single output for classification or regression tasks.

Step Function (Heaviside)

A threshold activation function defined as $f(x) = \begin{cases} 0 & \text{for } x < 0 \\ 1 & \text{for } x \ge 0 \end{cases}$, producing a discrete binary activation output.

Linear Activation Function

An activation function defined as $u = x$ that performs no transformation on the weighted input sum, primarily utilized in regression tasks.

Sigmoid Activation Function

An S-shaped non-linear activation function defined as $f(x) = \frac{1}{1 + e^{-x}}$, restricting output values to the interval $(0, 1)$, making it suitable for probability estimation and binary classification.

Hyperbolic Tangent Function ($\tanh$)

An activation function defined as $u = \frac{e^{2x} - 1}{e^{2x} + 1}$, which squashes inputs into the range $[-1, 1]$ and serves as an alternative to the sigmoid function when classes are encoded as $\{-1, +1\}$.

Rectified Linear Unit (ReLU)

An activation function defined as $f(x) = \max(0, x)$, which outputs the input directly if $x > 0$ and outputs $0$ if $x \le 0$.

Perceptron Learning Rule

An update formula for adjusting single-layer perceptron weights upon error, defined as $w_i \leftarrow w_i + \alpha (y_{\text{real}} - y) x_i$, where $\alpha$ is the learning rate.

Linear Separability Limitation

The structural constraint that prevents a single-layer perceptron from solving non-linearly separable problems (such as the XOR function), as it can only establish a single straight hyperplanar decision boundary.

Multilayer Perceptron (MLP)

A feedforward artificial neural network architecture containing an input layer, one or more hidden layers performing intermediate non-linear computations, and an output layer, capable of learning non-linear functions.

Synaptic Weight ($W$)

An adjustable parameter assigned to an input connection of a neuron that determines the importance and relative contribution of that input to the overall activation.

Bias ($b$)

A trainable constant parameter added to the weighted input sum of a neuron to shift the activation function, independent of the input features.

Forward Pass

The operational step where input features flow forward through the layers and neurons of a network to compute intermediate activations and produce a predicted output.

Backpropagation

An algorithm for training multilayer neural networks that propagates the output prediction error backward layer-by-layer to calculate the partial derivatives of the loss with respect to each weight.

Chain Rule (Calculus)

The differentiation principle used in backpropagation to decompose the partial derivative of the error with respect to any weight into a product of local derivatives: $\frac{\partial E}{\partial W} = \frac{\partial E}{\partial \text{Predicted}} \times \frac{\partial \text{Predicted}}{\partial s} \times \frac{\partial s}{\partial W}$.

Squared Error Function

A quadratic cost function used to measure prediction inaccuracy, defined as $E = \frac{1}{2}(\text{desired} - \text{predicted})^2$.

Gradient Descent

An iterative optimization algorithm that updates weights and biases by stepping in the direction opposite to the error gradient according to $W_{\text{new}} = W_{\text{old}} - \eta \frac{\partial E}{\partial W}$ in order to reach a minimum.

Learning Rate ($\eta$)

A hyperparameter setting the step size for weight adjustments during gradient descent; values that are too small cause sluggish convergence, while values that are too large cause destabilization, loss escalation, or overshooting.

Epoch

A single full training cycle during which every sample in the training dataset has passed through the network once.

Batch Gradient Descent

A gradient descent variant that computes the total error over the entire training set before executing a single weight update per epoch, providing stable updates at the cost of high computational time on large datasets.

Stochastic Gradient Descent (SGD)

A gradient descent variant that calculates the error and updates model weights immediately after examining each individual randomly selected training example, resulting in fast but fluctuating updates.

Mini-batch Gradient Descent

A compromise gradient descent method that computes errors and executes weight updates on small batches of training samples (e.g., $32$ examples) rather than individual instances or the entire dataset.

Dense Layer

A fully connected layer in Keras where every neuron receives inputs from every feature or neuron of the previous layer.

Binary Cross-Entropy

A standard loss function applied to binary classification models that quantifies the divergence between predicted probabilities and ground-truth binary targets.

Pizza Sauce Detection Model Architecture (TP2)

A two-layer network configured to solve the XOR-like pizza sauce error problem using a hidden Dense layer of $2$ sigmoid neurons and an output Dense layer of $1$ sigmoid neuron, comprising a total of $9$ trainable parameters.

Single-Neuron Classifier (TP1)

A minimal neural network model constructed with a single Dense sigmoid neuron (Dense(units=1, input_dim=2, activation='sigmoid')) to classify two ingredient categories (corn vs. olives) using shape and color, requiring $3$ trainable parameters.