GenAI Architecture and Neural Networks
Hierarchy of Artificial Intelligence Systems
Artificial Intelligence (AI): The overarching field encompassing all software systems and machines designed to perform tasks that typically require human intelligence.
Machine Learning (ML): A subfield of AI focused on algorithms that analyze data, learn underlying patterns, and make data-driven predictions without explicit rule-based programming.
Deep Learning (DL): An advanced subset of ML utilizing multi-layered artificial neural networks designed to process complex data abstractions and high-dimensional representations.
Generative AI (GenAI): A specialized domain of deep learning focused on synthesizing brand-new content (including natural text, images, audio, video, and code) based on statistical patterns learned from vast training datasets.

Biological Foundations of Neural Computation
Biological Inspiration: Artificial neural networks are computational models inspired by the structural and functional properties of biological human brains.
Neurons: Biological nerve cells serving as fundamental processing units.
Chemical transmitters within cerebral fluid alter the electrical potential across neuron membranes.
When the electrical potential crosses a specific physiological threshold, the neuron fires (spikes), propagating an electrical pulse of fixed strength down its path.
Axons and Synapses:
Axons branch out to connect one neuron to thousands of neighboring neurons.
Connection strengths reflect statistical correlations between firing events.
Synaptic connections that fire simultaneously strengthen over time; inactive or non-correlated connections weaken and atrophy.
Every individual neuron performs discrete information input, processing, and output operations.


Mathematical Architecture of Artificial Neurons
Artificial Neuron Elements:
Inputs (): Signals received from external sources or preceding neural layers.
Weights (): Scalar multipliers representing the connection strength assigned to each input signal.
Bias (): A constant intercept term added to the weighted input sum, functioning similarly to the intercept in linear regression models.

Summation Equation:
The linear combination of inputs, weights, and bias produces an intermediate output :
Activation Functions:
Mathematical transformations applied to the linear combination to introduce non-linearity into the network:
Threshold Function / Perceptron Step: Determines binary activation (fire vs. not fire) via a step function .
Identity Function: Passes the input directly without alteration:
Sigmoid Function: Squeezes real-valued numbers into a continuous range between and :
Hyperbolic Tangent (Tanh) Function: Maps inputs to a zero-centered continuous range between and :
Rectified Linear Unit (ReLU): Outputs zero for negative inputs and linear identity for positive inputs:
Leaky ReLU: Prevents dying neurons by permitting a small non-zero slope () for negative inputs:
Exponential Linear Unit (ELU): Smooths negative values exponentially:
Softmax Function: Converts raw output logits into a normalized probability distribution across multiple classes:
Softplus Function: A smooth, continuous approximation of ReLU:
Multi-Layer Artificial Neural Networks (ANN)
Layered Structure:
Input Layer: Entry nodes representing observable input variables or feature parameters.
Hidden Layer(s): Intermediate computational processing stages that transform input signals into high-level abstraction representations ("thoughts"). Hidden layers operate as unobservable mathematical black boxes.
Output Layer: Final layer producing output predictions, discrete label classifications, or estimated scalar values.

Multi-Class Classification Structure:
Every output node computes a linear combination of hidden signals passed through an activation function.
For a multi-class task with target classes, the network contains distinct output nodes.
Output node evaluates the class conditional probability .

Functional Scope and Independence:
Neurons within a network do not function independently; each node integrates weighted signals from all connected preceding nodes, establishing interdependent global network computations.
Neural networks support both classification (discrete categorizations) and regression (continuous target variables) objectives.
Parameter Estimation and Network Training
Trainable Network Parameters:
Weights: One weight parameter exists for every directed connection arrow between nodes.
Biases: One bias parameter exists for every single neuron situated in hidden or output layers.
Single Hidden Layer Parameter Calculation:
Let be the number of input nodes.
Let be the number of hidden nodes.
Let be the number of output nodes.
Number of weights between Input and Hidden Layer =
Number of biases in Hidden Layer =
Number of weights between Hidden and Output Layer =
Number of biases in Output Layer =
Total Learnable Parameters Formula:
Parameter Scaling: Total trainable parameters increase sharply with every added hidden layer or additional node, significantly raising computational complexity.
Backpropagation Training Flow:
Forward Pass: Input data passes forward through layers to compute output predictions.
Backward Pass: Losses are calculated by comparing predictions against real labels, and error gradients move backwards through the network to update all weights and biases iteratively.
User Hyperparameters:
Number of hidden layers and number of neurons per hidden layer (determined based on sample data volume and signal noise level).
Choice of specific activation functions tailored to the learning task.
Convolutional Neural Networks (CNN) for Computer Vision
Image Representation: Images are mathematically stored as multi-dimensional matrices containing raw pixel brightness values.
Limitation of Standard Neural Networks:
Flattening an image matrix into a standard 1D vector and training a fully connected ANN yields low accuracy.
Standard networks discard spatial layouts, local structural arrangements, and pixel positional correlations.
CNN Functionality:
Specifically engineered to extract local spatial structures and temporal patterns in images using specialized sliding matrix filters.

Structural Layers of Convolutional Neural Networks
Convolutional Layer:
Extracts localized spatial patterns (e.g., eye colors, feather textures, edge directions, mouth shapes).

Convolution Mechanism:
Defines multiple local spatial filter matrices (kernels) of specific small dimensions.
Slides (convolves) each filter step-by-step across all spatial coordinates of the image.
Computes the dot product between the overlaying filter values and image pixel sub-matrices, adding a bias value:
Feature Maps: The output matrix generated by sliding a filter. Applying multiple unique filters yields multiple feature maps, each capturing distinct structural patterns (such as vertical lines or upward arrows).
Sample Convolution Output Matrix:
An image matrix processed by an upward arrow filter generates a feature map matrix:
Pooling Layer:
Downsampling: Aggregates local spatial details into smaller matrix representations, reducing computational cost while retaining critical visual features.
Max Pooling Procedure:
Segments input feature map matrices into smaller, non-overlapping grid sub-matrices (such as regions).
Extracts and keeps only the highest numeric value inside each sub-matrix region.
Max Pooling Numerical Worked Example:
Input Feature Map Matrix:
Sub-matrix 1 (Top-Left ):
Sub-matrix 2 (Top-Right ):
Sub-matrix 3 (Bottom-Left ):
Sub-matrix 4 (Bottom-Right ):
Resulting Downsampled Matrix:
Fully Connected Layer:
Stretches and flattens downsampled feature maps into a single 1D vector of length .
Inputs this vector into standard fully connected layers to compute final class output probabilities.
Transformer Architecture and Large Language Models
Transformer Definition: A neural network architecture designed to process sequential data and evaluate contextual relationships among all elements using attention mechanisms.
Origins and History:
Introduced in 2017 in the research paper Attention Is All You Need by Vaswani et al. (Google Brain and Google Research).
Originally created for machine translation tasks.
Achieved BLEU scores of on WMT 2014 English-to-German (improving state-of-the-art by over BLEU) and on WMT 2014 English-to-French after training for days across GPUs.
Enables highly efficient parallelized training across text, image, audio, video, and multimodal systems.

AI Systems Taxonomy Distinction:
Transformer: The foundational deep neural network architecture.
LLM (Large Language Model): A massive neural model implemented using Transformer architecture.
ChatGPT: A user-facing application built on top of underlying GPT language models.
Operational Dynamics of Large Language Models
Next-Token Prediction: Core objective where models process preceding context text to calculate a softmax probability distribution across the vocabulary, selecting the most probable next token.

Autoregressive Generation: Sequence synthesis occurs sequentially one token at a time; each newly predicted token is concatenated to the context window to determine subsequent tokens.
Self-Supervised Learning: Models learn directly from raw unlabelled text corpora where the text itself provides the ground-truth target tokens during continuous next-word prediction.
Tokenization: The process of breaking continuous document strings into lists of sub-word fragments or token strings.
String Example:
"Our meeting today was better than yesterday."Token Array:
["our", "meeting", "today", "was", "better", "than", "yesterday"]

Internal Transformer Components
Word Embeddings: Converts each text token into a dense mathematical vector of numbers encoding semantic parameters learned during training. Tokens occurring in similar semantic contexts project into close vector spaces:
"coffee""tea""car"
Positional Encoding: Injects sequence order information into token vector representations. Prevents identical words in different orders from having identical representations:
Self-Attention Mechanism:
Enables each token to dynamically compute relevance scores and gather information from all other tokens in the sequence.
Does not treat preceding words equally. For example, in the sentence "The bank approved the loan because it considered the business profitable.", self-attention assigns high attention weight connecting "it" to "bank" and low weight to uninformative words.
Query (), Key (), Value () Vector Formulations:
Query (): Vector representing the token seeking context (e.g., "loan").
Key (): Vector representing candidate contextual tokens (e.g., "the", "bank", "approved").
Value (): Information vector aggregated based on calculated pairwise dot-product attention weights.
Multi-Head Attention: Executes multiple self-attention operations concurrently in parallel, allowing models to track diverse contextual relationships across different sub-spaces.
Transformer Learned Parameters and Scaling
Learned Parameters across Components:
Token Embeddings: Numerical vector representations for every vocabulary token.
Positional Information: Parameters encoding token sequence positions.
Self-Attention: Matrix weights for Query (), Key (), Value (), and Attention Outputs.
Feed-Forward Networks: Dense matrix weights transforming intermediate token representations.
Output Layer: Projection weights utilized to compute final next-token probabilities.

Model Scale Parameters Comparison:
Llama 3.1 405B: total parameters | active parameters per token.
DeepSeek-V3: total parameters | active parameters per token.
Llama 4 Maverick: total parameters | active parameters per token.
Qwen3-235B-A22B: total parameters | active parameters per token.
Kimi K2: total parameters | active parameters per token.
Proprietary models (OpenAI GPT series, Anthropic Claude series, Google Gemini series) do not publicly disclose exact parameter numbers.

Image Generation via Diffusion Models
Diffusion Model Concept: Generative models that synthesize high-fidelity image or video content by progressively converting unstructured random Gaussian noise into coherent visual structures.

Forward Diffusion Training Process:
Systematic noise addition: Gaussian noise is added step-by-step to real training images over time steps ().
By Step , the original image transitions entirely into unpatterned pure noise.
A neural network learns to accurately predict and estimate the noise added at each step.

Reverse Diffusion Generation Process:
Generation begins with pure random noise.
The trained neural model iteratively predicts and subtracts noise step-by-step, restoring visual details.
Typically requires approximately sequential denoising steps to synthesize an image.
Text Guidance: Input text prompts guide cross-attention mechanisms during the denoising steps to direct the emerging image toward matching specified prompt concepts.
