GenAI Architecture and Neural Networks

Hierarchy of Artificial Intelligence Systems

  • Artificial Intelligence (AI): The overarching field encompassing all software systems and machines designed to perform tasks that typically require human intelligence.

  • Machine Learning (ML): A subfield of AI focused on algorithms that analyze data, learn underlying patterns, and make data-driven predictions without explicit rule-based programming.

  • Deep Learning (DL): An advanced subset of ML utilizing multi-layered artificial neural networks designed to process complex data abstractions and high-dimensional representations.

  • Generative AI (GenAI): A specialized domain of deep learning focused on synthesizing brand-new content (including natural text, images, audio, video, and code) based on statistical patterns learned from vast training datasets.

Artificial Intelligence Taxonomy

Biological Foundations of Neural Computation

  • Biological Inspiration: Artificial neural networks are computational models inspired by the structural and functional properties of biological human brains.

  • Neurons: Biological nerve cells serving as fundamental processing units.

    • Chemical transmitters within cerebral fluid alter the electrical potential across neuron membranes.

    • When the electrical potential crosses a specific physiological threshold, the neuron fires (spikes), propagating an electrical pulse of fixed strength down its path.

  • Axons and Synapses:

    • Axons branch out to connect one neuron to thousands of neighboring neurons.

    • Connection strengths reflect statistical correlations between firing events.

    • Synaptic connections that fire simultaneously strengthen over time; inactive or non-correlated connections weaken and atrophy.

    • Every individual neuron performs discrete information input, processing, and output operations.

Biological Neuron Signal TransmissionBiological Neural Network Interconnections

Mathematical Architecture of Artificial Neurons

  • Artificial Neuron Elements:

    • Inputs (x1,x2,…,xkx_1, x_2, \dots, x_k): Signals received from external sources or preceding neural layers.

    • Weights (w1,w2,…,wkw_1, w_2, \dots, w_k): Scalar multipliers representing the connection strength assigned to each input signal.

    • Bias (bb): A constant intercept term added to the weighted input sum, functioning similarly to the intercept in linear regression models.

Single Artificial Neuron Diagram
  • Summation Equation:

    • The linear combination of inputs, weights, and bias produces an intermediate output hh:   h=∑i=1kwixi+bh = \sum_{i=1}^{k} w_i x_i + b

  • Activation Functions:

    • Mathematical transformations applied to the linear combination hh to introduce non-linearity into the network:

    • Threshold Function / Perceptron Step: Determines binary activation (fire vs. not fire) via a step function Φ(z)\Phi(z).

    • Identity Function: Passes the input directly without alteration: f(z)=zf(z) = z

    • Sigmoid Function: Squeezes real-valued numbers into a continuous range between 00 and 11:       σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}

    • Hyperbolic Tangent (Tanh) Function: Maps inputs to a zero-centered continuous range between −1-1 and 11:       tanh⁡(z)=ez−e−zez+e−z\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}

    • Rectified Linear Unit (ReLU): Outputs zero for negative inputs and linear identity for positive inputs:       ReLU(z)=max⁡(0,z)\text{ReLU}(z) = \max(0, z)

    • Leaky ReLU: Prevents dying neurons by permitting a small non-zero slope (0.10.1) for negative inputs:       f(z)={0.1zif z<0zif z>0f(z) = \begin{cases} 0.1z & \text{if } z < 0 \\ z & \text{if } z > 0 \end{cases}

    • Exponential Linear Unit (ELU): Smooths negative values exponentially:       f(z)={ez−1if z<0zif z>0f(z) = \begin{cases} e^z - 1 & \text{if } z < 0 \\ z & \text{if } z > 0 \end{cases}

    • Softmax Function: Converts raw output logits into a normalized probability distribution across multiple classes:       Softmax(zj)=ezj∑iezi\text{Softmax}(z_j) = \frac{e^{z_j}}{\sum_{i} e^{z_i}}

    • Softplus Function: A smooth, continuous approximation of ReLU:       Softplus(z)=log⁡(1+ez)\text{Softplus}(z) = \log(1 + e^z)

Multi-Layer Artificial Neural Networks (ANN)

  • Layered Structure:

    • Input Layer: Entry nodes representing observable input variables or feature parameters.

    • Hidden Layer(s): Intermediate computational processing stages that transform input signals into high-level abstraction representations ("thoughts"). Hidden layers operate as unobservable mathematical black boxes.

    • Output Layer: Final layer producing output predictions, discrete label classifications, or estimated scalar values.

Multi-Layer Neural Network Topology
  • Multi-Class Classification Structure:

    • Every output node computes a linear combination of hidden signals passed through an activation function.

    • For a multi-class task with 55 target classes, the network contains 55 distinct output nodes.

    • Output node ii evaluates the class conditional probability P(y=classi)P(y = \text{class}_i).

Five Class Output Neural Network Structure
  • Functional Scope and Independence:

    • Neurons within a network do not function independently; each node integrates weighted signals from all connected preceding nodes, establishing interdependent global network computations.

    • Neural networks support both classification (discrete categorizations) and regression (continuous target variables) objectives.

Parameter Estimation and Network Training

  • Trainable Network Parameters:

    • Weights: One weight parameter exists for every directed connection arrow between nodes.

    • Biases: One bias parameter exists for every single neuron situated in hidden or output layers.

  • Single Hidden Layer Parameter Calculation:

    • Let LL be the number of input nodes.

    • Let MM be the number of hidden nodes.

    • Let NN be the number of output nodes.

    • Number of weights between Input and Hidden Layer = L×ML \times M

    • Number of biases in Hidden Layer = MM

    • Number of weights between Hidden and Output Layer = M×NM \times N

    • Number of biases in Output Layer = NN

    • Total Learnable Parameters Formula:     Total Parameters=(L+1)×M+(M+1)×N\text{Total Parameters} = (L + 1) \times M + (M + 1) \times N

  • Parameter Scaling: Total trainable parameters increase sharply with every added hidden layer or additional node, significantly raising computational complexity.

  • Backpropagation Training Flow:

    • Forward Pass: Input data passes forward through layers to compute output predictions.

    • Backward Pass: Losses are calculated by comparing predictions against real labels, and error gradients move backwards through the network to update all weights and biases iteratively.

  • User Hyperparameters:

    • Number of hidden layers and number of neurons per hidden layer (determined based on sample data volume and signal noise level).

    • Choice of specific activation functions tailored to the learning task.

Convolutional Neural Networks (CNN) for Computer Vision

  • Image Representation: Images are mathematically stored as multi-dimensional matrices containing raw pixel brightness values.

  • Limitation of Standard Neural Networks:

    • Flattening an image matrix into a standard 1D vector and training a fully connected ANN yields low accuracy.

    • Standard networks discard spatial layouts, local structural arrangements, and pixel positional correlations.

  • CNN Functionality:

    • Specifically engineered to extract local spatial structures and temporal patterns in images using specialized sliding matrix filters.

Convolutional Neural Network Architecture Pipeline

Structural Layers of Convolutional Neural Networks

  • Convolutional Layer:

    • Extracts localized spatial patterns (e.g., eye colors, feather textures, edge directions, mouth shapes).

Local Feature Focused Region
  • Convolution Mechanism:

    • Defines multiple local spatial filter matrices (kernels) of specific small dimensions.

    • Slides (convolves) each filter step-by-step across all spatial coordinates of the image.

    • Computes the dot product between the overlaying filter values and image pixel sub-matrices, adding a bias value:       Feature Map Value=Dot Product(Filter,Image Segment)+Bias\text{Feature Map Value} = \text{Dot Product}(\text{Filter}, \text{Image Segment}) + \text{Bias}

  • Feature Maps: The output matrix generated by sliding a filter. Applying multiple unique filters yields multiple feature maps, each capturing distinct structural patterns (such as vertical lines or upward arrows).

  • Sample Convolution Output Matrix:

    • An image matrix processed by an upward arrow filter generates a feature map matrix:       (3113111101102112)\begin{pmatrix} 3 & 1 & 1 & 3 \\ 1 & 1 & 1 & 1 \\ 0 & 1 & 1 & 0 \\ 2 & 1 & 1 & 2 \end{pmatrix}

    • Pooling Layer:

  • Downsampling: Aggregates local spatial details into smaller matrix representations, reducing computational cost while retaining critical visual features.

  • Max Pooling Procedure:

    • Segments input feature map matrices into smaller, non-overlapping grid sub-matrices (such as 2×22 \times 2 regions).

    • Extracts and keeps only the highest numeric value inside each sub-matrix region.

  • Max Pooling Numerical Worked Example:

    • Input 4×44 \times 4 Feature Map Matrix:       (3113111101102112)\begin{pmatrix} 3 & 1 & 1 & 3 \\ 1 & 1 & 1 & 1 \\ 0 & 1 & 1 & 0 \\ 2 & 1 & 1 & 2 \end{pmatrix}

    • Sub-matrix 1 (Top-Left 2×22 \times 2): (3111)→Maximum=3\begin{pmatrix} 3 & 1 \\ 1 & 1 \end{pmatrix} \rightarrow \text{Maximum} = 3

    • Sub-matrix 2 (Top-Right 2×22 \times 2): (1311)→Maximum=3\begin{pmatrix} 1 & 3 \\ 1 & 1 \end{pmatrix} \rightarrow \text{Maximum} = 3

    • Sub-matrix 3 (Bottom-Left 2×22 \times 2): (0121)→Maximum=2\begin{pmatrix} 0 & 1 \\ 2 & 1 \end{pmatrix} \rightarrow \text{Maximum} = 2

    • Sub-matrix 4 (Bottom-Right 2×22 \times 2): (1012)→Maximum=2\begin{pmatrix} 1 & 0 \\ 1 & 2 \end{pmatrix} \rightarrow \text{Maximum} = 2

    • Resulting Downsampled 2×22 \times 2 Matrix:       (3322)\begin{pmatrix} 3 & 3 \\ 2 & 2 \end{pmatrix}

    • Fully Connected Layer:

  • Stretches and flattens downsampled K×KK \times K feature maps into a single 1D vector of length K2K^2.

  • Inputs this vector into standard fully connected layers to compute final class output probabilities.

Transformer Architecture and Large Language Models

  • Transformer Definition: A neural network architecture designed to process sequential data and evaluate contextual relationships among all elements using attention mechanisms.

  • Origins and History:

    • Introduced in 2017 in the research paper Attention Is All You Need by Vaswani et al. (Google Brain and Google Research).

    • Originally created for machine translation tasks.

    • Achieved BLEU scores of 28.428.4 on WMT 2014 English-to-German (improving state-of-the-art by over 22 BLEU) and 41.841.8 on WMT 2014 English-to-French after training for 3.53.5 days across 88 GPUs.

    • Enables highly efficient parallelized training across text, image, audio, video, and multimodal systems.

Attention Is All You Need Paper Abstract
  • AI Systems Taxonomy Distinction:

    • Transformer: The foundational deep neural network architecture.

    • LLM (Large Language Model): A massive neural model implemented using Transformer architecture.

    • ChatGPT: A user-facing application built on top of underlying GPT language models.

Operational Dynamics of Large Language Models

  • Next-Token Prediction: Core objective where models process preceding context text to calculate a softmax probability distribution across the vocabulary, selecting the most probable next token.

Next Word Prediction Probability Mechanism
  • Autoregressive Generation: Sequence synthesis occurs sequentially one token at a time; each newly predicted token is concatenated to the context window to determine subsequent tokens.

  • Self-Supervised Learning: Models learn directly from raw unlabelled text corpora where the text itself provides the ground-truth target tokens during continuous next-word prediction.

  • Tokenization: The process of breaking continuous document strings into lists of sub-word fragments or token strings.

    • String Example: "Our meeting today was better than yesterday."

    • Token Array: ["our", "meeting", "today", "was", "better", "than", "yesterday"]

Self Supervised Training Process Visualized

Internal Transformer Components

  • Word Embeddings: Converts each text token into a dense mathematical vector of numbers encoding semantic parameters learned during training. Tokens occurring in similar semantic contexts project into close vector spaces:

    • "coffee" →[0.8,0.4,0.7]\rightarrow [0.8, 0.4, 0.7]

    • "tea" →[0.7,0.3,0.6]\rightarrow [0.7, 0.3, 0.6]

    • "car" →[0.1,0.9,0.2]\rightarrow [0.1, 0.9, 0.2]

  • Positional Encoding: Injects sequence order information into token vector representations. Prevents identical words in different orders from having identical representations:   Token Input Representation=Word Embedding+Positional Encoding\text{Token Input Representation} = \text{Word Embedding} + \text{Positional Encoding}

  • Self-Attention Mechanism:

    • Enables each token to dynamically compute relevance scores and gather information from all other tokens in the sequence.

    • Does not treat preceding words equally. For example, in the sentence "The bank approved the loan because it considered the business profitable.", self-attention assigns high attention weight connecting "it" to "bank" and low weight to uninformative words.

    • Query (QQ), Key (KK), Value (VV) Vector Formulations:

    • Query (QQ): Vector representing the token seeking context (e.g., "loan").

    • Key (KK): Vector representing candidate contextual tokens (e.g., "the", "bank", "approved").

    • Value (VV): Information vector aggregated based on calculated pairwise dot-product attention weights.

  • Multi-Head Attention: Executes multiple self-attention operations concurrently in parallel, allowing models to track diverse contextual relationships across different sub-spaces.

Transformer Learned Parameters and Scaling

  • Learned Parameters across Components:

    • Token Embeddings: Numerical vector representations for every vocabulary token.

    • Positional Information: Parameters encoding token sequence positions.

    • Self-Attention: Matrix weights for Query (QQ), Key (KK), Value (VV), and Attention Outputs.

    • Feed-Forward Networks: Dense matrix weights transforming intermediate token representations.

    • Output Layer: Projection weights utilized to compute final next-token probabilities.

Learned Parameters in Transformer Components
  • Model Scale Parameters Comparison:

    • Llama 3.1 405B: 405 billion405\text{ billion} total parameters | 405 billion405\text{ billion} active parameters per token.

    • DeepSeek-V3: 671 billion671\text{ billion} total parameters | 37 billion37\text{ billion} active parameters per token.

    • Llama 4 Maverick: 400 billion400\text{ billion} total parameters | 17 billion17\text{ billion} active parameters per token.

    • Qwen3-235B-A22B: 235 billion235\text{ billion} total parameters | 22 billion22\text{ billion} active parameters per token.

    • Kimi K2: 1 trillion1\text{ trillion} total parameters | 32 billion32\text{ billion} active parameters per token.

    • Proprietary models (OpenAI GPT series, Anthropic Claude series, Google Gemini series) do not publicly disclose exact parameter numbers.

Recent Model Parameter Scaling Comparison

Image Generation via Diffusion Models

  • Diffusion Model Concept: Generative models that synthesize high-fidelity image or video content by progressively converting unstructured random Gaussian noise into coherent visual structures.

Evolution of Generative Media Output Quality
  • Forward Diffusion Training Process:

    • Systematic noise addition: Gaussian noise is added step-by-step to real training images over TT time steps (Step 0→Step 1→Step 2→⋯→Step T\text{Step } 0 \rightarrow \text{Step } 1 \rightarrow \text{Step } 2 \rightarrow \dots \rightarrow \text{Step } T).

    • By Step TT, the original image transitions entirely into unpatterned pure noise.

    • A neural network learns to accurately predict and estimate the noise added at each step.

Forward Noising Diffusion Process
  • Reverse Diffusion Generation Process:

    • Generation begins with pure random noise.

    • The trained neural model iteratively predicts and subtracts noise step-by-step, restoring visual details.

    • Typically requires approximately ∼100\sim 100 sequential denoising steps to synthesize an image.

    • Text Guidance: Input text prompts guide cross-attention mechanisms during the denoising steps to direct the emerging image toward matching specified prompt concepts.

Reverse Denoising Generation Steps Visualized