Large Language Models and Advanced AI Development Fundamentals

Tokens and the Currency of LLMs

• What Is a Token?: A token is the basic unit of text that a Large Language Model (LLM) processes. It is the unit against which everything else in the life of a model is measured. Depending on the specific tokenizer and language used, a token can be:

  • A full word: "hello" -> $1$ token.

  • A subword fragment: "tokenization" -> "token" + "ization" -> $2$ tokens.

  • A punctuation mark: "." -> $1$ token.

  • A single character: Common for rare words or non-English text.

• Rule of Thumb: $1,000$ tokens is approximately equal to $750$ English words.

• Model Specificity: Different models use different tokenizers; therefore, the same sentence may yield different token counts depending on the model. Tools like the OpenAI Tokenizer can be used to visualize this.

• Why Tokens Matter:

  • Cost: Most LLM APIs charge based on the number of tokens in both input (prompt) and output (completion). Long system prompts or request for detailed essays increase costs significantly.

  • Speed: More input tokens require more computation before the first word of a response appears, leading to higher latency.

  • Context Window: Every model has a hard limit on the total number of tokens it can process at once.

The Context Window and the "Lost in the Middle" Problem

• Definition: The context window is the maximum number of tokens (covering both input and output) that a model can process and consider at one time. Once this limit is reached, older information is "pushed out" to make room for new tokens, causing the model to "forget" earlier parts of the conversation.

• Typical Context Window Sizes:     | Model | Approximate Context Window |     | :--- | :--- |     | GPT-3.5 Turbo | $16,385$ tokens |     | GPT-4o | $128,000$ tokens |     | Claude 3.5 Sonnet | $200,000$ tokens |     | Gemini 1.5 Pro | $2,000,000$ tokens |

• Where This Shows Up:

  • Long customer support sessions where the model forgets the initial complaint.

  • RAG systems where long retrieved documents push out system instructions.

  • Agent loops where tool outputs accumulate and overflow the window.

• The "Lost in the Middle" Phenomenon: Research shows that LLMs pay the most attention to information at the very beginning and the very end of a long input. Information placed in the middle receives significantly less reliable attention, creating a U-shaped attention pattern.

• Design Solutions:

  • Place critical instructions at the start or end of the prompt.

  • Use re-ranking in RAG systems to surface relevant chunks at the top.

  • Summarize older conversation history rather than appending everything indefinitely.

Self-Supervised Learning: How LLMs Learn

• The Training Data Problem: Training an LLM requires billions of examples, making human-labeled data impractical. Self-supervised learning allows the model to generate its own labels from raw, unlabeled data.

• The Objective: For LLMs, the self-supervised objective is next-token prediction. The model reads text, predicts the next token, compares it to the actual data, and updates its weights based on the error. This process repeats across billions of examples from books, websites, and code repositories.

• Emergent Capabilities: By predicting the next word, the model internalizes:

  • Grammar and syntax.

  • Facts about the world.

  • Reasoning patterns and logical structure.

  • Common sense relationships.

• Comparison of Learning Approaches:     | Learning Type | Who Provides Labels? | Example |     | :--- | :--- | :--- |     | Supervised | Human annotators | Labeling email as spam or not spam |     | Unsupervised | No labels (find patterns) | Clustering documents by topic |     | Self-supervised | The data itself | Predicting the next word from preceding words |

Model Generation and Properties

• Autoregressive Text Generation: Models like GPT, Claude, and Gemini generate output one token at a time. Each new token is conditioned on the full input plus everything already generated. Generation is strictly left-to-right; the model cannot go back and revise earlier output without regenerating.

• Temperature: Controls how the model samples the next token. A higher temperature leads to more varied, creative output, while a lower temperature is more deterministic and consistent.

• Statelessness: Every API call is independent. The model has no memory of previous calls. To simulate a conversation, the application layer must include the full conversation history in every new API call. The model is effectively a function: . All state is gone once the function returns.

Model Categorization and Deployment

• Foundation Models: Large models trained on broad, general-purpose datasets (e.g., GPT-4, Claude, Llama). They serve as a base for many different tasks.

• Fine-Tuning: Adjusting a foundation model on a narrow, task-specific dataset (e.g., medical records) to specialize its behavior without the cost of training from scratch.

• Narrow AI vs. AGI:

  • Narrow AI (Weak AI): Highly capable in specific domains (generating code, text) but lacks genuine understanding or autonomous goals. Performance degrades outside the trained distribution.

  • AGI (Artificial General Intelligence): A hypothetical system that can perform any intellectual task a human can, learning continuously and transferring knowledge across contexts. AGI does not yet exist.

• Quantization: A technique to reduce model size for practical deployment by converting high-precision weights (e.g., 32-bit floats) to lower-precision formats (e.g., 8-bit or 4-bit integers).     | Precision | Memory ($7B$ model) | Typical Use |     | :--- | :--- | :--- |     | FP32 (32-bit float) | $\text{~28 GB}$ | Research/training |     | FP16 (16-bit) | $\text{~14 GB}$ | Production training |     | INT8 (8-bit integer) | $\text{~7 GB}$ | Server deployment |     | INT4 (4-bit integer) | $\text{~3.5 GB}$ | Consumer hardware (e.g., Ollama) |

Advanced Prompt Engineering Techniques

• Prompt Sensitivity: Small wording changes can activate different patterns in the model. Treating prompts like code (versioning, testing) is essential.

• Delimiters: Characters or tags (e.g., triple quotes """, XML tags <tag></tag>) used to separate instructions from user-supplied data. This prevents Prompt Injection (overriding instructions) and Prompt Leaking (extracting system prompts).

• Self-Consistency: Improving reasoning reliability by sampling multiple independent paths and selecting the most frequent final answer (majority vote).

• Grounding: Constraining the LLM to generate answers based only on provided reference documents to reduce hallucinations.

• Output Guardrails: Quality gates applied after generation to validate responses. Types include content filtering, policy enforcement, and PII (Personally Identifiable Information) detection.

Advanced RAG Systems

• Sparse vs. Dense Retrieval:

  • Sparse (Keyword-based): Uses algorithms like BM25 to find exact keyword matches (e.g., IDs, product codes).

  • Dense (Semantic/Vector-based): Uses embeddings to find conceptual matches regardless of exact wording.

  • Hybrid Search: Combines both methods and merges results using Reciprocal Rank Fusion (RRF): $RRF\text{ score} = \frac{1}{k + \text{rank}_i}$ (where $k$ is usually $60$).

• Re-Ranking: A two-stage process where a fast retriever gets top-50 results, and a precise Cross-Encoder re-scores them for the final top-5. Bi-encoders (retrieval) are fast but less aware of query-document interactions compared to Cross-encoders.

• Query Decomposition: Breaking a complex multi-part question into simpler sub-questions. This can be Parallel (independent sub-questions) or Sequential (multi-hop where one answer leads to the next query).

• RAG Evaluation (RAGAS Metrics):

  • Context Precision: Are relevant chunks at the top of the list?

  • Context Recall: Was all the needed information actually retrieved?

  • Answer Relevancy: Does the response address the user's intent?

  • Faithfulness: Is the answer factually grounded in the provided context?