Generative AI Exam 2

RAG purpose

Ground LLM outputs in external knowledge to reduce hallucinations

RAG pipeline

Index, retrieve, generate

Index steps

Ingest and clean, chunk, embed, and store (vector DB + metadata)

Ingest and clean

Convert to plain texts and clean so it's readable (extract text, fix broken encodings and headers/footers, capture source URL, last-updated, and owner as metadata)

Chunk

Segment text into digestible chunks with overlap (~200-500 token units with small overlaps and attached metadata)

Embed

Compute embeddings for each chunk (numerical representation that captures meaning)

Store

Vector database stores, indexes, and searches embeddings

Retrieval

Encode query, compute similarity scores, and retrieve top-K most relevant chunks to build context

Similarity scores between query vector and document chunks

Cosine similarity or dot product

Sparse approach

Token matches used for keyword queries and exact matches

Dense approach

Semantic vectors used for similarity search and semantic QA

Hybrid approach

Combines sparse and dense signals for improved recall and precision

Precision

If a model predicts a positive outcome, how likely is that prediction to be correct?

Recall

Given all relevant instances, how many did the model actually detect?

Generate prompt

Query + retrieved chunks + instructions

Hit rate

Fraction of queries where the correct document appears

Mean reciprocal rank (MRR)

How early does the first correct answer appear? (1/rank of first relevant result)

Normalized discounted cumulative gain (NDCG)

Weighted rating of the entire ranking, not just the first hit

Exact matching (EM)/F1

Exact match and overlap of generated answers

Pitfalls of oversized chunks

Matches are vague and the context window is stuffed with extra fluff

Pitfalls of chunks that are too small

Passages lose meaning, the model gets fragmented without enough context

Purpose of post-training

Tailor outputs to domain needs (format, tone, policy, tool-use), cheaper than pre-training, works with RAG

Supervised Fine-Tuning (SFT)

Next-token cross-entropy on target responses (mask system/prompt as needed; length-normalize)

SFT data

instruction, response pairs, normalize templates, deduplication, filter unsafe/personally identifiable information (PII), tag metadata

Parameter-efficient fine tuning (PEFT)

Low-Rank Adaptation (LoRA) or Quantized LoRA (QLoRA) to train small, reusable modules on top of frozen base weights

RLHF (Reinforcement Learning from Human Feedback)

Reward model + PPO with KL regularization to a reference

Not studied (48)

You haven't studied these terms yet!

Select these 48

RLHF benefits

RLHF drawbacks

Higher operational complexity

LoRA

Inserts low-rank adapters into attention/MLP, trains only adapters causing massive parameter savings with small quality loss

QLoRA

4-bit quantized base + LoRA adapters to reduce memory further

Hyperparameters to reason about

Epochs, learning rate, warmup steps/ratio, weight decay, effective batch size (batch * grand accumulation)

Overfitting

Aggressive LR/epochs lead to repetition or echoing

SFT deliverables pattern

Saved adapters/tokenizer, prompt template, inference function for product integration

DPO (Direct Preference Optimization)

Logistic loss on log-probability gaps (chosen vs. rejected) with reference correction, beta controls preference strength

DPO data

Prompt, chosen, rejected , pairs; uses frozen reference policy

Reference policy

Compares gaps vs. frozen base/SFT model; beta sweeps are implementation-dependent

When to use DPO

complements SFT for subjective qualities (helpfulness, tone, refusals)

DPO evaluation

formatting adherence, pairwise win-rate, safety/jailbreak tests, business KPIs

DPO vs. RLHF

Avoids a separate reward model and PPO loop; turns alignment into a direct logistic objective

Typical workflow

SFT, collect preference pairs, DPO fine-tuning, evaluate HHH (helpful, honest, harmless) + task KPIs

Tooling

TRL's 'DPOTrainer' (with beta and other hyperparameters as in SFT) for pairwise preferences

AI agents

Goal-directed loops that plan, call tools, write/execute code, and refine using feedback

Sponsored By LinkedIn for Marketing

Watch & learn: Smash your lead & conversion goals with LinkedIn AdsNot all ad formats are created equal. Watch our free webinars to learn which are best for driving leads and conversions on LinkedIn: https://lnkd.in/d_wF8gpPLearn More

Vibe coding

Specify the "feel" and constraints of the solution (architecture, style, design rules, performance/latency targets, acceptance criteria) rather than line-by-line instructions, guide with examples and guardrails

Vibe coding workflow

Define vibe, provide scaffolds, add exemplars, set guardrails, iterate, log decisions, lock rules

Define vibe

Goals, constraints, tests, non-goals

Provide scaffolds

Interfaces, stubs, layout, freeze public APIs

Add examplars

Code/style snippets to follow/avoid

Set guardrails

Tests, types, linters, CI as hard checks

Iterate

Generate, run, review and refine in tight cycles

Log decisions

Record rationale and update criteria

Lock rules

CI/pre-commit to enforce vibe

Limitations of vibe coding

Ambiguity and drift, non-determinism under parallel edits, hallucinated APIs, style inconsistency, security/secret mishandling, missing refactors, dependency/version surprises

Steering vs. Fine-tuning (SFT/DPO)

Fine-tuning has more durable changes, but slower, coarse (changes weights) adn compute-intensive

Interpretation

Uses sparse autoencoders (SAEs) to untangle activations into human-named features (e.g., polite tone, numbers, lists), then inspects the residual stream encodings

Interpreting benefits

Enables bias audits, compliance checks, debugging, etc.

Steering

Control model behavior along interpretable axes without retraining

2 common steering methods

SAE feature steering (select a feature and increase/decrease activation) and steering vectors (activation addition adds a learned direction to the hidden state)

Steering benefits

Brand-tone control, toxicity reduction, truthfulness nudges, region/policy alignment, and rapid iteration without costly retraining

Residual stream

Residual connection combines output from the MLP layer

Autoencoder

Compresses and depresses activations with the goal of minimizing reconstruction loss

Benefits of SAE feature steering

Interpretability, transparency, fine-grained control, and knowledge audit

Steering vector in activation engineering

In place of a trained SAE, you can use contrasting prompts to be used on the residual stream

Alpha in steering vectors

Controls how hard and in which direction you push along the steering vector

Operational advantages of steering vector benefits

Instant, reversible control at inference, direct and reproduceable effect (less context/wWording sensitive than prompting) and easier than fine tuning (no labels, no weight updates, immediate rollback)

Fine-grained control of steering vector

Adjust alpha for strength and flip sign to reverse behavior, cheaper and more precise than fine-tuning which needs new hyperparameter runs and data curations for each adjustment

Interpretability and governance of steering vectors

Each vector is a contrast labelable in plain english; simple to review, share, and rollback

Steering vector applications

Customer support tone shift, safety/refusal and overclaiming controls for compliance, and brand voice and regional personalization at scale

Epoch

One full pass through the training dataset (higher = more aggressive learning)

Learning rate

How fast the model learns (higher = more aggressive learning)

Warmup steps/ratio

A short startup phase where learning speed gradually increases to avoid sudden shocks (more = more conservative training)

Weight decay

A small penalty that nudges weights toward zero to prevent overfitting (higher = more conservative training)

Effective batch size

The total number of examples used before each update (larger = more aggressive learning)

KL divergence

Used in RLHF to penalize diverging too far from the original model

PPO

Proximal Policy Optimization (reinforcement learning model used in RLHF)

```