ML System Design

Design a RAG system for RCA (root cause analysis) on production logs. On-call engineer asks: “Why is checkout failing?” Seconds matter.

TB-scale logs, millions near-duplicate lines, exact tokens (error codes, service names, trace IDs) hide the signal.

Why vanilla RAG fails: Embeddings blur exact identifiers into fuzzy vectors. “stripe-gateway timeout” becomes semantic mush, loses diagnostic precision.

Your fix: Hybrid retrieval.

• Keyword first (BM25) → catches exact error codes, service names, IDs

• Dense second (embeddings) → semantic bridges (“checkout timeout” ↔ “payment latency”)

• Rerank with cross-encoder

• Return log spans with citations

Why it works: Keywords nail the noise floor, dense carries the semantic hops LLM can’t make alone.

Follow-up prep: chunking strategy (trace ID, not token count), template extraction (Drain), evals (retrieval vs generation), online metrics (deflection, thumbs).

Do I need hybrid retrieval, or is dense-only fine? How do I decide?

Decide by how much signal lives in exact tokens.

• Mostly exact identifiers (logs, code, IDs, error codes) → hybrid is load-bearing, keyword does the heavy lifting

• Mostly prose (HR docs, policies, guides) → dense carries it, keyword rescues edge cases (section numbers, acronyms, product names)

• Default to hybrid; weight it by corpus. BM25 = exact, dense = meaning, fuse with RRF.

Interview line: “I’d default to hybrid but weight it by how much of the signal is exact tokens vs semantics.”

What retrieval actually is? lexical vs dense

Lexical retrieval (BM25). Matches literal words. If you search stripe-gateway, it finds documents containing the string stripe-gateway. It scores by: how rare is this term across the corpus (rare = valuable), how often does it appear in this doc, how long is the doc. That’s basically it. It cannot know that “car” and “automobile” are related — zero semantic understanding.

Dense retrieval (embeddings). A model maps text to a vector, say 768 numbers. Similar meaning lands near each other in that space. So “checkout is broken” and “payment flow failing” score high together even with zero shared words. That’s the superpower.

Q: You’re indexing product documentation for a normal enterprise RAG — policy docs, onboarding guides, HR manuals. Would you still bother with hybrid retrieval, or is dense-only fine here? And say why.

If it’s prose — HR manuals, policy docs, onboarding guides — most queries are semantic, like ‘what’s the parental leave policy,’ and dense retrieval alone gets you most of the way. Where it slips is exact references: ‘policy 4.2.1’ versus ‘4.3.1,’ acronyms, a product name the embedding model never saw in training. Those embed to nearly identical vectors, so dense retrieval can’t tell them apart. So even here I’d add a keyword pass — not because dense is broken, but to rescue that exact-match tail.

If it’s logs or code, the calculus flips. The corpus is almost entirely exact identifiers — error codes, trace IDs, service names — which is exactly what embeddings compress away. There, keyword retrieval is doing the heavy lifting and dense is the supporting layer for semantic hops.

So I default to hybrid, but I weight it by corpus. Same architecture, different center of gravity. The principle I’m applying is: match retrieval to how much of your signal is exact tokens versus semantics — and I’d confirm that empirically with a retrieval eval set rather than guessing.

Chunking — why it matters, and how do I chunk when the natural unit is too big?

Why it’s #1: retrieval can only return a chunk you made. Bad boundary = lost before embeddings even run. Sets the ceiling.

Principle: chunk on natural boundaries, not fixed length. “512 tokens” is a smell (cuts mid-idea).

• Logs → trace ID / incident window

• Prose → section/heading + slight overlap

• Code → function/class

When the unit is too big (huge trace):

1. Dedup first — template extraction (Drain) collapses near-duplicate lines. Often kills the size problem.

2. Hierarchical / parent-child — child chunk per service span + one summary parent for the whole trace. Retrieve parent to route, drill to child for detail. (a.k.a. small-to-big.)

3. Overlap is the prose tool; dedup + hierarchy are the log-native tools.

Tradeoff: too big = diluted embedding + cost; too small = severed context. One coherent idea per chunk.