LLM Fundamentals¶
Last reviewed: October 2026
What's changed recently
- Reasoning models are standard. Frontier models spend extra "thinking" tokens before answering (test-time compute), usually with an effort or budget setting. More thinking buys accuracy on hard tasks at the cost of latency and money.
- Sampling knobs don't always apply. Some reasoning models fix or ignore temperature and top-p, so reliability comes from structured outputs, validation, and evals rather than temperature alone.
- Mixture-of-Experts (MoE) is the common frontier architecture, including most large open-weight models: many parameters in total, only a fraction active per token.
- Post-training goes beyond SFT and RLHF. Preference optimization (DPO and variants) and reinforcement fine-tuning with verifiable rewards are now common adaptation tools, alongside LoRA.
The concepts every GenAI engineer should be fluent in: how models represent text, generate it, and how you adapt them. This underpins every other topic.
From text to tokens to output¶
flowchart LR
T[Text] --> TOK[Tokenizer]
TOK --> EMB[Embeddings]
EMB --> TR[Transformer layers - attention]
TR --> LOGITS[Next-token probabilities]
LOGITS --> SAMP[Sampling: temp / top-p]
SAMP --> OUT[Generated token]
OUT -->|append, repeat| TR
- Tokens — models read/write tokens (~¾ of a word each), not characters. Cost and context limits are measured in tokens.
- Embeddings — tokens/text map to vectors capturing meaning (basis of RAG & vector search).
- Transformer + attention — each token attends to others; self-attention is the core mechanism.
- Autoregressive generation — the model predicts the next token, appends it, and repeats.
Sampling controls¶
| Parameter | Effect |
|---|---|
| Temperature | Higher = more random/creative; lower = more deterministic (some reasoning models ignore it) |
| Top-p (nucleus) | Sample from the smallest set of tokens summing to p |
| Top-k | Sample from the k most likely tokens |
| Max tokens | Caps output length |
| Stop sequences | End generation at a marker |
For factual/structured tasks → low temperature. For brainstorming → higher.
Context window¶
The max tokens (input + output) a model can consider at once. Larger windows fit more context but cost more and can dilute attention ("lost in the middle") — see Context Engineering.
Ways to adapt a model¶
flowchart TB
Q[Need to change...] --> K{Knowledge or behavior?}
K -- fresh/private knowledge --> RAG[RAG - retrieve at query time]
K -- style/format/behavior --> FT[Fine-tuning]
K -- quick steering --> PROMPT[Prompting / few-shot]
FT --> LORA[LoRA / PEFT - cheap fine-tune]
| Approach | Changes | Cost | Use when |
|---|---|---|---|
| Prompting / few-shot | Behavior at call time | Cheapest | Quick steering, format |
| RAG | Adds knowledge | Low | Fresh/private/changing facts + citations |
| Fine-tuning | Bakes in behavior/style | High | Consistent style/format, narrow domain |
| LoRA / PEFT | Efficient fine-tune | Medium | Fine-tune without full retrain |
| Preference / RL fine-tuning | Aligns outputs to preferences or graded rewards | High | Tasks with clear graders (DPO, reinforcement fine-tuning) |
Rule of thumb: need knowledge → RAG; need behavior/style → fine-tune; need quick change → prompt. Most enterprise apps start with RAG.
Other essentials¶
- Hallucination — plausible but wrong output; mitigate with grounding (RAG), "say you don't know," low temp, and verification.
- Context vs parametric knowledge — what's in the prompt vs what's baked in the weights (and possibly stale).
- Embeddings ≠ the LLM — a separate (often smaller) model produces vectors for retrieval.
- Cost/latency scale with tokens (input + output, including hidden reasoning tokens) and model size.
- Reasoning vs non-reasoning models — reasoning models trade latency and cost for accuracy on multi-step problems; use them where the task needs it, not by default.
- MoE — only some "expert" sub-networks run per token, so a model's total parameter count overstates its per-token compute.
Interview deep dive¶
60-second talking points¶
- "Models work in tokens; cost, context, and limits are all token-based."
- "Temperature/top-p trade determinism for creativity."
- "Knowledge → RAG; behavior → fine-tune; quick change → prompt."
Scenario & system-design questions¶
A stakeholder asks: should we fine-tune or use RAG?
Ask what needs to change. Fresh/private/changing knowledge or a need for citations → RAG (cheaper, current, no retraining). Consistent style, format, or domain behavior → fine-tuning (or LoRA). Often RAG first; fine-tune only if behavior still isn't right.
Outputs are too random / too repetitive. What do you tune?
Lower temperature (and/or top-p) for more deterministic, factual output; raise it for creativity. Add stop sequences and max-tokens for control. For structure, use structured output rather than just temperature.
Why do larger context windows not always help?
More tokens cost more and add latency, and models attend less to the middle of long contexts ("lost in the middle") — so relevance and ordering matter more than raw size.
Pitfalls interviewers probe¶
- Confusing tokens with words/characters.
- Fine-tuning to add knowledge (that's RAG's job).
- High temperature for factual tasks.
- Assuming bigger context = better answers.
- Thinking the embedding model is the same as the chat LLM.
Rapid-fire¶
| Q | A |
|---|---|
| What's a token? | ~¾ word unit models read/write; basis of cost/limits |
| Temperature? | Randomness of sampling (low=deterministic) |
| Context window? | Max tokens (in+out) considered at once |
| RAG vs fine-tune? | Knowledge vs behavior/style |
| LoRA/PEFT? | Efficient fine-tuning without full retrain |
| Hallucination fix? | Ground (RAG), "say you don't know", low temp, verify |
How interviewers probe this¶
When would you pick a reasoning model over a fast non-reasoning model?
A strong answer ties it to the task: multi-step math, code, planning, or ambiguous analysis benefits from reasoning; extraction, classification, and short chat usually don't. Measure accuracy, latency, and cost on your eval set, tune effort or thinking budget, and route only the hard slice to the expensive mode.
Explain why the same prompt gives different answers, and how you make outputs dependable.
Sampling randomness, nondeterminism from batching and floating-point math on the serving side, and silent model updates if versions aren't pinned. Make it dependable with pinned versions, structured outputs and schema validation, low temperature where supported, retries with repair, and evals that measure the distribution of answers, not a single run.
Fine-tune, RAG, or a bigger model? Walk through the decision.
Diagnose the gap first. Missing or changing knowledge points to RAG. A format or behavior gap points to prompting, then fine-tuning. A reasoning-capability gap points to a stronger or reasoning model. Factor in data availability, maintenance cost (re-tuning on every base-model upgrade), and latency.
What actually limits context length in practice?
Attention cost and KV-cache memory grow with sequence length, so long contexts cost more and slow down. Quality also degrades with long inputs well before the advertised limit. Budget and order context deliberately instead of filling the window.