Question 1

Why is evaluating LLM applications different from traditional testing?

Accepted Answer

Traditional tests assume determinism, but LLMs break three assumptions: non-determinism (the same prompt can give different outputs each run), fuzzy correctness (quality is multi-dimensional and subjective with no single right answer), and silent regression (there is no automatic CI equivalent to catch quality drops). So a passing test is one data point, not a verdict, and LLM evaluation becomes a measurement discipline rather than a debugging task.

Question 2

What is temperature in LLM evaluation?

Accepted Answer

Temperature controls how random a model's outputs are. Near 0 the model is nearly deterministic; higher values make it sample from a wider range of tokens and produce more varied responses. Top-p (nucleus sampling) is related but uses a probability cutoff. The standard practice is to set temperature to 0 during eval runs to lock in determinism, and only test at production temperature when creative variance is needed.

Question 3

What is a rubric in LLM evaluation?

Accepted Answer

A rubric operationalizes a vague criterion by breaking it into specific, scorable questions, much like a code-review checklist. Instead of asking 'is this helpful?', it asks things like 'does it directly answer the question, avoid hedging, stay under 200 words, and read clearly?' The rubric is what makes evaluation reproducible, so two reviewers (human or AI) reach similar scores. Without one, every evaluation is just an opinion.

Question 4

What is a golden set?

Accepted Answer

A golden set is a curated collection of high-quality test cases that serves as your ground truth; everything in your eval is measured against it. A good golden set is seeded from real, anonymized production queries rather than examples you imagine, because real users phrase things unexpectedly and hit edge cases. Treat it as a versioned, critical artifact and update it whenever you discover new failure modes.

Question 5

What are the three ways to score LLM outputs?

Accepted Answer

Human evaluation is the gold standard and closest to ground truth, but slow and expensive, so it is used strategically. Heuristic or code-based evaluation is fast and cheap but only checks structural properties, not quality. LLM-as-judge uses a capable model to apply your rubric at scale. A complete system combines all three, since every automated method is an approximation of human judgment.

Question 6

What is LLM-as-judge?

Accepted Answer

LLM-as-judge uses a more capable model to score your application's outputs. You give it the original input, the output, and your rubric, and it returns a score plus an explanation. It makes evaluation scalable across thousands of outputs without human hours. Pointwise judging scores one output 1-5; pairwise judging compares two outputs and is more reliable but roughly twice as costly. Judges have biases, so they approximate human judgment rather than replace it.

Question 7

What is judge calibration?

Accepted Answer

Judge calibration measures how well an LLM judge matches human judgment before you rely on it. You have both humans and the judge score a sample of outputs and check how often they agree. High agreement means the judge is a good proxy; low agreement means it may be measuring the wrong thing. Calibrate before use and recalibrate periodically, especially after changing the judge model or its prompt.

Question 8

What is the RAG triad?

Accepted Answer

The RAG triad is the standard framework for evaluating Retrieval-Augmented Generation across three dimensions: faithfulness (did the answer come from the retrieved context?), answer relevance (did the response actually answer the question?), and context precision (did retrieval fetch the right documents?). It acts as an observability layer that isolates which stage (retrieval, augmentation, or generation) is failing so you know what to fix.

Question 9

What is the difference between offline and online evaluation?

Accepted Answer

Offline evaluation happens before shipping: you run changes against your golden dataset like a CI pipeline, using your best judge model since the example set is finite. Online evaluation happens in production: you continuously sample live outputs to catch edge cases and scale failures, relying on cheaper heuristics and smaller judges because cost matters. You need both, since offline catches known issues and online catches unexpected ones.

Question 10

What are common LLM evaluation anti-patterns?

Accepted Answer

Common anti-patterns include vibe-based evaluation (informal spot-checking that creates false confidence), the single-sample trap (running once and reporting one number), Goodhart's Law (over-optimizing a metric until it stops measuring quality), eval-production mismatch (golden sets that reflect expectations rather than real user behavior), and ignoring tail failures (a 92% pass rate hides catastrophic failures in the other 8%). Always review failure cases, not just averages.

Llm Interview Questions

Related Topics

LLM Evaluation Fundamentals(3 questions)

Why is evaluating LLM applications fundamentally different from traditional software testing?

What are `temperature` and `top-p`, and how should you set them during evaluation?

Why isn't a single evaluation run enough, even at `temperature = 0`?

Eval Primitives(4 questions)

What are evaluation criteria and quality dimensions in LLM evaluation?

What is a rubric, and why is it important?

What is a golden set, and how do you build a good one?

What are pass/fail thresholds and eval coverage?

Scoring Methods(5 questions)

What are the three main approaches to scoring LLM outputs, and what are their trade-offs?

What is heuristic / code-based evaluation, and what are its limits?

What is semantic similarity evaluation, and where does it fall short?

What is LLM-as-judge, and what is the difference between pointwise and pairwise evaluation?

What is judge calibration, and why does it matter?

RAG Evaluation(1 questions)

What is the RAG triad, and what are the most common RAG failure patterns?

Offline vs Online Evaluation(1 questions)

What is the difference between offline and online evaluation, and why do you need both?

Anti-Patterns & Eval Stack(1 questions)

What are the common LLM evaluation anti-patterns, and what does a complete eval stack look like?