1 November 2026 • 4 min read
This sheet is for product managers who get an evaluation report with names like BLEU, ROUGE, faithfulness and context precision, and need to know what each number actually checks. Every entry comes from the paper that introduced the metric or from the Ragas library documentation. The Ragas docs change between versions, so check the version you have installed.
| Term | Meaning |
|---|---|
| Hallucination | In the survey by Ji and co-authors, the most inclusive and standard definition is generated content that is nonsensical or unfaithful to the provided source content. |
| Intrinsic hallucination | Output that contradicts the source content. Their example: a summary says a vaccine was approved in 2021 when the source says 2019. |
| Extrinsic hallucination | Output that cannot be verified from the source, so the source neither supports nor contradicts it. The survey notes it is not always wrong, because it can come from factually correct outside information, but most of the literature still treats it with caution. |
The split matters for your product. If your feature answers from a document you supply, an intrinsic hallucination is a clear bug. An extrinsic one may be true but you cannot show the user where it came from.
These two compare the model's output with one or more human-written references. They count words and word sequences. They do not read for meaning.
| Term | Meaning |
|---|---|
| BLEU | Introduced in 2002 for machine translation. It measures how close a machine translation is to one or more reference human translations. |
| Modified n-gram precision | BLEU's counting rule. For each candidate word, the count is clipped by its maximum count in any single reference, so repeating a word does not raise the score. |
| Brevity penalty | A multiplicative factor in BLEU that lowers the score of candidates shorter than the references. The final score is the geometric mean of the modified n-gram precisions, multiplied by this penalty. |
| ROUGE | Stands for Recall-Oriented Understudy for Gisting Evaluation. It compares a computer-generated summary with ideal summaries written by humans. |
| ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S | The four measures in the 2004 paper. N counts overlapping n-grams; L uses the longest common subsequence; W is a weighted version of L that remembers the length of consecutive matches; S counts skip-bigrams, which are pairs of words in sentence order with any gap allowed. |
BLEU's authors handle the problem of many valid wordings by allowing several references, each of which may use a different word choice. If you have only one reference answer per question, a correct answer in different words can still score low. For chat and question-answering features, treat these scores as a rough regression check, not a quality grade.
These four are defined in the Ragas docs. The main versions use an LLM as the judge (the docs also list a non-LLM context precision), so the judge model and its prompt are part of what you are measuring.
| Term | Meaning |
|---|---|
| Faithfulness | Whether a response is factually consistent with the retrieved context. A response counts as faithful if all its claims can be supported by that context. Score is supported claims divided by total claims, from 0 to 1. |
| Answer relevancy | Whether the response directly and appropriately addresses the original question. It ignores factual accuracy. Ragas generates artificial questions from the response (default 3) and averages their embedding similarity to the real question. |
| Context precision | Whether the retriever ranks relevant chunks above irrelevant ones. It is the mean of precision at each rank. There are versions that compare against a reference answer and versions that compare against the generated response. |
| Context recall | How many of the relevant pieces of information were retrieved. It always needs a reference to compare against. Score is reference claims supported by the retrieved context divided by total reference claims. |
Read them as two pairs. Context precision and recall test the retriever. Faithfulness and answer relevancy test the generated answer. If faithfulness is low but context recall is high, the right passage was found and the model ignored or distorted it.
No. Ragas faithfulness checks claims against the retrieved context only. If the retrieved passage is wrong, a faithful answer repeats the error.
They were built for translation and summaries, with reference texts. They suit a fixed test set where you compare versions of the same system. They are a weak guide to whether one answer is better than another.
Context precision and context recall. Low recall means relevant material was never retrieved. Low precision means it was retrieved but ranked below irrelevant chunks.
For BLEU, ROUGE and context recall, yes. Faithfulness needs the response and the retrieved context, and answer relevancy needs the question and the response. Context precision has a version that works without a reference.
Nine stories, every morning at 09:15 IST.
Get the brief free →