RAG Axioms¶
Constraints on generated responses for a request
RAG axioms compare two generated responses \(r_1\), \(r_2\) for the same request \(q\) (optionally including the retrieved context \(C\)) and implement the interface Axiom[GenerationRequest, GenerationResponse] (see the data model).
They were introduced in our ICTIR 2025 paper [Merker et al., 2025] and address the five utility dimensions of generated responses proposed by Gienapp et al. [Gienapp et al., 2024]: coherence, coverage, consistency, correctness, and clarity.
Below, each axiom is listed with its intuition, followed by how it is implemented in ir_axioms.
The formalizations follow the paper, where, e.g., \(>_{10\%}\) means that only a difference of more than 10 % counts as "larger".
In the implementation, this margin is the margin_fraction parameter of each axiom: if the compared values of both responses are within this relative margin, the axiom expresses no preference.
The aspect-based axioms use the configured AspectExtraction tool, which extracts spaCy noun chunks by default.
In our ICTIR 2025 experiments, we used spaCy entities instead:
from ir_axioms.axiom import COV1
from ir_axioms.tools import SpacyEntitiesAspectExtraction
COV1(aspect_extraction=SpacyEntitiesAspectExtraction(), margin_fraction=0.5)
To evaluate the axioms against human judgments, follow the RAG evaluation walkthrough. The API reference documents all classes and parameters.
Coherence¶
Logical coherence: Is the response well-structured? Stylistic coherence: Does the response have a uniform style of speech?
COH1¶
COH1 WordLengthDeviationCoherenceAxiom — "Prefer responses with less variance in avg. word length across sentences." [Merker et al., 2025]
- Formalization
- \(\sigma(\mathrm{SWAWL}_{r_1}) < \sigma(\mathrm{SWAWL}_{r_2}) \Rightarrow r_1 >_\text{COH1} r_2\), where \(\mathrm{SWAWL}_r\) is the set of the average word lengths of the sentences in \(r\).
- Implementation
- Computes the average word length of each sentence and prefers the response with the lower standard deviation of these averages, as a proxy for a uniform style.
- Parameters
margin_fraction=0.0.- Tools
SentenceTokenizer,TermTokenizer.
COH2¶
COH2 SubjectVerbClosenessCoherenceAxiom — "Prefer responses with subject–verb pairs closer together." [Merker et al., 2025]
- Formalization
- \(\text{avg-SV-dist}(r_1) <_{10\%} \text{avg-SV-dist}(r_2) \Rightarrow r_1 >_\text{COH2} r_2\), where \(\text{avg-SV-dist}(r) = \frac{1}{|\mathrm{SVO}_r|} \sum_{(\mathit{sub}, \mathit{ver}, \mathit{obj}) \in \mathrm{SVO}_r} |\mathrm{pos}(\mathit{sub}) - \mathrm{pos}(\mathit{ver})|\) for the subject–verb–object triples \(\mathrm{SVO}_r\) in \(r\).
- Implementation
- Extracts subject–verb–object triples with spaCy and prefers the response with the lower average (maximum) distance between subjects and their verbs, as a proxy for easy-to-follow sentence structures.
- Parameters
margin_fraction=0.1,language_name="en_core_web_sm"(spaCy model).
Coverage¶
Broad coverage: Does the response cover diverse information? Deep coverage: Does the response provide in-depth and highly informative content?
COV1¶
COV1 AspectCountCoverageAxiom — "Prefer responses with more extracted aspects." [Merker et al., 2025]
- Formalization
- \(|\mathrm{ASP}_{r_1}| > |\mathrm{ASP}_{r_2}| \Rightarrow r_1 >_\text{COV1} r_2\), where \(\mathrm{ASP}_r\) is the set of aspects extracted from \(r\).
- Implementation
- Extracts the aspects of each response and prefers the response with more distinct aspects.
- Parameters
margin_fraction=0.0.- Tools
AspectExtraction.
COV2¶
COV2 AspectRedundancyCoverageAxiom — "Prefer responses with less redundant extracted aspects." [Merker et al., 2025]
- Formalization
- \(\text{avg-ASP-sim}(r_1) <_{20\%} \text{avg-ASP-sim}(r_2) \Rightarrow r_1 >_\text{COV2} r_2\), where \(\text{avg-ASP-sim}(r) = \frac{1}{|\mathrm{ASP}_r|^2} \sum_{a, a' \in \mathrm{ASP}_r} \mathrm{sim}(a, a')\).
- Implementation
- Extracts the aspects of each response and prefers the response with the lower mean pairwise similarity between its aspects.
- Parameters
margin_fraction=0.2.- Tools
AspectExtraction,SentenceSimilarity.
COV3¶
COV3 AspectSimilaritySentenceCountCoverageAxiom — "Prefer responses with more sentences covering aspects from the query." [Merker et al., 2025]
- Formalization
- \(\text{avg-ASP}_q\text{-cov}(r_1) >_{50\%} \text{avg-ASP}_q\text{-cov}(r_2) \Rightarrow r_1 >_\text{COV3} r_2\), where \(\text{avg-ASP}_q\text{-cov}(r) = \frac{1}{|\mathrm{ASP}_q|} \sum_{a \in \mathrm{ASP}_q} \sum_{s \in r} \mathrm{sim}(a, s)\) for the sentences \(s\) of \(r\).
- Implementation
- Extracts the aspects of the request text and counts the response's sentences covering them, weighting each sentence by its mean similarity to the aspects. Prefers the response with the higher weighted count.
- Parameters
margin_fraction=0.5.- Tools
AspectExtraction,SentenceTokenizer,SentenceSimilarity.
Consistency¶
External consistency: Does the response accurately convey the information from the sources? Internal consistency: Does the response not contradict itself?
CONS1¶
CONS1 AspectSimilaritySentenceCountConsistencyAxiom — "Prefer responses with more sentences covering aspects from the context." [Merker et al., 2025]
- Formalization
- \(\text{avg-ASP}_C\text{-cov}(r_1) >_{50\%} \text{avg-ASP}_C\text{-cov}(r_2) \Rightarrow r_1 >_\text{CONS1} r_2\), where \(\text{avg-ASP}_C\text{-cov}(r) = \frac{1}{|\mathrm{ASP}_C|} \sum_{a \in \mathrm{ASP}_C} \sum_{s \in r} \mathrm{sim}(a, s)\) for the aspects \(\mathrm{ASP}_C\) of the context documents \(C\).
- Implementation
- Like COV3, but extracts the aspects from the retrieved context \(C\) instead of the request text. Expresses no preference if the request has no context.
- Parameters
margin_fraction=0.5.- Tools
AspectExtraction,SentenceTokenizer,SentenceSimilarity.
CONS2¶
CONS2 RougeConsistencyAxiom — "Prefer responses with higher text overlap with contexts." [Merker et al., 2025]
- Formalization
- \(\text{ROUGE-L}(r_1, c') >_{10\%} \text{ROUGE-L}(r_2, c') \Rightarrow r_1 >_\text{CONS2} r_2\), where \(c'\) is the concatenated text of the context documents \(C\).
- Implementation
- Prefers the response with the higher ROUGE-Lsum F-measure [Lin, 2004] against the concatenated retrieved context. Expresses no preference if the request has no context.
- Parameters
margin_fraction=0.1.
CONS3¶
CONS3 EntityContradictionConsistencyAxiom — "Penalize aspects mentioned in contradictory phrases." [Merker et al., 2025]
- Formalization
- \(\frac{|\mathrm{ASP}^E_{r_1}|}{|\mathrm{ASP}_{r_1}|} <_{20\%} \frac{|\mathrm{ASP}^E_{r_2}|}{|\mathrm{ASP}_{r_2}|} \Rightarrow r_1 >_\text{CONS3} r_2\), where \(\mathrm{ASP}^E_r\) are the aspects of \(r\) that appear in both negated and non-negated passages of \(r\).
- Implementation
- Detects named entities and their negation (with negspaCy), counts entities that occur both negated and non-negated, and prefers the response with the lower proportion of such contradictory entities.
- Parameters
margin_fraction=0.2,language_name="en_core_web_sm"(spaCy model).
Correctness¶
Factuality: Does the response state things that are verifiable? Topical correctness: Does the response state things within the scope of the user's information need?
CORR1¶
CORR1 CitationSentenceCorrectnessAxiom — "Prefer responses with more sentences referencing sources." [Merker et al., 2025]
- Formalization
- \(\frac{|S^\text{ref}_{r_1}|}{|S_{r_1}|} >_{10\%} \frac{|S^\text{ref}_{r_2}|}{|S_{r_2}|} \Rightarrow r_1 >_\text{CORR1} r_2\), where \(S_r\) are the sentences of \(r\) and \(S^\text{ref}_r \subseteq S_r\) those containing a reference.
- Implementation
- Prefers the response with the higher proportion of sentences that contain a citation of the form
[1]. - Parameters
margin_fraction=0.1.- Tools
SentenceTokenizer.
Clarity¶
Language clarity: Is the response concise, comprehensible, lexically and grammatically correct, and accessible? Content clarity: Does the response clearly communicate the most salient information and where it stems from?
CLAR1¶
CLAR1 LanguageToolGrammarErrorProportionClarityAxiom — "Prefer responses with fewer grammar errors." [Merker et al., 2025]
- Formalization
- \(\frac{\mathrm{length}(\mathrm{ERR}_{r_1})}{\mathrm{length}(r_1)} <_{10\%} \frac{\mathrm{length}(\mathrm{ERR}_{r_2})}{\mathrm{length}(r_2)} \Rightarrow r_1 >_\text{CLAR1} r_2\), where \(\mathrm{ERR}_r\) are the spans of grammatical errors in \(r\).
- Implementation
- Checks the response with LanguageTool (grammar rules only) and prefers the response with the lower proportion of characters covered by errors.
- Parameters
margin_fraction=0.1,language="en-US".
CLAR2¶
CLAR2 FleschReadingEaseClarityAxiom — "Prefer responses with better readability." [Merker et al., 2025]
- Formalization
- \(\text{Flesch}(r_1) < \text{Flesch}(r_2) \Rightarrow r_1 >_\text{CLAR2} r_2\) (Eq. 11 in the paper).
- Implementation
- Compares the Flesch reading ease scores [Flesch, 1948] of the responses.
- Parameters
margin_fraction=0.0,language_name="en_core_web_sm"(spaCy model).
Known issue
Both the formalization in the paper (Eq. 11) and the implementation prefer the response with the lower Flesch reading ease score. As higher Flesch scores indicate easier text, this prefers the less readable response, contradicting the intuition (see the roadmap).
Adapted Retrieval Axioms¶
Many retrieval axioms can also be applied to generated responses, by treating the request as the query and each response as a document [Merker et al., 2025].
The GEN_* axioms wrap the corresponding retrieval axiom accordingly:
| Axiom | Class | Wrapped retrieval axiom |
|---|---|---|
| GEN_TFC1 | GenerativeTfc1Axiom |
TFC1 |
| GEN_LNC1 | GenerativeLnc1Axiom |
LNC1 |
| GEN_TF_LNC | GenerativeTfLncAxiom |
TF-LNC |
| GEN_REG | GenerativeRegAxiom |
REG |
| GEN_ANTI_REG | GenerativeAntiRegAxiom |
ANTI-REG |
| GEN_AND | GenerativeAndAxiom |
AND |
| GEN_LEN_AND | GenerativeLenAndAxiom |
LEN-AND |
| GEN_M_AND | GenerativeModifiedAndAxiom |
M-AND |
| GEN_DIV | GenerativeDivAxiom |
DIV |
| GEN_LEN_DIV | GenerativeLenDivAxiom |
LEN-DIV |
| GEN_STMC1 | GenerativeStmc1Axiom |
STMC1 |
| GEN_STMC2 | GenerativeStmc2Axiom |
STMC2 |
| GEN_PROX1 | GenerativeProx1Axiom |
PROX1 |
| GEN_PROX2 | GenerativeProx2Axiom |
PROX2 |
| GEN_PROX3 | GenerativeProx3Axiom |
PROX3 |
| GEN_PROX4 | GenerativeProx4Axiom |
PROX4 |
| GEN_PROX5 | GenerativeProx5Axiom |
PROX5 |
| GEN_ArgUC | GenerativeArgumentativeUnitsCountAxiom |
ArgUC |
| GEN_QTArg | GenerativeQueryTermOccurrenceInArgumentativeUnitsAxiom |
QTArg |
| GEN_QTPArg | GenerativeQueryTermPositionInArgumentativeUnitsAxiom |
QTPArg |
| GEN_aSL | GenerativeAverageSentenceLengthAxiom |
aSL |
Retrieval axioms that need retrieval scores or collection statistics (e.g., LB1, TFC3, M-TDC) have no generative counterpart.
Oracle Axioms¶
To evaluate RAG axioms, ir_axioms includes pseudo-axioms that derive preferences from human or LLM judgments (similar to ORACLE and ORIG for retrieval).
They read their judgments from local files, e.g., from the data/ directory of the repository.
Nugget Oracle¶
TrecRagNuggetAxiom — "Prefer the response that covers more information nuggets."
- Implementation
- Derives preferences from the nugget assignments of the TREC 2024 RAG track [Pradeep et al., 2024].
- Parameters
assignments_path(nugget assignments file),score_type("all","vital", or"weighted"nuggets),strict=False,margin_fraction=0.0.
Crowd Oracle¶
TrecRagCrowdAxiom — "Prefer the response that crowd workers judged more useful."
- Implementation
- Derives preferences from crowdsourced pairwise utility judgments of RAG responses for a utility dimension.
- Parameters
responses_path,ratings_path,utility_type(e.g.,"overall","coherence","broad coverage","topical correctness"),margin_fraction=0.1.
LLM Ratings¶
TrecRagLlmOrigAxiom — "Prefer the response that an LLM rated higher."
- Implementation
- Derives preferences from response ratings of an LLM, as an ORIG-like baseline for RAG.
- Parameters
ratings_path,margin_fraction=0.0.
Utility Dimensions in Related Work¶
Several frameworks describe utility dimensions of retrieval-augmented generation. The following table puts each into context with the utility dimensions of Gienapp et al. [Gienapp et al., 2024] that we used for our axioms.
| Framework | Dimension | Corresponds to |
|---|---|---|
| Gienapp et al. [Gienapp et al., 2024] | coherence, coverage, consistency, correctness, clarity | (same) |
| overall quality | — | |
| RAGAs [Es et al., 2024] | context precision, context recall, context entities recall | retrieval-focused |
| noise sensitivity | ≃ factual correctness | |
| answer relevance / response relevancy | ≃ deep coverage | |
| faithfulness | ≃ external consistency | |
| context relevance | ≃ broad coverage | |
| multi-modal faithfulness, multi-modal relevance | only textual context considered yet | |
| LLM-Rubric [Hashemi et al., 2024] | satisfaction | — |
| naturalness | ≃ language clarity / content clarity | |
| grounding sources | retrieval-focused | |
| citation presence, citation suitability | ≃ factual correctness | |
| citation optimality | ≃ factual correctness / external consistency | |
| redundancy | ≃ broad coverage | |
| conciseness | ≃ content clarity | |
| efficiency | architecture-specific | |
| ARES [Saad-Falcon et al., 2024] | context relevance | retrieval-focused |
| answer faithfulness | ≃ external consistency | |
| answer relevance | ≃ deep coverage | |
| Wallat et al. [Wallat et al., 2025] | citation correctness, citation faithfulness | ≃ factual correctness |
| Rosset et al. [Rosset et al., 2023] | usefulness | ≃ overall quality |
| relevance | ≃ deep coverage / topical correctness | |
| groundedness | ≃ factual correctness | |
| truthfulness | ≃ external consistency | |
| relevant/irrelevant grounding | ≃ external consistency / factual correctness | |
| thoroughness | ≃ broad coverage |
Missing Axioms and Axiom Ideas¶
The above axioms do not yet completely cover all individual aspects of response quality. The below list contains ideas on how to extend the set of axioms further:
- Factual-grounding-based axioms (main obstacle: fact checking often uses opaque models)
- Axioms from the viewpoint of the query, like EXAM/RUBRIC
- Content clarity axioms based on discourse structure
- More aspects of stylistic coherence
- Penalize too many tense switches
- Prefer response with easier co-reference resolution
- Consider other citation format or URLs for citation count
- Axioms for multi-modal outputs (e.g., generated images)
We are happy to collaborate to get more diverse axioms implemented. Please create a pull request to propose new ideas and refer to our guide on axiom development to realize an idea as code. See also the roadmap and related axioms from the literature.
References¶
@inproceedings{es2024ragas,
author = {Shahul Es and
Jithin James and
Luis Espinosa Anke and
Steven Schockaert},
title = {RAGAs: Automated Evaluation of Retrieval Augmented Generation},
booktitle = {Proceedings of the 18th Conference of the European Chapter of the
Association for Computational Linguistics, {EACL} 2024 - System Demonstrations,
St. Julians, Malta, March 17-22, 2024},
pages = {150--158},
publisher = {Association for Computational Linguistics},
year = {2024},
url = {https://doi.org/10.18653/v1/2024.eacl-demo.16},
doi = {10.18653/V1/2024.EACL-DEMO.16}
}@article{flesch1948new,
author = {Rudolph Flesch},
title = {A new readability yardstick},
journal = {Journal of Applied Psychology},
volume = {32},
number = {3},
pages = {221--233},
year = {1948},
url = {https://doi.org/10.1037/h0057532},
doi = {10.1037/h0057532}
}@inproceedings{gienapp2024evaluating,
author = {Lukas Gienapp and
Harrisen Scells and
Niklas Deckers and
Janek Bevendorff and
Shuai Wang and
Johannes Kiesel and
Shahbaz Syed and
Maik Fr{\"{o}}be and
Guido Zuccon and
Benno Stein and
Matthias Hagen and
Martin Potthast},
title = {Evaluating Generative Ad Hoc Information Retrieval},
booktitle = {Proceedings of the 47th International {ACM} {SIGIR} Conference on
Research and Development in Information Retrieval, {SIGIR} 2024, Washington
DC, USA, July 14-18, 2024},
pages = {1916--1929},
publisher = {{ACM}},
year = {2024},
url = {https://doi.org/10.1145/3626772.3657849},
doi = {10.1145/3626772.3657849}
}@inproceedings{hashemi2024llmrubric,
author = {Helia Hashemi and
Jason Eisner and
Corby Rosset and
Benjamin Van Durme and
Chris Kedzie},
title = {{LLM-Rubric}: {A} Multidimensional, Calibrated Approach to Automated
Evaluation of Natural Language Texts},
booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), {ACL} 2024, Bangkok, Thailand,
August 11-16, 2024},
pages = {13806--13834},
publisher = {Association for Computational Linguistics},
year = {2024},
url = {https://doi.org/10.18653/v1/2024.acl-long.745},
doi = {10.18653/V1/2024.ACL-LONG.745}
}@inproceedings{lin2004rouge,
author = {Chin{-}Yew Lin},
title = {{ROUGE}: {A} Package for Automatic Evaluation of Summaries},
booktitle = {Text Summarization Branches Out, Barcelona, Spain, July 2004},
pages = {74--81},
publisher = {Association for Computational Linguistics},
year = {2004},
url = {https://aclanthology.org/W04-1013/}
}@inproceedings{merker2025axioms,
author = {Jan Heinrich Merker and
Maik Fr{\"{o}}be and
Benno Stein and
Martin Potthast and
Matthias Hagen},
title = {Axioms for Retrieval-Augmented Generation},
booktitle = {Proceedings of the 2025 International {ACM} {SIGIR} Conference on
Innovative Concepts and Theories in Information Retrieval, {ICTIR}
2025, Padua, Italy, July 18, 2025},
pages = {67--77},
publisher = {{ACM}},
year = {2025},
url = {https://doi.org/10.1145/3731120.3744601},
doi = {10.1145/3731120.3744601}
}@article{pradeep2024initial,
author = {Ronak Pradeep and
Nandan Thakur and
Shivani Upadhyay and
Daniel Campos and
Nick Craswell and
Jimmy Lin},
title = {Initial Nugget Evaluation Results for the {TREC} 2024 {RAG} Track
with the AutoNuggetizer Framework},
journal = {CoRR},
volume = {abs/2411.09607},
year = {2024},
url = {https://doi.org/10.48550/arXiv.2411.09607},
doi = {10.48550/ARXIV.2411.09607},
eprinttype = {arXiv},
eprint = {2411.09607}
}@inproceedings{rosset2023axiomatic,
author = {Corby Rosset and
Guoqing Zheng and
Victor Dibia and
Ahmed Awadallah and
Paul Bennett},
title = {Axiomatic Preference Modeling for Longform Question Answering},
booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural
Language Processing, {EMNLP} 2023, Singapore, December 6-10, 2023},
pages = {11445--11475},
publisher = {Association for Computational Linguistics},
year = {2023},
url = {https://doi.org/10.18653/v1/2023.emnlp-main.702},
doi = {10.18653/V1/2023.EMNLP-MAIN.702}
}@inproceedings{saadfalcon2024ares,
author = {Jon Saad{-}Falcon and
Omar Khattab and
Christopher Potts and
Matei Zaharia},
title = {{ARES:} An Automated Evaluation Framework for Retrieval-Augmented Generation
Systems},
booktitle = {Proceedings of the 2024 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies
(Volume 1: Long Papers), {NAACL} 2024, Mexico City, Mexico, June 16-21,
2024},
pages = {338--354},
publisher = {Association for Computational Linguistics},
year = {2024},
url = {https://doi.org/10.18653/v1/2024.naacl-long.20},
doi = {10.18653/V1/2024.NAACL-LONG.20}
}@inproceedings{wallat2025correctness,
author = {Jonas Wallat and
Maria Heuss and
Maarten de Rijke and
Avishek Anand},
title = {Correctness is not Faithfulness in Retrieval Augmented Generation
Attributions},
booktitle = {Proceedings of the 2025 International {ACM} {SIGIR} Conference on
Innovative Concepts and Theories in Information Retrieval, {ICTIR}
2025, Padua, Italy, July 18, 2025},
pages = {22--32},
publisher = {{ACM}},
year = {2025},
url = {https://doi.org/10.1145/3731120.3744592},
doi = {10.1145/3731120.3744592}
}