Related Axioms

Axiomatic constraints from the literature, and which of them ir_axioms implements

Source

This page is adapted from the resource page Axiomatic Thinking for Information Retrieval by Hui Fang and ChengXiang Zhai. We revised the list for this documentation: fixed typos and dates, replaced outdated links with DOIs, marked which constraints are implemented in ir_axioms, and added more recent work (marked with ). Please refer to the original page for the authoritative version.

Axiomatic thinking has been adopted for the development of both retrieval models and evaluation metrics. The general idea is to seek a set of desirable properties, expressed mathematically as formal constraints, to guide the search for an optimal solution. The explicit expression of desirable properties makes it possible to analytically address issues that would otherwise appear to be purely empirical, provides theoretical guidance on how to optimize a retrieval model or evaluation metric, and allows applying the identified constraints directly in many practical applications.

Below, each constraint is listed with its intuition and source. For constraints on retrieval models, the field "In ir_axioms" links to the corresponding implementation in the retrieval axioms or RAG axioms, if any. The roadmap lists constraints that we would like to add.

Resources

Tutorials

SIGIR 2014 tutorial "Axiomatic Analysis and Optimization of Information Retrieval Models" by Hui Fang and ChengXiang Zhai (slides as PPTX).

ICTIR 2013 tutorial by ChengXiang Zhai and Hui Fang [Zhai and Fang, 2013].

Workshop

SIGIR 2017 Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR) [Amigó et al., 2017] (website, report [Amigó et al., 2018]).

Mailing list

AX4IR Google Group (ax4ir@googlegroups.com).

Constraints for Retrieval Models

Term Frequency, Term Discrimination, and Length Normalization

TFC1 — "Give a higher score to a document with more occurrences of a query term." [Fang et al., 2004]

In ir_axioms

TFC1

TFC2 — "The increase in the retrieval score due to an increase in TF should be smaller for larger TFs." [Fang et al., 2004; Fang et al., 2011]

In ir_axioms

Not implemented (why?).

TFC3 — "If two documents have the same total occurrences of all query terms and all query terms have the same term discrimination value, give a higher score to the document covering more distinct query terms." [Fang et al., 2011]

In ir_axioms

TFC3

TDC — "Penalize terms that are popular in the collection." [Fang et al., 2004]

In ir_axioms

M-TDC, the modified TDC by Shi et al. [Shi et al., 2005].

LNC1 — "The score of a document should decrease if an occurrence of a non-relevant word is added." [Fang et al., 2004]

In ir_axioms

LNC1

LNC2 — "Avoid over-penalizing long relevant documents." [Fang et al., 2004]

In ir_axioms

Not implemented (why?).

TF-LNC — "Regulate the interaction between TF and document length." [Fang et al., 2004]

In ir_axioms

TF-LNC

Lower-Bounding Term Frequency

LB1 — "The presence–absence gap (0–1 gap) should not be closed due to length normalization." [Lv and Zhai, 2011]

In ir_axioms

LB1

LB2 — "Repeated occurrences of an already matched query term are not as important as the first occurrence of an otherwise absent query term." [Lv and Zhai, 2011]

In ir_axioms

Not implemented (why?).

Semantic Term Matching

STMC1 — "Give a higher score to a document with a term that is more semantically related to a query term." [Fang and Zhai, 2006]

In ir_axioms

STMC1

STMC2 — "Avoid over-favoring semantically similar terms." [Fang and Zhai, 2006]

In ir_axioms

STMC2

STMC3 — "Favor semantically similar terms." [Fang and Zhai, 2006]

In ir_axioms

Not implemented.

TSSC1, TSSC2 — Term semantic similarity constraints for query expansion. [Fang, 2008]

In ir_axioms

Not implemented.

Term Proximity

Proximity — "Term proximity should positively contribute to the retrieval score of a document." [Tao and Zhai, 2007]

In ir_axioms

Not implemented as such; see PROX1–PROX5 below.

Convex curve — "The contribution of a distance measure should drop quickly when the distance is small and become nearly constant as the distance grows." [Tao and Zhai, 2007]

In ir_axioms

Not implemented.

PROX1–PROX5 — "Prefer documents with query terms closer together or earlier." [Hagen et al., 2016]

In ir_axioms

PROX1–PROX5

Query Term Relations

REG — "Give a higher score to a document that covers more query aspects." [Zheng and Fang, 2010]

In ir_axioms

REG, ANTI-REG, ASPECT-REG

AND — "If two query terms have an AND relation, documents with both terms should be ranked higher than those with only one of them." [Wu and Fang, 2012]

In ir_axioms

AND, M-AND, LEN-AND, LEN-M-AND

Diversity — Axioms characterizing result diversification functions. [Gollapudi and Sharma, 2009]

In ir_axioms

DIV (pairwise reformulation)

Pseudo-Relevance Feedback

Document score — "Documents with higher scores should be given more weight in the feedback weight function." [Clinchant and Gaussier, 2010; Clinchant and Gaussier, 2013]

Proximity — "Feedback terms should be close to query terms in documents." [Clinchant and Gaussier, 2010; Clinchant and Gaussier, 2013]

Document frequency — "Feedback terms should receive higher weights when they occur more often in the feedback set." [Clinchant and Gaussier, 2011; Clinchant and Gaussier, 2011]

In ir_axioms

Not implemented (ir_axioms does not cover query expansion yet).

Translation Language Models

Constraints for translation language models for retrieval [Karimzadehgan and Zhai, 2012]:

General 1 — "For all translation language models, the self-translation probability should be the same."

General 2 — "The self-translation probability should be larger than the probability of translating any other word to this word."

General 3 — "A word is more likely to be translated to itself than to any other word."

Additional 4 — "If word \(u\) occurs more often than word \(v\) in the context of word \(w\), and both co-occur similarly with all other words, the probability of translating \(u\) to \(w\) should be higher."

Additional 5 — "If both \(u\) and \(v\) co-occur equally with word \(w\), but \(v\) co-occurs with many more other words than \(u\), the probability of translating \(u\) to \(w\) should be higher."

In ir_axioms

Not applicable (constraints on the translation model rather than on rankings).

Multi-Criteria Relevance Ranking

Compatibility — Compatibility constraint for score transformations in linear combinations of relevance criteria. [Gerani et al., 2012]

In ir_axioms

Not implemented.

Axiomatic Re-Ranking

Besides PROX1–PROX5 and ORIG, Hagen et al. [Hagen et al., 2016] proposed further axioms for re-ranking:

QPHRA — "A document containing all the query phrases is favored over a document not containing all phrases." [Hagen et al., 2016]

In ir_axioms

Not implemented (see also PROX3).

RSIM — "From any cluster of similar (near-duplicate) documents, favor only one document over all the others." [Hagen et al., 2016]

In ir_axioms

Not implemented.

P-RANK — "Prefer a document with higher PageRank." [Hagen et al., 2016]

In ir_axioms

Not implemented.

ORIG — "Give some voting power to the reasonable ideas underlying the basis retrieval model." [Hagen et al., 2016]

In ir_axioms

ORIG

Argument Retrieval

ArgUC — "Prefer more argumentative units." [Bondarenko et al., 2018; Bondarenko et al., 2019]

QTArg — "Prefer more query terms in argumentative units." [Bondarenko et al., 2018; Bondarenko et al., 2019]

QTPArg — "Prefer earlier query terms in argumentative units." [Bondarenko et al., 2018; Bondarenko et al., 2019]

aSLDoc — "Prefer passages with 12–20 words per sentence." [Bondarenko et al., 2018; Bondarenko et al., 2019]

In ir_axioms

ArgUC, QTArg, QTPArg, aSL

CompArg — "Prefer more comparative objects in argumentative units." [Reimer et al., 2022]

CompPArg — "Prefer earlier comparative objects in argumentative units." [Reimer et al., 2022]

ArgQ — "Prefer higher argument quality." [Reimer et al., 2022]

In ir_axioms

Not implemented (proposed for comparative questions).

QArgSim_avg — "Prefer the document whose argumentative units on average are more similar to the query." [Heinrich et al., 2025]

QArgSim_max — "Prefer the document whose most similar argumentative unit is more similar to the query." [Heinrich et al., 2025]

QSenSim_avg — "Prefer the document whose sentences on average are more similar to the query." [Heinrich et al., 2025]

QSenSim_max — "Prefer the document whose most similar sentence is more similar to the query." [Heinrich et al., 2025]

In ir_axioms

Not implemented. These refine STMC1 from terms to argumentative units and sentences, each in an exact and a relaxed (10 % margin) variant.

Generated Responses

Long-form QA — Axioms for preference modeling of long-form answers, e.g., usefulness, groundedness, and thoroughness. [Rosset et al., 2023]

In ir_axioms

Not implemented; related to the RAG axioms.

RAG utility — Axioms for coherence, coverage, consistency, correctness, and clarity of generated responses. [Merker et al., 2025]

In ir_axioms

RAG axioms

Axioms for Neural Ranking Models

Retrieval axioms have also been used to analyze, regularize, and explain neural ranking models:

Diagnosis

Diagnostic datasets derived from axioms reveal which constraints neural models satisfy [Rennings et al., 2019; Câmara and Hauff, 2020; MacAvaney et al., 2022], also with relaxed term matching [Parry et al., 2026].

Regularization

Axioms serve as an additional loss when training neural models [Rosset et al., 2019].

Explanation

Axiomatic preferences explain the rankings of neural models [Völske et al., 2021].

Re-ranking

Combined axiomatic preferences re-rank initial results [Hagen et al., 2016; Bondarenko et al., 2022].

Constraints for Evaluation Metrics

These constraints characterize evaluation metrics rather than retrieval models and are thus not applicable to ir_axioms.

General Constraints

Strong definiteness — "The measure must be definable for any gold standard and system output." [Sebastiani, 2015]

Fixed range — "The metric score must be bounded from below and above." [Sebastiani, 2015; Moffat, 2013]

Classification Metrics

Strict monotonicity — "Replacing an incorrect decision by a correct decision increases the score." [Sebastiani, 2015; Sokolova, 2006]

Continuous differentiability — "The measure must be continuous and differentiable over the true positives and true negatives." [Sebastiani, 2015]

Symmetry — "The measure should be invariant with respect to switching the roles of the class and its complement." [Sebastiani, 2015; Sokolova, 2006]

Absolute weighting — "A parameter of the measure determines whether adding one document of each class into a system's categorization improves the score." (task-dependent) [Amigó et al., 2017]

Non-informativeness fixed score — "Any non-informative (random) classification output achieves a fixed score." (task-dependent) [Amigó et al., 2017]

Non-informativeness growing score — "The score of a non-informative output for a class correlates with the output class size." (task-dependent) [Amigó et al., 2017]

Clustering Metrics

Cluster homogeneity — "Joining clusters with items from different categories decreases the score." [Rosenberg and Hirschberg, 2007; Amigó et al., 2009]

Cluster completeness — "Items belonging to the same category should appear in the same cluster." [Rosenberg and Hirschberg, 2007; Dom, 2001; Amigó et al., 2009; Meilă, 2003]

Rag bag — "Adding noise (single-item categories) to a clean cluster is worse than joining noisy items in the same cluster." [Amigó et al., 2009]

Cluster size vs. quantity — "A small error in a big cluster is preferable to many small errors in small clusters." [Amigó et al., 2009; Meilă, 2003]

Ranking Metrics

Priority — "Swapping two documents in the ranking according to their relevance increases the score." [Moffat, 2013; Ferrante et al., 2015; Amigó et al., 2013]

Deepness — "There is an area of the ranking that is never explored by the user; the relevance of documents at the top has more effect." [Moffat, 2013; Ferrante et al., 2015; Amigó et al., 2013]

Deepness threshold — "One relevant document at the top is better than many relevant documents after many non-relevant documents." [Amigó et al., 2013]

Closeness threshold — "There is an area that is always explored by the user: there exists a small \(n\) such that \(n\) relevant documents in the first \(2n\) positions are better than one relevant document in the first position." [Amigó et al., 2013]

Confidence — "Adding noise at the end of the ranking decreases the score." [Amigó et al., 2013]

See also the axiometrics framework [Busin and Mizzaro, 2013].

Constraints for Similarity

These constraints characterize similarity functions and are also not applicable to ir_axioms.

Metric-Space-Based Similarity

Maximality, Symmetry, Triangle inequality — Similarity as a metric space. [Shepard, 1987]

Feature Contrast Model

Matching — "Similarity is a function of the common and distinctive features of the objects." [Tversky, 1977]

Monotonicity — "Similarity increases whenever the common features increase or the differences decrease." [Tversky, 1977]

Independence — "Object features affect similarity independently." [Tversky, 1977]

Information-Theoretic Similarity

Identity — "Adding or removing features to an object decreases its similarity to the original." [Amigó et al., 2017; Amigó et al., 2020]

Identity specificity — "Adding new features increases the self-similarity." [Amigó et al., 2017; Amigó et al., 2020]

Unexpectedness — "When adding a feature, similarity decreases to a greater extent if the added feature is less expected." [Amigó et al., 2017; Amigó et al., 2020]

Dependency — "Adding new features to both objects increases their similarity whenever their respective conditional probabilities grow." [Amigó et al., 2017; Amigó et al., 2020]

Asymmetry — "An object is more similar to any of its parts than vice versa." [Amigó et al., 2017; Amigó et al., 2020]

Axioms Beyond Retrieval

Framing

A first axiomatization of frames in communication [Allein et al., 2022]:

Axiom 1 — "Frames exist."

Axiom 2 — "Exposure to frames has measurable effects."

Axiom 3 — "A frame can be defined by what belongs to the frame, or by what does not belong to the frame (frame dualism)."

Further Reading

More axiomatic analyses of retrieval models, as listed on the original page:

Foundations

Aboutness axioms [Bruza and Huibers, 1994], the exploration of axiomatic approaches to retrieval [Fang and Zhai, 2005], and their diagnostic evaluation [Fang et al., 2011].

Constraint analyses

Learned term-weighting schemes [Cummins and O'Riordan, 2007], measuring constraint violations [Cummins and O'Riordan, 2009], analysing ranking functions [Cummins and O'Riordan, 2011], length normalization [Cummins and O'Riordan, 2012], and heuristic constraints in function discovery [Goswami et al., 2015].

Term frequency normalization

Multi-topical documents [Na et al., None], adaptive and log-logistic interpretations of BM25 [Lv and Zhai, 2011; Lv and Zhai, 2012], very long documents [Lv and Zhai, 2011], and query length heuristics [Lv, 2015].

Information-based models

Information-based retrieval models [Clinchant and Gaussier, 2010; Clinchant and Gaussier, 2011] and cross-language retrieval [Li and Gaussier, 2012; Rahimi et al., 2014].

Pseudo-relevance feedback

Document frequency [Clinchant and Gaussier, 2011; Clinchant and Gaussier, 2011], theoretical analyses [Clinchant and Gaussier, 2013], and smoothing methods [Hazimeh and Zhai, 2015].

Other tasks

Rating-based ranking of items [Zhang et al., 2011], medical records search [Wang et al., 2014], and query performance prediction [Makarenkov et al., 2015].

References

Liesbeth Allein, Philipp Cimiano, Ivan Habernal, Martin Potthast, Arno Simons, and Benno Stein. Towards Operationalizing Frames through Axiomatization. In Framing in Communication: From Theories to Computation (Dagstuhl Seminar 22131), 2022. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. [bib]

@inproceedings{allein2022towards,
  author       = {Liesbeth Allein and
                  Philipp Cimiano and
                  Ivan Habernal and
                  Martin Potthast and
                  Arno Simons and
                  Benno Stein},
  title        = {Towards Operationalizing Frames through Axiomatization},
  booktitle    = {Framing in Communication: From Theories to Computation (Dagstuhl Seminar 22131)},
  series       = {Dagstuhl Reports},
  volume       = {12},
  publisher    = {Schloss Dagstuhl - Leibniz-Zentrum f{\"{u}}r Informatik},
  year         = {2022},
  url          = {https://downloads.webis.de/publications/papers/allein_2022.pdf}
}

Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information Retrieval, 12 (4) : 461–486, 2009. [bib] [doi]

@article{amigo2009comparison,
  author       = {Enrique Amig{\'{o}} and
                  Julio Gonzalo and
                  Javier Artiles and
                  Felisa Verdejo},
  title        = {A comparison of extrinsic clustering evaluation metrics based on formal constraints},
  journal      = {Information Retrieval},
  volume       = {12},
  number       = {4},
  pages        = {461--486},
  year         = {2009},
  url          = {https://doi.org/10.1007/s10791-008-9066-8},
  doi          = {10.1007/s10791-008-9066-8}
}

Enrique Amigó, Julio Gonzalo, and Felisa Verdejo. A general evaluation measure for document organization tasks. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 643–652, 2013. ACM. [bib] [doi]

@inproceedings{amigo2013general,
  author       = {Enrique Amig{\'{o}} and
                  Julio Gonzalo and
                  Felisa Verdejo},
  title        = {A general evaluation measure for document organization tasks},
  booktitle    = {Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval},
  pages        = {643--652},
  publisher    = {ACM},
  year         = {2013},
  url          = {https://doi.org/10.1145/2484028.2484081},
  doi          = {10.1145/2484028.2484081}
}

Enrique Amigó, Julio Gonzalo, Fernando Giner, and Felisa Verdejo. An Axiomatic Account of Similarity. In Proceedings of the SIGIR 2017 Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR), 2017. [bib]

@inproceedings{amigo2017account,
  author       = {Enrique Amig{\'{o}} and
                  Julio Gonzalo and
                  Fernando Giner and
                  Felisa Verdejo},
  title        = {An Axiomatic Account of Similarity},
  booktitle    = {Proceedings of the SIGIR 2017 Workshop on Axiomatic Thinking for
                  Information Retrieval and Related Tasks (ATIR)},
  year         = {2017}
}

Enrique Amigó, Hui Fang, Stefano Mizzaro, and ChengXiang Zhai. Axiomatic Thinking for Information Retrieval. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1419–1420, 2017. ACM. [bib] [doi]

@inproceedings{amigo2017axiomatic,
  author       = {Enrique Amig{\'{o}} and
                  Hui Fang and
                  Stefano Mizzaro and
                  ChengXiang Zhai},
  title        = {Axiomatic Thinking for Information Retrieval},
  booktitle    = {Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages        = {1419--1420},
  publisher    = {ACM},
  year         = {2017},
  url          = {https://doi.org/10.1145/3077136.3084369},
  doi          = {10.1145/3077136.3084369}
}

Enrique Amigó, Hui Fang, Stefano Mizzaro, and ChengXiang Zhai. Report on the SIGIR 2017 Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR). ACM SIGIR Forum, 51 (3) : 99–106, 2018. [bib] [doi]

@article{amigo2018report,
  author       = {Enrique Amig{\'{o}} and
                  Hui Fang and
                  Stefano Mizzaro and
                  ChengXiang Zhai},
  title        = {Report on the SIGIR 2017 Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR)},
  journal      = {ACM SIGIR Forum},
  volume       = {51},
  number       = {3},
  pages        = {99--106},
  year         = {2018},
  url          = {https://doi.org/10.1145/3190580.3190596},
  doi          = {10.1145/3190580.3190596}
}

Enrique Amigó, Fernando Giner, Julio Gonzalo, and Felisa Verdejo. On the foundations of similarity in information access. Information Retrieval Journal, 23 (3) : 216–254, 2020. [bib] [doi]

@article{amigo2020foundations,
  author       = {Enrique Amig{\'{o}} and
                  Fernando Giner and
                  Julio Gonzalo and
                  Felisa Verdejo},
  title        = {On the foundations of similarity in information access},
  journal      = {Information Retrieval Journal},
  volume       = {23},
  number       = {3},
  pages        = {216--254},
  year         = {2020},
  url          = {https://doi.org/10.1007/s10791-020-09375-z},
  doi          = {10.1007/s10791-020-09375-z}
}

Alexander Bondarenko, Michael Völske, Alexander Panchenko, Chris Biemann, Benno Stein, and Matthias Hagen. Webis at TREC 2018: Common Core Track. In Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018, Gaithersburg, Maryland, USA, November 14-16, 2018, 2018. National Institute of Standards and Technology (NIST). [bib]

@inproceedings{bondarenko2018webis,
  author       = {Alexander Bondarenko and
                  Michael V{\"{o}}lske and
                  Alexander Panchenko and
                  Chris Biemann and
                  Benno Stein and
                  Matthias Hagen},
  title        = {Webis at {TREC} 2018: Common Core Track},
  booktitle    = {Proceedings of the Twenty-Seventh Text REtrieval Conference, {TREC}
                  2018, Gaithersburg, Maryland, USA, November 14-16, 2018},
  series       = {{NIST} Special Publication},
  volume       = {500-331},
  publisher    = {National Institute of Standards and Technology {(NIST)}},
  year         = {2018},
  url          = {https://trec.nist.gov/pubs/trec27/papers/Webis-CC.pdf}
}

Alexander Bondarenko, Maik Fröbe, Vaibhav Kasturia, Michael Völske, Benno Stein, and Matthias Hagen. Webis at TREC 2019: Decision Track. In Proceedings of the Twenty-Eighth Text REtrieval Conference, TREC 2019, Gaithersburg, Maryland, USA, November 13-15, 2019, 2019. National Institute of Standards and Technology (NIST). [bib]

@inproceedings{bondarenko2019webis,
  author       = {Alexander Bondarenko and
                  Maik Fr{\"{o}}be and
                  Vaibhav Kasturia and
                  Michael V{\"{o}}lske and
                  Benno Stein and
                  Matthias Hagen},
  title        = {Webis at {TREC} 2019: Decision Track},
  booktitle    = {Proceedings of the Twenty-Eighth Text REtrieval Conference, {TREC}
                  2019, Gaithersburg, Maryland, USA, November 13-15, 2019},
  series       = {{NIST} Special Publication},
  volume       = {1250},
  publisher    = {National Institute of Standards and Technology {(NIST)}},
  year         = {2019},
  url          = {https://trec.nist.gov/pubs/trec28/papers/Webis.D.pdf}
}

Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völske, and Matthias Hagen. Axiomatic Retrieval Experimentation with ir_axioms. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11-15, 2022, pages 3131–3140, 2022. ACM. [bib] [doi]

@inproceedings{bondarenko2022axiomatic,
  author       = {Alexander Bondarenko and
                  Maik Fr{\"{o}}be and
                  Jan Heinrich Reimer and
                  Benno Stein and
                  Michael V{\"{o}}lske and
                  Matthias Hagen},
  title        = {Axiomatic Retrieval Experimentation with ir{\_}axioms},
  booktitle    = {{SIGIR} '22: The 45th International {ACM} {SIGIR} Conference on Research
                  and Development in Information Retrieval, Madrid, Spain, July 11-15, 2022},
  pages        = {3131--3140},
  publisher    = {{ACM}},
  year         = {2022},
  url          = {https://doi.org/10.1145/3477495.3531743},
  doi          = {10.1145/3477495.3531743}
}

P. D. Bruza and T. W. C. Huibers. Investigating Aboutness Axioms using Information Fields. In SIGIR '94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pages 112–121, 1994. Springer London. [bib] [doi]

@inproceedings{bruza1994investigating,
  author       = {P. D. Bruza and
                  T. W. C. Huibers},
  title        = {Investigating Aboutness Axioms using Information Fields},
  booktitle    = {SIGIR '94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval},
  pages        = {112--121},
  publisher    = {Springer London},
  year         = {1994},
  url          = {https://doi.org/10.1007/978-1-4471-2099-5\_12},
  doi          = {10.1007/978-1-4471-2099-5\_12}
}

Luca Busin and Stefano Mizzaro. Axiometrics. In Proceedings of the 2013 Conference on the Theory of Information Retrieval, pages 22–29, 2013. ACM. [bib] [doi]

@inproceedings{busin2013axiometrics,
  author       = {Luca Busin and
                  Stefano Mizzaro},
  title        = {Axiometrics},
  booktitle    = {Proceedings of the 2013 Conference on the Theory of Information Retrieval},
  pages        = {22--29},
  publisher    = {ACM},
  year         = {2013},
  url          = {https://doi.org/10.1145/2499178.2499182},
  doi          = {10.1145/2499178.2499182}
}

Stéphane Clinchant and Eric Gaussier. Information-based models for ad hoc IR. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 234–241, 2010. ACM. [bib] [doi]

@inproceedings{clinchant2010information,
  author       = {St{\'{e}}phane Clinchant and
                  Eric Gaussier},
  title        = {Information-based models for ad hoc IR},
  booktitle    = {Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval},
  pages        = {234--241},
  publisher    = {ACM},
  year         = {2010},
  url          = {https://doi.org/10.1145/1835449.1835490},
  doi          = {10.1145/1835449.1835490}
}

Stéphane Clinchant and Eric Gaussier. Retrieval constraints and word frequency distributions a log-logistic model for IR. Information Retrieval, 14 (1) : 5–25, 2011. [bib] [doi]

@article{clinchant2010retrieval,
  author       = {St{\'{e}}phane Clinchant and
                  Eric Gaussier},
  title        = {Retrieval constraints and word frequency distributions a log-logistic model for IR},
  journal      = {Information Retrieval},
  volume       = {14},
  number       = {1},
  pages        = {5--25},
  year         = {2011},
  url          = {https://doi.org/10.1007/s10791-010-9143-7},
  doi          = {10.1007/s10791-010-9143-7}
}

Stéphane Clinchant and Eric Gaussier. Is Document Frequency Important for PRF?. In Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011, pages 89–100, 2011. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{clinchant2011document,
  author       = {St{\'{e}}phane Clinchant and
                  Eric Gaussier},
  title        = {Is Document Frequency Important for PRF?},
  booktitle    = {Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011},
  series       = {Lecture Notes in Computer Science},
  pages        = {89--100},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2011},
  url          = {https://doi.org/10.1007/978-3-642-23318-0\_10},
  doi          = {10.1007/978-3-642-23318-0\_10}
}

Stéphane Clinchant and Éric Gaussier. A Document Frequency Constraint for Pseudo-Relevance Feedback Models. In CORIA 2011 - COnférence en Recherche d'Information et Applications, 2011. [bib]

@inproceedings{clinchant2011frequency,
  author       = {St{\'{e}}phane Clinchant and
                  {\'{E}}ric Gaussier},
  title        = {A Document Frequency Constraint for Pseudo-Relevance Feedback Models},
  booktitle    = {CORIA 2011 - COnf{\'{e}}rence en Recherche d'Information et Applications},
  year         = {2011},
  url          = {http://www.asso-aria.org/coria/2011/73.pdf}
}

Stéphane Clinchant and Eric Gaussier. A Theoretical Analysis of Pseudo-Relevance Feedback Models. In Proceedings of the 2013 Conference on the Theory of Information Retrieval, pages 6–13, 2013. ACM. [bib] [doi]

@inproceedings{clinchant2013theoretical,
  author       = {St{\'{e}}phane Clinchant and
                  Eric Gaussier},
  title        = {A Theoretical Analysis of Pseudo-Relevance Feedback Models},
  booktitle    = {Proceedings of the 2013 Conference on the Theory of Information Retrieval},
  pages        = {6--13},
  publisher    = {ACM},
  year         = {2013},
  url          = {https://doi.org/10.1145/2499178.2499179},
  doi          = {10.1145/2499178.2499179}
}

Ronan Cummins and Colm O'Riordan. An axiomatic comparison of learned term-weighting schemes in information retrieval: clarifications and extensions. Artificial Intelligence Review, 28 (1) : 51–68, 2007. [bib] [doi]

@article{cummins2007axiomatic,
  author       = {Ronan Cummins and
                  Colm O'Riordan},
  title        = {An axiomatic comparison of learned term-weighting schemes in information retrieval: clarifications and extensions},
  journal      = {Artificial Intelligence Review},
  volume       = {28},
  number       = {1},
  pages        = {51--68},
  year         = {2007},
  url          = {https://doi.org/10.1007/s10462-008-9074-5},
  doi          = {10.1007/s10462-008-9074-5}
}

Ronan Cummins and Colm O'Riordan. Measuring constraint violations in information retrieval. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 722–723, 2009. ACM. [bib] [doi]

@inproceedings{cummins2009measuring,
  author       = {Ronan Cummins and
                  Colm O'Riordan},
  title        = {Measuring constraint violations in information retrieval},
  booktitle    = {Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval},
  pages        = {722--723},
  publisher    = {ACM},
  year         = {2009},
  url          = {https://doi.org/10.1145/1571941.1572096},
  doi          = {10.1145/1571941.1572096}
}

Ronan Cummins and Colm O'Riordan. Analysing Ranking Functions in Information Retrieval Using Constraints. In Information Extraction from the Internet, 2011. [bib]

@incollection{cummins2011analysing,
  author       = {Ronan Cummins and
                  Colm O'Riordan},
  title        = {Analysing Ranking Functions in Information Retrieval Using Constraints},
  booktitle    = {Information Extraction from the Internet},
  year         = {2011}
}

Ronan Cummins and Colm O'Riordan. A constraint to automatically regulate document-length normalisation. In Proceedings of the 21st ACM international conference on Information and knowledge management, pages 2443–2446, 2012. ACM. [bib] [doi]

@inproceedings{cummins2012constraint,
  author       = {Ronan Cummins and
                  Colm O'Riordan},
  title        = {A constraint to automatically regulate document-length normalisation},
  booktitle    = {Proceedings of the 21st ACM international conference on Information and knowledge management},
  pages        = {2443--2446},
  publisher    = {ACM},
  year         = {2012},
  url          = {https://doi.org/10.1145/2396761.2398662},
  doi          = {10.1145/2396761.2398662}
}

Arthur Câmara and Claudia Hauff. Diagnosing BERT with Retrieval Heuristics. In Advances in Information Retrieval, 42nd European Conference on IR Research, ECIR 2020, pages 605–618, 2020. Springer International Publishing. [bib] [doi]

@inproceedings{camara2020diagnosing,
  author       = {Arthur C{\^{a}}mara and
                  Claudia Hauff},
  title        = {Diagnosing BERT with Retrieval Heuristics},
  booktitle    = {Advances in Information Retrieval, 42nd European Conference on IR Research, ECIR 2020},
  series       = {Lecture Notes in Computer Science},
  pages        = {605--618},
  publisher    = {Springer International Publishing},
  year         = {2020},
  url          = {https://doi.org/10.1007/978-3-030-45439-5\_40},
  doi          = {10.1007/978-3-030-45439-5\_40}
}

Byron E. Dom. An Information-Theoretic External Cluster-Validity Measure. IBM Research, 2001. [bib]

@techreport{dom2001information,
  author       = {Byron E. Dom},
  title        = {An Information-Theoretic External Cluster-Validity Measure},
  institution  = {IBM Research},
  year         = {2001}
}

Hui Fang, Tao Tao, and ChengXiang Zhai. A formal study of information retrieval heuristics. In SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 25-29, 2004, pages 49–56, 2004. ACM. [bib] [doi]

@inproceedings{fang2004formal,
  author       = {Hui Fang and
                  Tao Tao and
                  ChengXiang Zhai},
  title        = {A formal study of information retrieval heuristics},
  booktitle    = {{SIGIR} 2004: Proceedings of the 27th Annual International {ACM} {SIGIR}
                  Conference on Research and Development in Information Retrieval,
                  Sheffield, UK, July 25-29, 2004},
  pages        = {49--56},
  publisher    = {{ACM}},
  year         = {2004},
  url          = {https://doi.org/10.1145/1008992.1009004},
  doi          = {10.1145/1008992.1009004}
}

Hui Fang and ChengXiang Zhai. An exploration of axiomatic approaches to information retrieval. In SIGIR 2005: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil, August 15-19, 2005, pages 480–487, 2005. ACM. [bib] [doi]

@inproceedings{fang2005exploration,
  author       = {Hui Fang and
                  ChengXiang Zhai},
  title        = {An exploration of axiomatic approaches to information retrieval},
  booktitle    = {{SIGIR} 2005: Proceedings of the 28th Annual International {ACM} {SIGIR}
                  Conference on Research and Development in Information Retrieval,
                  Salvador, Brazil, August 15-19, 2005},
  pages        = {480--487},
  publisher    = {{ACM}},
  year         = {2005},
  url          = {https://doi.org/10.1145/1076034.1076116},
  doi          = {10.1145/1076034.1076116}
}

Hui Fang and ChengXiang Zhai. Semantic term matching in axiomatic approaches to information retrieval. In SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, August 6-11, 2006, pages 115–122, 2006. ACM. [bib] [doi]

@inproceedings{fang2006semantic,
  author       = {Hui Fang and
                  ChengXiang Zhai},
  title        = {Semantic term matching in axiomatic approaches to information retrieval},
  booktitle    = {{SIGIR} 2006: Proceedings of the 29th Annual International {ACM} {SIGIR}
                  Conference on Research and Development in Information Retrieval,
                  Seattle, Washington, USA, August 6-11, 2006},
  pages        = {115--122},
  publisher    = {{ACM}},
  year         = {2006},
  url          = {https://doi.org/10.1145/1148170.1148193},
  doi          = {10.1145/1148170.1148193}
}

Hui Fang. A Re-examination of Query Expansion Using Lexical Resources. In Proceedings of ACL-08: HLT, pages 139–147, 2008. Association for Computational Linguistics. [bib]

@inproceedings{fang2008reexamination,
  author       = {Hui Fang},
  title        = {A Re-examination of Query Expansion Using Lexical Resources},
  booktitle    = {Proceedings of ACL-08: HLT},
  pages        = {139--147},
  publisher    = {Association for Computational Linguistics},
  year         = {2008},
  url          = {https://aclanthology.org/P08-1017/}
}

Hui Fang, Tao Tao, and ChengXiang Zhai. Diagnostic Evaluation of Information Retrieval Models. ACM Trans. Inf. Syst., 29 (2) : 7:1–7:42, 2011. [bib] [doi]

@article{fang2011diagnostic,
  author       = {Hui Fang and
                  Tao Tao and
                  ChengXiang Zhai},
  title        = {Diagnostic Evaluation of Information Retrieval Models},
  journal      = {{ACM} Trans. Inf. Syst.},
  volume       = {29},
  number       = {2},
  pages        = {7:1--7:42},
  year         = {2011},
  url          = {https://doi.org/10.1145/1961209.1961210},
  doi          = {10.1145/1961209.1961210}
}

Marco Ferrante, Nicola Ferro, and Maria Maistro. Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pages 21–30, 2015. ACM. [bib] [doi]

@inproceedings{ferrante2015towards,
  author       = {Marco Ferrante and
                  Nicola Ferro and
                  Maria Maistro},
  title        = {Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness},
  booktitle    = {Proceedings of the 2015 International Conference on The Theory of Information Retrieval},
  pages        = {21--30},
  publisher    = {ACM},
  year         = {2015},
  url          = {https://doi.org/10.1145/2808194.2809452},
  doi          = {10.1145/2808194.2809452}
}

Shima Gerani, ChengXiang Zhai, and Fabio Crestani. Score Transformation in Linear Combination for Multi-criteria Relevance Ranking. In Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012, pages 256–267, 2012. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{gerani2012score,
  author       = {Shima Gerani and
                  ChengXiang Zhai and
                  Fabio Crestani},
  title        = {Score Transformation in Linear Combination for Multi-criteria Relevance Ranking},
  booktitle    = {Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012},
  series       = {Lecture Notes in Computer Science},
  pages        = {256--267},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2012},
  url          = {https://doi.org/10.1007/978-3-642-28997-2\_22},
  doi          = {10.1007/978-3-642-28997-2\_22}
}

Sreenivas Gollapudi and Aneesh Sharma. An axiomatic approach for result diversification. In Proceedings of the 18th International Conference on World Wide Web, WWW 2009, Madrid, Spain, April 20-24, 2009, pages 381–390, 2009. ACM. [bib] [doi]

@inproceedings{gollapudi2009axiomatic,
  author       = {Sreenivas Gollapudi and
                  Aneesh Sharma},
  title        = {An axiomatic approach for result diversification},
  booktitle    = {Proceedings of the 18th International Conference on World Wide Web,
                  {WWW} 2009, Madrid, Spain, April 20-24, 2009},
  pages        = {381--390},
  publisher    = {{ACM}},
  year         = {2009},
  url          = {https://doi.org/10.1145/1526709.1526761},
  doi          = {10.1145/1526709.1526761}
}

Parantapa Goswami, Massih-Reza Amini, and Eric Gaussier. Study of Heuristic IR Constraints Under Function Discovery Framework. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pages 365–368, 2015. ACM. [bib] [doi]

@inproceedings{goswami2015study,
  author       = {Parantapa Goswami and
                  Massih-Reza Amini and
                  Eric Gaussier},
  title        = {Study of Heuristic IR Constraints Under Function Discovery Framework},
  booktitle    = {Proceedings of the 2015 International Conference on The Theory of Information Retrieval},
  pages        = {365--368},
  publisher    = {ACM},
  year         = {2015},
  url          = {https://doi.org/10.1145/2808194.2809479},
  doi          = {10.1145/2808194.2809479}
}

Matthias Hagen, Michael Völske, Steve Göring, and Benno Stein. Axiomatic Result Re-Ranking. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, pages 721–730, 2016. ACM. [bib] [doi]

@inproceedings{hagen2016axiomatic,
  author       = {Matthias Hagen and
                  Michael V{\"{o}}lske and
                  Steve G{\"{o}}ring and
                  Benno Stein},
  title        = {Axiomatic Result Re-Ranking},
  booktitle    = {Proceedings of the 25th {ACM} International Conference on Information
                  and Knowledge Management, {CIKM} 2016, Indianapolis, IN, USA, October
                  24-28, 2016},
  pages        = {721--730},
  publisher    = {{ACM}},
  year         = {2016},
  url          = {https://doi.org/10.1145/2983323.2983704},
  doi          = {10.1145/2983323.2983704}
}

Hussein Hazimeh and ChengXiang Zhai. Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pages 141–150, 2015. ACM. [bib] [doi]

@inproceedings{hazimeh2015axiomatic,
  author       = {Hussein Hazimeh and
                  ChengXiang Zhai},
  title        = {Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback},
  booktitle    = {Proceedings of the 2015 International Conference on The Theory of Information Retrieval},
  pages        = {141--150},
  publisher    = {ACM},
  year         = {2015},
  url          = {https://doi.org/10.1145/2808194.2809471},
  doi          = {10.1145/2808194.2809471}
}

Maximilian Heinrich, Marvin Vogel, Alexander Bondarenko, Matthias Hagen, and Benno Stein. Axiomatic Re-Ranking for Argument Retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2025, Padua, Italy, July 13-18, 2025, pages 2864–2869, 2025. ACM. [bib] [doi]

@inproceedings{heinrich2025axiomatic,
  author       = {Maximilian Heinrich and
                  Marvin Vogel and
                  Alexander Bondarenko and
                  Matthias Hagen and
                  Benno Stein},
  title        = {Axiomatic Re-Ranking for Argument Retrieval},
  booktitle    = {Proceedings of the 48th International {ACM} {SIGIR} Conference on
                  Research and Development in Information Retrieval, {SIGIR} 2025, Padua,
                  Italy, July 13-18, 2025},
  pages        = {2864--2869},
  publisher    = {{ACM}},
  year         = {2025},
  url          = {https://doi.org/10.1145/3726302.3730169},
  doi          = {10.1145/3726302.3730169}
}

Maryam Karimzadehgan and ChengXiang Zhai. Axiomatic Analysis of Translation Language Model for Information Retrieval. In Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012, pages 268–280, 2012. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{karimzadehgan2012axiomatic,
  author       = {Maryam Karimzadehgan and
                  ChengXiang Zhai},
  title        = {Axiomatic Analysis of Translation Language Model for Information Retrieval},
  booktitle    = {Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012},
  series       = {Lecture Notes in Computer Science},
  pages        = {268--280},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2012},
  url          = {https://doi.org/10.1007/978-3-642-28997-2\_23},
  doi          = {10.1007/978-3-642-28997-2\_23}
}

Bo Li and Eric Gaussier. An Information-Based Cross-Language Information Retrieval Model. In Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012, pages 281–292, 2012. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{li2012information,
  author       = {Bo Li and
                  Eric Gaussier},
  title        = {An Information-Based Cross-Language Information Retrieval Model},
  booktitle    = {Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012},
  series       = {Lecture Notes in Computer Science},
  pages        = {281--292},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2012},
  url          = {https://doi.org/10.1007/978-3-642-28997-2\_24},
  doi          = {10.1007/978-3-642-28997-2\_24}
}

Yuanhua Lv and ChengXiang Zhai. Adaptive term frequency normalization for BM25. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1985–1988, 2011. ACM. [bib] [doi]

@inproceedings{lv2011adaptive,
  author       = {Yuanhua Lv and
                  ChengXiang Zhai},
  title        = {Adaptive term frequency normalization for BM25},
  booktitle    = {Proceedings of the 20th ACM international conference on Information and knowledge management},
  pages        = {1985--1988},
  publisher    = {ACM},
  year         = {2011},
  url          = {https://doi.org/10.1145/2063576.2063871},
  doi          = {10.1145/2063576.2063871}
}

Yuanhua Lv and ChengXiang Zhai. When documents are very long, BM25 fails!. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 1103–1104, 2011. ACM. [bib] [doi]

@inproceedings{lv2011documents,
  author       = {Yuanhua Lv and
                  ChengXiang Zhai},
  title        = {When documents are very long, BM25 fails!},
  booktitle    = {Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval},
  pages        = {1103--1104},
  publisher    = {ACM},
  year         = {2011},
  url          = {https://doi.org/10.1145/2009916.2010070},
  doi          = {10.1145/2009916.2010070}
}

Yuanhua Lv and ChengXiang Zhai. Lower-bounding term frequency normalization. In Proceedings of the 20th ACM Conference on Information and Knowledge Management, CIKM 2011, Glasgow, United Kingdom, October 24-28, 2011, pages 7–16, 2011. ACM. [bib] [doi]

@inproceedings{lv2011lower,
  author       = {Yuanhua Lv and
                  ChengXiang Zhai},
  title        = {Lower-bounding term frequency normalization},
  booktitle    = {Proceedings of the 20th {ACM} Conference on Information and Knowledge
                  Management, {CIKM} 2011, Glasgow, United Kingdom, October 24-28, 2011},
  pages        = {7--16},
  publisher    = {{ACM}},
  year         = {2011},
  url          = {https://doi.org/10.1145/2063576.2063584},
  doi          = {10.1145/2063576.2063584}
}

Yuanhua Lv and ChengXiang Zhai. A Log-Logistic Model-Based Interpretation of TF Normalization of BM25. In Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012, pages 244–255, 2012. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{lv2012loglogistic,
  author       = {Yuanhua Lv and
                  ChengXiang Zhai},
  title        = {A Log-Logistic Model-Based Interpretation of TF Normalization of BM25},
  booktitle    = {Advances in Information Retrieval, 34th European Conference on IR Research, ECIR 2012},
  series       = {Lecture Notes in Computer Science},
  pages        = {244--255},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2012},
  url          = {https://doi.org/10.1007/978-3-642-28997-2\_21},
  doi          = {10.1007/978-3-642-28997-2\_21}
}

Yuanhua Lv. A Study of Query Length Heuristics in Information Retrieval. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1747–1750, 2015. ACM. [bib] [doi]

@inproceedings{lv2015study,
  author       = {Yuanhua Lv},
  title        = {A Study of Query Length Heuristics in Information Retrieval},
  booktitle    = {Proceedings of the 24th ACM International on Conference on Information and Knowledge Management},
  pages        = {1747--1750},
  publisher    = {ACM},
  year         = {2015},
  url          = {https://doi.org/10.1145/2806416.2806592},
  doi          = {10.1145/2806416.2806592}
}

Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, and Arman Cohan. ABNIRML: Analyzing the Behavior of Neural IR Models. Transactions of the Association for Computational Linguistics, 10 : 224–239, 2022. [bib] [doi]

@article{macavaney2022abnirml,
  author       = {Sean MacAvaney and
                  Sergey Feldman and
                  Nazli Goharian and
                  Doug Downey and
                  Arman Cohan},
  title        = {ABNIRML: Analyzing the Behavior of Neural IR Models},
  journal      = {Transactions of the Association for Computational Linguistics},
  volume       = {10},
  pages        = {224--239},
  year         = {2022},
  url          = {https://doi.org/10.1162/tacl\_a\_00457},
  doi          = {10.1162/tacl\_a\_00457}
}

Victor Makarenkov, Bracha Shapira, and Lior Rokach. Theoretical Categorization of Query Performance Predictors. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pages 369–372, 2015. ACM. [bib] [doi]

@inproceedings{makarenkov2015theoretical,
  author       = {Victor Makarenkov and
                  Bracha Shapira and
                  Lior Rokach},
  title        = {Theoretical Categorization of Query Performance Predictors},
  booktitle    = {Proceedings of the 2015 International Conference on The Theory of Information Retrieval},
  pages        = {369--372},
  publisher    = {ACM},
  year         = {2015},
  url          = {https://doi.org/10.1145/2808194.2809475},
  doi          = {10.1145/2808194.2809475}
}

Marina Meilă. Comparing Clusterings by the Variation of Information. In Learning Theory and Kernel Machines, 16th Annual Conference on Computational Learning Theory, COLT 2003, pages 173–187, 2003. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{meila2003comparing,
  author       = {Marina Meil{\u{a}}},
  title        = {Comparing Clusterings by the Variation of Information},
  booktitle    = {Learning Theory and Kernel Machines, 16th Annual Conference on Computational Learning Theory, COLT 2003},
  series       = {Lecture Notes in Computer Science},
  pages        = {173--187},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2003},
  url          = {https://doi.org/10.1007/978-3-540-45167-9\_14},
  doi          = {10.1007/978-3-540-45167-9\_14}
}

Jan Heinrich Merker, Maik Fröbe, Benno Stein, Martin Potthast, and Matthias Hagen. Axioms for Retrieval-Augmented Generation. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval, ICTIR 2025, Padua, Italy, July 18, 2025, pages 67–77, 2025. ACM. [bib] [doi]

@inproceedings{merker2025axioms,
  author       = {Jan Heinrich Merker and
                  Maik Fr{\"{o}}be and
                  Benno Stein and
                  Martin Potthast and
                  Matthias Hagen},
  title        = {Axioms for Retrieval-Augmented Generation},
  booktitle    = {Proceedings of the 2025 International {ACM} {SIGIR} Conference on
                  Innovative Concepts and Theories in Information Retrieval, {ICTIR}
                  2025, Padua, Italy, July 18, 2025},
  pages        = {67--77},
  publisher    = {{ACM}},
  year         = {2025},
  url          = {https://doi.org/10.1145/3731120.3744601},
  doi          = {10.1145/3731120.3744601}
}

Alistair Moffat. Seven Numeric Properties of Effectiveness Metrics. In Information Retrieval Technology, 9th Asia Information Retrieval Societies Conference, AIRS 2013, pages 1–12, 2013. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{moffat2013seven,
  author       = {Alistair Moffat},
  title        = {Seven Numeric Properties of Effectiveness Metrics},
  booktitle    = {Information Retrieval Technology, 9th Asia Information Retrieval Societies Conference, AIRS 2013},
  series       = {Lecture Notes in Computer Science},
  pages        = {1--12},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2013},
  url          = {https://doi.org/10.1007/978-3-642-45068-6\_1},
  doi          = {10.1007/978-3-642-45068-6\_1}
}

Seung-Hoon Na, In-Su Kang, and Jong-Hyeok Lee. Improving Term Frequency Normalization for Multi-topical Documents and Application to Language Modeling Approaches. In Advances in Information Retrieval, 30th European Conference on IR Research, ECIR 2008, pages 382–393, None. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{na2008improving,
  author       = {Seung-Hoon Na and
                  In-Su Kang and
                  Jong-Hyeok Lee},
  title        = {Improving Term Frequency Normalization for Multi-topical Documents and Application to Language Modeling Approaches},
  booktitle    = {Advances in Information Retrieval, 30th European Conference on IR Research, ECIR 2008},
  series       = {Lecture Notes in Computer Science},
  pages        = {382--393},
  publisher    = {Springer Berlin Heidelberg},
  year         = {None},
  url          = {https://doi.org/10.1007/978-3-540-78646-7\_35},
  doi          = {10.1007/978-3-540-78646-7\_35}
}

Andrew Parry, Maik Fröbe, Martin Potthast, Sean MacAvaney, Benno Stein, Matthias Hagen, and Debasis Ganguly. Relaxed Term Matching for Neural Axiomatic Diagnostics. In Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval, ICTIR 2026, Melbourne, Australia, pages 274–284, 2026. ACM. [bib] [doi]

@inproceedings{parry2026relaxed,
  author       = {Andrew Parry and
                  Maik Fr{\"{o}}be and
                  Martin Potthast and
                  Sean MacAvaney and
                  Benno Stein and
                  Matthias Hagen and
                  Debasis Ganguly},
  title        = {Relaxed Term Matching for Neural Axiomatic Diagnostics},
  booktitle    = {Proceedings of the 2026 International {ACM} {SIGIR} Conference on
                  Innovative Concepts and Theories in Information Retrieval, {ICTIR} 2026,
                  Melbourne, Australia},
  pages        = {274--284},
  publisher    = {{ACM}},
  year         = {2026},
  url          = {https://doi.org/10.1145/3805713.3820405},
  doi          = {10.1145/3805713.3820405}
}

Razieh Rahimi, Azadeh Shakery, and Irwin King. Axiomatic Analysis of Cross-Language Information Retrieval. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 1875–1878, 2014. ACM. [bib] [doi]

@inproceedings{rahimi2014axiomatic,
  author       = {Razieh Rahimi and
                  Azadeh Shakery and
                  Irwin King},
  title        = {Axiomatic Analysis of Cross-Language Information Retrieval},
  booktitle    = {Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management},
  pages        = {1875--1878},
  publisher    = {ACM},
  year         = {2014},
  url          = {https://doi.org/10.1145/2661829.2661915},
  doi          = {10.1145/2661829.2661915}
}

Jan Heinrich Reimer, Johannes Huck, and Alexander Bondarenko. Grimjack at Touché 2022: Axiomatic Re-ranking and Query Reformulation. In Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, September 5th - to - 8th, 2022, 2022. CEUR-WS.org. [bib]

@inproceedings{reimer2022grimjack,
  author       = {Jan Heinrich Reimer and
                  Johannes Huck and
                  Alexander Bondarenko},
  title        = {Grimjack at Touch{\'{e}} 2022: Axiomatic Re-ranking and Query Reformulation},
  booktitle    = {Proceedings of the Working Notes of CLEF 2022 - Conference and Labs
                  of the Evaluation Forum, Bologna, Italy, September 5th - to - 8th, 2022},
  series       = {CEUR Workshop Proceedings},
  volume       = {3180},
  publisher    = {CEUR-WS.org},
  year         = {2022},
  url          = {https://downloads.webis.de/publications/papers/reimer_2022a.pdf}
}

Daniël Rennings, Felipe Moraes, and Claudia Hauff. An Axiomatic Approach to Diagnosing Neural IR Models. In Advances in Information Retrieval, 41st European Conference on IR Research, ECIR 2019, pages 489–503, 2019. Springer International Publishing. [bib] [doi]

@inproceedings{rennings2019axiomatic,
  author       = {Daniël Rennings and
                  Felipe Moraes and
                  Claudia Hauff},
  title        = {An Axiomatic Approach to Diagnosing Neural IR Models},
  booktitle    = {Advances in Information Retrieval, 41st European Conference on IR Research, ECIR 2019},
  series       = {Lecture Notes in Computer Science},
  pages        = {489--503},
  publisher    = {Springer International Publishing},
  year         = {2019},
  url          = {https://doi.org/10.1007/978-3-030-15712-8\_32},
  doi          = {10.1007/978-3-030-15712-8\_32}
}

Andrew Rosenberg and Julia Hirschberg. V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 410–420, 2007. Association for Computational Linguistics. [bib]

@inproceedings{rosenberg2007vmeasure,
  author       = {Andrew Rosenberg and
                  Julia Hirschberg},
  title        = {V-Measure: {A} Conditional Entropy-Based External Cluster Evaluation Measure},
  booktitle    = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural
                  Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  pages        = {410--420},
  publisher    = {Association for Computational Linguistics},
  year         = {2007},
  url          = {https://aclanthology.org/D07-1043/}
}

Corby Rosset, Bhaskar Mitra, Chenyan Xiong, Nick Craswell, Xia Song, and Saurabh Tiwary. An Axiomatic Approach to Regularizing Neural Ranking Models. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 981–984, 2019. ACM. [bib] [doi]

@inproceedings{rosset2019axiomatic,
  author       = {Corby Rosset and
                  Bhaskar Mitra and
                  Chenyan Xiong and
                  Nick Craswell and
                  Xia Song and
                  Saurabh Tiwary},
  title        = {An Axiomatic Approach to Regularizing Neural Ranking Models},
  booktitle    = {Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages        = {981--984},
  publisher    = {ACM},
  year         = {2019},
  url          = {https://doi.org/10.1145/3331184.3331296},
  doi          = {10.1145/3331184.3331296}
}

Corby Rosset, Guoqing Zheng, Victor Dibia, Ahmed Awadallah, and Paul Bennett. Axiomatic Preference Modeling for Longform Question Answering. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 11445–11475, 2023. Association for Computational Linguistics. [bib] [doi]

@inproceedings{rosset2023axiomatic,
  author       = {Corby Rosset and
                  Guoqing Zheng and
                  Victor Dibia and
                  Ahmed Awadallah and
                  Paul Bennett},
  title        = {Axiomatic Preference Modeling for Longform Question Answering},
  booktitle    = {Proceedings of the 2023 Conference on Empirical Methods in Natural
                  Language Processing, {EMNLP} 2023, Singapore, December 6-10, 2023},
  pages        = {11445--11475},
  publisher    = {Association for Computational Linguistics},
  year         = {2023},
  url          = {https://doi.org/10.18653/v1/2023.emnlp-main.702},
  doi          = {10.18653/V1/2023.EMNLP-MAIN.702}
}

Fabrizio Sebastiani. An Axiomatically Derived Measure for the Evaluation of Classification Algorithms. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pages 11–20, 2015. ACM. [bib] [doi]

@inproceedings{sebastiani2015axiomatically,
  author       = {Fabrizio Sebastiani},
  title        = {An Axiomatically Derived Measure for the Evaluation of Classification Algorithms},
  booktitle    = {Proceedings of the 2015 International Conference on The Theory of Information Retrieval},
  pages        = {11--20},
  publisher    = {ACM},
  year         = {2015},
  url          = {https://doi.org/10.1145/2808194.2809449},
  doi          = {10.1145/2808194.2809449}
}

Roger N. Shepard. Toward a Universal Law of Generalization for Psychological Science. Science, 237 (4820) : 1317–1323, 1987. [bib] [doi]

@article{shepard1987toward,
  author       = {Roger N. Shepard},
  title        = {Toward a Universal Law of Generalization for Psychological Science},
  journal      = {Science},
  volume       = {237},
  number       = {4820},
  pages        = {1317--1323},
  year         = {1987},
  url          = {https://doi.org/10.1126/science.3629243},
  doi          = {10.1126/science.3629243}
}

Shuming Shi, Ji-Rong Wen, Qing Yu, Ruihua Song, and Wei-Ying Ma. Gravitation-based model for information retrieval. In SIGIR 2005: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil, August 15-19, 2005, pages 488–495, 2005. ACM. [bib] [doi]

@inproceedings{shi2005gravitation,
  author       = {Shuming Shi and
                  Ji{-}Rong Wen and
                  Qing Yu and
                  Ruihua Song and
                  Wei{-}Ying Ma},
  title        = {Gravitation-based model for information retrieval},
  booktitle    = {{SIGIR} 2005: Proceedings of the 28th Annual International {ACM} {SIGIR}
                  Conference on Research and Development in Information Retrieval,
                  Salvador, Brazil, August 15-19, 2005},
  pages        = {488--495},
  publisher    = {{ACM}},
  year         = {2005},
  url          = {https://doi.org/10.1145/1076034.1076117},
  doi          = {10.1145/1076034.1076117}
}

Marina Sokolova. Assessing Invariance Properties of Evaluation Measures. In Proceedings of the NIPS 2006 Workshop on Testing Deployable Learning and Decision Systems, 2006. [bib]

@inproceedings{sokolova2006assessing,
  author       = {Marina Sokolova},
  title        = {Assessing Invariance Properties of Evaluation Measures},
  booktitle    = {Proceedings of the NIPS 2006 Workshop on Testing Deployable Learning
                  and Decision Systems},
  year         = {2006}
}

Tao Tao and ChengXiang Zhai. An exploration of proximity measures in information retrieval. In SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 23-27, 2007, pages 295–302, 2007. ACM. [bib] [doi]

@inproceedings{tao2007exploration,
  author       = {Tao Tao and
                  ChengXiang Zhai},
  title        = {An exploration of proximity measures in information retrieval},
  booktitle    = {{SIGIR} 2007: Proceedings of the 30th Annual International {ACM} {SIGIR}
                  Conference on Research and Development in Information Retrieval,
                  Amsterdam, The Netherlands, July 23-27, 2007},
  pages        = {295--302},
  publisher    = {{ACM}},
  year         = {2007},
  url          = {https://doi.org/10.1145/1277741.1277794},
  doi          = {10.1145/1277741.1277794}
}

Amos Tversky. Features of similarity. Psychological Review, 84 (4) : 327–352, 1977. [bib] [doi]

@article{tversky1977features,
  author       = {Amos Tversky},
  title        = {Features of similarity},
  journal      = {Psychological Review},
  volume       = {84},
  number       = {4},
  pages        = {327--352},
  year         = {1977},
  url          = {https://doi.org/10.1037/0033-295X.84.4.327},
  doi          = {10.1037/0033-295X.84.4.327}
}

Michael Völske, Alexander Bondarenko, Maik Fröbe, Benno Stein, Jaspreet Singh, Matthias Hagen, and Avishek Anand. Towards Axiomatic Explanations for Neural Ranking Models. In ICTIR '21: The 2021 ACM SIGIR International Conference on the Theory of Information Retrieval, Virtual Event, Canada, July 11, 2021, pages 13–22, 2021. ACM. [bib] [doi]

@inproceedings{volske2021towards,
  author       = {Michael V{\"{o}}lske and
                  Alexander Bondarenko and
                  Maik Fr{\"{o}}be and
                  Benno Stein and
                  Jaspreet Singh and
                  Matthias Hagen and
                  Avishek Anand},
  title        = {Towards Axiomatic Explanations for Neural Ranking Models},
  booktitle    = {{ICTIR} '21: The 2021 {ACM} {SIGIR} International Conference on the
                  Theory of Information Retrieval, Virtual Event, Canada, July 11, 2021},
  pages        = {13--22},
  publisher    = {{ACM}},
  year         = {2021},
  url          = {https://doi.org/10.1145/3471158.3472256},
  doi          = {10.1145/3471158.3472256}
}

Yue Wang, Xitong Liu, and Hui Fang. A Study of Concept-based Weighting Regularization for Medical Records Search. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 603–612, 2014. Association for Computational Linguistics. [bib] [doi]

@inproceedings{wang2014study,
  author       = {Yue Wang and
                  Xitong Liu and
                  Hui Fang},
  title        = {A Study of Concept-based Weighting Regularization for Medical Records Search},
  booktitle    = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages        = {603--612},
  publisher    = {Association for Computational Linguistics},
  year         = {2014},
  url          = {https://doi.org/10.3115/v1/P14-1057},
  doi          = {10.3115/v1/P14-1057}
}

Hao Wu and Hui Fang. Relation Based Term Weighting Regularization. In Advances in Information Retrieval - 34th European Conference on IR Research, ECIR 2012, Barcelona, Spain, April 1-5, 2012. Proceedings, pages 109–120, 2012. Springer. [bib] [doi]

@inproceedings{wu2012relation,
  author       = {Hao Wu and
                  Hui Fang},
  title        = {Relation Based Term Weighting Regularization},
  booktitle    = {Advances in Information Retrieval - 34th European Conference on {IR}
                  Research, {ECIR} 2012, Barcelona, Spain, April 1-5, 2012. Proceedings},
  series       = {Lecture Notes in Computer Science},
  volume       = {7224},
  pages        = {109--120},
  publisher    = {Springer},
  year         = {2012},
  url          = {https://doi.org/10.1007/978-3-642-28997-2\_10},
  doi          = {10.1007/978-3-642-28997-2\_10}
}

ChengXiang Zhai and Hui Fang. Axiomatic Analysis and Optimization of Information Retrieval Models. In Proceedings of the 2013 Conference on the Theory of Information Retrieval, pages 3–3, 2013. ACM. [bib] [doi]

@inproceedings{zhai2013axiomatic,
  author       = {ChengXiang Zhai and
                  Hui Fang},
  title        = {Axiomatic Analysis and Optimization of Information Retrieval Models},
  booktitle    = {Proceedings of the 2013 Conference on the Theory of Information Retrieval},
  pages        = {3--3},
  publisher    = {ACM},
  year         = {2013},
  url          = {https://doi.org/10.1145/2499178.2499205},
  doi          = {10.1145/2499178.2499205}
}

Dell Zhang, Robert Mao, Haitao Li, and Joanne Mao. How to Count Thumb-Ups and Thumb-Downs: User-Rating Based Ranking of Items from an Axiomatic Perspective. In Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011, pages 238–249, 2011. Springer Berlin Heidelberg. [bib] [doi]

@inproceedings{zhang2011count,
  author       = {Dell Zhang and
                  Robert Mao and
                  Haitao Li and
                  Joanne Mao},
  title        = {How to Count Thumb-Ups and Thumb-Downs: User-Rating Based Ranking of Items from an Axiomatic Perspective},
  booktitle    = {Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011},
  series       = {Lecture Notes in Computer Science},
  pages        = {238--249},
  publisher    = {Springer Berlin Heidelberg},
  year         = {2011},
  url          = {https://doi.org/10.1007/978-3-642-23318-0\_22},
  doi          = {10.1007/978-3-642-23318-0\_22}
}

Wei Zheng and Hui Fang. Query Aspect Based Term Weighting Regularization in Information Retrieval. In Advances in Information Retrieval, 32nd European Conference on IR Research, ECIR 2010, Milton Keynes, UK, March 28-31, 2010. Proceedings, pages 344–356, 2010. Springer. [bib] [doi]

@inproceedings{zheng2010query,
  author       = {Wei Zheng and
                  Hui Fang},
  title        = {Query Aspect Based Term Weighting Regularization in Information Retrieval},
  booktitle    = {Advances in Information Retrieval, 32nd European Conference on {IR}
                  Research, {ECIR} 2010, Milton Keynes, UK, March 28-31, 2010. Proceedings},
  series       = {Lecture Notes in Computer Science},
  volume       = {5993},
  pages        = {344--356},
  publisher    = {Springer},
  year         = {2010},
  url          = {https://doi.org/10.1007/978-3-642-12275-0\_31},
  doi          = {10.1007/978-3-642-12275-0\_31}
}