Utility Axioms¶

Reference points, baselines, and building blocks

Besides the axioms derived from the literature, ir_axioms includes "pseudo-axioms" that serve as reference points in experiments, as baselines, or as building blocks for combining axioms. Most of them work with any input and output types. The API reference documents all classes and parameters.

Reference Axioms¶

ORIG¶

ORIG OriginalAxiom — "Follow the original ranking." [Hagen et al., 2016]

Implementation: Prefers the document with the better (lower) rank, or, if no ranks are given, the higher score.
Usage: Compare (combinations of) axioms to an initial ranking, e.g., one returned by BM25, or use the original ranking as a fallback: axiom | ORIG().

ORACLE¶

ORACLE OracleAxiom — "Follow the human relevance judgments." [Bondarenko et al., 2022]

Implementation: Prefers the document with the higher relevance label (e.g., from TREC qrels).
Usage: Incorporate "ground-truth" relevance information into axiomatic experiments, e.g., as the target for learning to combine axioms.
Variants: TrecOracleAxiom: derives the same preferences directly from a trectools TrecQrel object.; For generated responses, see the oracle axioms for RAG.

Baseline Axioms¶

NOP¶

NOP NopAxiom — "Never prefer any output."

Implementation: Always returns a zero preference.

RANDOM¶

RANDOM RandomAxiom — "Prefer outputs at random."

Implementation: Returns random preferences (see the known issue).
Parameters: seed=None.

GT and LT¶

GT GreaterThanAxiom — "Prefer the greater output."

LT LessThanAxiom — "Prefer the smaller output."

Implementation: Compares the outputs directly, for any comparable outputs (e.g., numbers).

DIST¶

DIST DistanceAxiom — "Prefer the greater output, proportionally to the difference."

Implementation: Returns the signed difference of two numeric outputs.

Arithmetic Axioms¶

The following axioms implement the operators for combining axioms and can also be instantiated directly.

Uniform¶

UniformAxiom — "Return a constant preference."

Usage: Weight other axioms, e.g., TFC1() + UniformAxiom(scalar=2.0) * TFC3().
Parameters: scalar.

Sum and Product¶

SumAxiom (operators +, -) — "Add the preferences."

ProductAxiom (operator *) — "Multiply the preferences."

MultiplicativeInverseAxiom (operator /) — "Invert the preferences (1 / preference)."

Parameters: axioms (or axiom for the inverse).

Vote¶

VoteAxiom (operator %) and MajorityVoteAxiom — "Prefer what the majority of axioms prefers."

Implementation: Counts the positive and negative preferences of the axioms and prefers the output with more votes, if these reach the minimum proportion of votes.
Parameters: axioms, minimum_votes=0.5 (0.5 for absolute majority, 0 for relative majority, 1 for consensus).

Conjunction and Cascade¶

ConjunctionAxiom (operator &) — "Prefer only if all axioms agree."

CascadeAxiom (operator |) — "Use the first non-zero preference."

Parameters: axioms.

Negation and Normalization¶

NegatedAxiom (unary -) — "Flip the preferences."

NormalizedAxiom (unary +, axiom.normalized()) — "Normalize the preferences to −1, 0, or 1."

Parameters: axiom.

Learned Axioms¶

Estimator¶

ScikitLearnEstimatorAxiom — "Prefer what a trained classifier predicts."

Implementation: Combines the preferences of multiple axioms with a scikit-learn estimator (e.g., a random forest classifier) that is trained to predict the preferences of a target axiom, usually ORACLE.
Usage: In PyTerrier, use the EstimatorKwikSortReranker to train such an axiom and re-rank with it (see the walkthrough).
Parameters: axioms, estimator.

Wrapper Axioms¶

Cached¶

CachedAxiom (axiom.cached(path)) — "Compute each preference only once."

Implementation: Stores the preferences of the wrapped axiom in an on-disk database at the given path.

Parallel¶

ParallelAxiom (axiom.parallel(n_jobs)) — "Compute preference matrices in parallel."

Parameters: n_jobs=None (number of parallel jobs).

References¶

Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völske, and Matthias Hagen. Axiomatic Retrieval Experimentation with ir_axioms. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11-15, 2022, pages 3131–3140, 2022. ACM. [bib] [doi]

@inproceedings{bondarenko2022axiomatic,
  author       = {Alexander Bondarenko and
                  Maik Fr{\"{o}}be and
                  Jan Heinrich Reimer and
                  Benno Stein and
                  Michael V{\"{o}}lske and
                  Matthias Hagen},
  title        = {Axiomatic Retrieval Experimentation with ir{\_}axioms},
  booktitle    = {{SIGIR} '22: The 45th International {ACM} {SIGIR} Conference on Research
                  and Development in Information Retrieval, Madrid, Spain, July 11-15, 2022},
  pages        = {3131--3140},
  publisher    = {{ACM}},
  year         = {2022},
  url          = {https://doi.org/10.1145/3477495.3531743},
  doi          = {10.1145/3477495.3531743}
}

Matthias Hagen, Michael Völske, Steve Göring, and Benno Stein. Axiomatic Result Re-Ranking. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, pages 721–730, 2016. ACM. [bib] [doi]

@inproceedings{hagen2016axiomatic,
  author       = {Matthias Hagen and
                  Michael V{\"{o}}lske and
                  Steve G{\"{o}}ring and
                  Benno Stein},
  title        = {Axiomatic Result Re-Ranking},
  booktitle    = {Proceedings of the 25th {ACM} International Conference on Information
                  and Knowledge Management, {CIKM} 2016, Indianapolis, IN, USA, October
                  24-28, 2016},
  pages        = {721--730},
  publisher    = {{ACM}},
  year         = {2016},
  url          = {https://doi.org/10.1145/2983323.2983704},
  doi          = {10.1145/2983323.2983704}
}