Core Concepts¶
Axioms, preferences, combinations, and tools
Axioms and Preferences¶
An axiom describes a pairwise constraint between two outputs given the same input, expressed as a preference.
In ir_axioms, every axiom extends the generic abstract base class Axiom[Input, Output]:
- Retrieval axioms compare two documents for a query:
Axiom[Query, Document]. - RAG axioms compare two generated responses for a request:
Axiom[GenerationRequest, GenerationResponse]. - Utility axioms (e.g.,
ORIGorRANDOM) work with any input/output types.
All implemented axioms are described in the axiom catalog.
The preference \(\mathrm{pref}_A(q, d_i, d_j)\) of an axiom \(A\) for an input \(q\) and outputs \(d_i\), \(d_j\) is a float with the following semantics [Bondarenko et al., 2022]:
A zero preference is returned when an axiom does not express a preference, for example, if the axiom's preconditions are not met. Preferences should be antisymmetric: flipping the outputs flips the sign of the preference.
Each axiom implements up to three methods:
| Method | Description |
|---|---|
preference(input, output1, output2) |
Pairwise preference between two outputs (must be implemented). |
preferences(input, outputs) |
Preference matrix for a sequence of outputs. The default implementation calls preference() for all pairs, but axioms can override it to compute costly features only once per output. |
batch_preferences(batch) |
Stacked preference matrices for multiple inputs (e.g., to batch model inference across queries). |
Instantiating Axioms¶
Each axiom class is also available under a short alias, e.g., TFC1 for Tfc1Axiom.
The aliases are created with lazy_inject() and resolve all dependencies of the axiom (e.g., tokenizers or index statistics) via dependency injection when called:
from ir_axioms.axiom import TFC1, Tfc1Axiom
axiom = TFC1() # Dependencies are injected automatically.
axiom = TFC1(margin_fraction=0.2) # Override parameters as keyword arguments.
Most axioms expose parameters to relax or strengthen their conditions, most notably margin_fraction, the relative tolerance below which two values (e.g., term frequencies) are considered equal.
The axiom catalog lists the parameters of each axiom.
Combining Axioms¶
Axioms can be combined with Python operators.
The result of a combination is again an Axiom, so combinations can be nested arbitrarily.
| Operator | Result | Description |
|---|---|---|
a + b |
SumAxiom |
Sum of the preferences. |
a - b |
SumAxiom |
Difference of the preferences (a + -b). |
a * b |
ProductAxiom |
Product of the preferences. |
a / b |
ProductAxiom |
Product with the multiplicative inverse of b's preferences. |
a % b |
VoteAxiom |
Majority vote (absolute majority, i.e., at least 50 % of the votes). |
a & b |
ConjunctionAxiom |
Preference only if all axioms agree, otherwise zero. |
a \| b |
CascadeAxiom |
First non-zero preference (i.e., b is the fallback when a has no preference). |
-a |
NegatedAxiom |
Flip the sign of the preferences. |
+a |
NormalizedAxiom |
Normalize the preferences to −1, 0, or 1 (same as a.normalized()). |
To weight an axiom by a constant, multiply with a UniformAxiom:
from ir_axioms.axiom import TFC1, TFC3, UniformAxiom
# Linear combination of TFC1 and TFC3.
axiom = TFC1() + UniformAxiom(scalar=2.0) * TFC3()
To control the required majority of a vote, instantiate the VoteAxiom (or MajorityVoteAxiom) directly:
from ir_axioms.axiom import MajorityVoteAxiom, ORIG, TFC1, LB1, PROX1, PROX2
axiom = MajorityVoteAxiom(
axioms=[TFC1(), LB1(), PROX1(), PROX2()],
minimum_votes=0.5, # 0.5 for absolute majority, 0 for relative majority, 1 for consensus.
) | ORIG()
Mathematical properties of combined axioms
Products and multiplicative inverses do not necessarily preserve asymmetry, irreflexivity, or transitivity of the preferences.
For example, for axiom * axiom, any negative preference's sign is flipped.
Caching and Parallelization¶
Some axioms are expensive to compute (e.g., axioms calling external APIs or neural models).
Cache an axiom's preferences on disk with cached(), so that each input–outputs tuple is computed only once:
from pathlib import Path
from ir_axioms.axiom import ArgUC
axiom = ArgUC().cached(Path("cache/ArgUC"))
Note
There is no implicit caching in ir_axioms ≥ 1.0. You have to choose the cache location explicitly.
To compute preference matrices in parallel, use parallel():
Preconditions¶
Several retrieval axioms only express a preference if some precondition holds, for example, if both documents have (approximately) the same length.
These axioms extend the PreconditionMixin and accept a precondition parameter:
from ir_axioms.axiom import LEN_AND
from ir_axioms.precondition import LEN, NOP
LEN_AND() # Default: length precondition with 10 % margin (LEN).
LEN_AND(precondition=LEN(margin_fraction=0.2)) # Relax the length precondition.
LEN_AND(precondition=NOP()) # Disable the precondition.
Any axiom can also serve as a precondition via axiom.as_precondition().
Tools and Dependency Injection¶
Axioms rely on tools like tokenizers, text statistics, or similarity measures.
These tools are declared as dataclass fields with interface types and are injected automatically using the injector library.
This way, the same axiom can, for example, use term statistics from a Terrier index or from a Pyserini index, depending on the integration you use.
The default ToolsModule binds the following tool interfaces.
All bindings use the singleton scope, i.e., each interface resolves to the same instance within one Injector.
Index statistics
Axioms that depend on collection statistics (e.g., inverse document frequencies for TFC3 or M-TDC) require an IndexStatistics binding, which is provided by the PyTerrier or Pyserini integrations.
To replace a tool for a single axiom, pass it as a keyword argument:
from ir_axioms.axiom import COV1
from ir_axioms.tools import SpacyEntitiesAspectExtraction
axiom = COV1(aspect_extraction=SpacyEntitiesAspectExtraction())
References¶
@inproceedings{bondarenko2022axiomatic,
author = {Alexander Bondarenko and
Maik Fr{\"{o}}be and
Jan Heinrich Reimer and
Benno Stein and
Michael V{\"{o}}lske and
Matthias Hagen},
title = {Axiomatic Retrieval Experimentation with ir{\_}axioms},
booktitle = {{SIGIR} '22: The 45th International {ACM} {SIGIR} Conference on Research
and Development in Information Retrieval, Madrid, Spain, July 11-15, 2022},
pages = {3131--3140},
publisher = {{ACM}},
year = {2022},
url = {https://doi.org/10.1145/3477495.3531743},
doi = {10.1145/3477495.3531743}
}