Quickstart¶
Compute axiomatic preferences
This quickstart computes axiomatic preferences for a handful of documents without any retrieval index.
All you need is a basic installation.
You can either run these commands in the Python REPL, or create a Jupyter notebook (pip install jupyter).
Queries and Documents¶
Axiomatic constraints in general compare two "outputs" (e.g., documents) for the same "input" (e.g., a query).
For retrieval axioms, inputs are Query and outputs are Document objects. Define some dummy data to play around with:
from ir_axioms.model import Query, Document
query = Query(
id="q1",
text="axiomatic information retrieval",
)
documents = [
Document(
id="d1",
text="Retrieval axioms formalize constraints for information retrieval models.",
rank=1,
score=12.3,
),
Document(
id="d2",
text="Axiomatic thinking in information retrieval: axiomatic constraints guide retrieval model design.",
rank=2,
score=11.0,
),
Document(
id="d3",
text="A recipe for apple pie with cinnamon.",
rank=3,
score=4.2,
),
]
The optional rank and score fields are used by the ORIG axiom, and the optional relevance field by the ORACLE axiom.
Pairwise Preferences¶
In ir_axioms, each axiom is implemented as a class, and most axioms are available under a short alias like TFC1 (see the axiom catalog).
Calling the alias creates an axiom instance with all its dependencies (e.g., tokenizers) resolved automatically.
Let's see which of the given documents, the TFC1 axiom prefers [Fang et al., 2004]:
The pairwise preference is either (learn more on the concept of preferences):
- positive if the first output should be ranked higher,
- negative if the second output should be ranked higher, or
- zero if the axiom has no preference, e.g., because a precondition is not met
So, TFC1 has preferred the second document because it contains more query term occurrences.
To compare all outputs at once, compute the preference matrix:
axiom.preferences(input=query, outputs=documents)
# Output: array([[ 0., -1., 1.],
# [ 1., 0., 1.],
# [-1., -1., 0.]])
The entry in row \(i\) and column \(j\) is the preference between the \(i\)-th and \(j\)-th output.
Combining Axioms¶
Axioms can be combined with Python operators.
For example, to require that two given axioms agree (&), and to fall back to the original ranking if they do not (|):
Or let multiple axioms vote and only keep preferences with an absolute majority:
from ir_axioms.axiom import LNC1, PROX4, MajorityVoteAxiom
axiom = MajorityVoteAxiom(
axioms=[TFC1(), LNC1(), PROX1(), PROX4()],
minimum_votes=0.5,
) | ORIG()
All operators are listed in the core concepts.
Re-Ranking¶
Based on an axiom's preferences, re-rank the documents with the KwikSort algorithm [Ailon et al., 2008; Hagen et al., 2016]:
axiom = TFC1()
reranked = axiom.rerank_kwiksort(input=query, ranking=documents)
[document.id for document in reranked]
# Output: ['d2', 'd1', 'd3']
Because TFC1 prefers the second document, it is pushed to the top of the ranking.
Axioms for Generated Responses¶
Axiomatic constraints can similarly be applied to retrieval-augmented generation (RAG), to compare the generated responses (GenerationResponse) from two RAG systems for the same request (GenerationRequest, optionally includes the retrieved context):
from ir_axioms.axiom import CLAR2, COV1, GEN_TFC1
from ir_axioms.model import GenerationRequest, GenerationResponse
request = GenerationRequest(
id="q1",
text="What are retrieval axioms?",
context=["Retrieval axioms are formal constraints that good retrieval models should satisfy."],
)
responses = [
GenerationResponse(
id="a",
text="Retrieval axioms are formal constraints on ranking functions [1]. They help to analyze retrieval models.",
),
GenerationResponse(id="b", text="Axioms. Stuff."),
]
COV1().preferences(input=request, outputs=responses) # Coverage
CLAR2().preferences(input=request, outputs=responses) # Clarity (readability)
GEN_TFC1().preferences(input=request, outputs=responses) # Adapted retrieval axiom
In this case, all axioms would prefer the first response.
Next Steps¶
- Learn the core concepts behind axioms, preferences, and tools.
- Use axioms with a real retrieval index via the PyTerrier integration.
- Follow the hands-on tutorial on TREC Deep Learning data.
References¶
@article{ailon2008aggregating,
author = {Nir Ailon and
Moses Charikar and
Alantha Newman},
title = {Aggregating inconsistent information: Ranking and clustering},
journal = {J. {ACM}},
volume = {55},
number = {5},
pages = {23:1--23:27},
year = {2008},
url = {https://doi.org/10.1145/1411509.1411513},
doi = {10.1145/1411509.1411513}
}@inproceedings{fang2004formal,
author = {Hui Fang and
Tao Tao and
ChengXiang Zhai},
title = {A formal study of information retrieval heuristics},
booktitle = {{SIGIR} 2004: Proceedings of the 27th Annual International {ACM} {SIGIR}
Conference on Research and Development in Information Retrieval,
Sheffield, UK, July 25-29, 2004},
pages = {49--56},
publisher = {{ACM}},
year = {2004},
url = {https://doi.org/10.1145/1008992.1009004},
doi = {10.1145/1008992.1009004}
}@inproceedings{hagen2016axiomatic,
author = {Matthias Hagen and
Michael V{\"{o}}lske and
Steve G{\"{o}}ring and
Benno Stein},
title = {Axiomatic Result Re-Ranking},
booktitle = {Proceedings of the 25th {ACM} International Conference on Information
and Knowledge Management, {CIKM} 2016, Indianapolis, IN, USA, October
24-28, 2016},
pages = {721--730},
publisher = {{ACM}},
year = {2016},
url = {https://doi.org/10.1145/2983323.2983704},
doi = {10.1145/2983323.2983704}
}