Walkthroughs¶

Step-by-step tutorials based on our example notebooks

These walkthroughs demonstrate the main use cases of ir_axioms on standard benchmark data. They build on the integrations and use axioms from the axiom catalog. Each walkthrough is based on one of the Jupyter notebooks in the repository.

Walkthrough	You will learn to…	Notebook
Hands-on tutorial	analyze retrieval systems axiomatically, find and fix ranking errors by axiomatic re-ranking	`grenoble2025_showcase.ipynb`
KwikSort re-ranking	re-rank BM25 results with a vote of many axioms	`pyterrier_kwiksort.ipynb`
Learning the ORACLE	train a classifier that combines axioms to predict relevance preferences	`pyterrier_kwiksort_learned.ipynb`
Axiomatic LTR features	use aggregated axiom preferences as features for LambdaMART	`pyterrier_ltr_features.ipynb`
Post-hoc analysis	analyze the consistency of runs and qrels with axioms	`pyterrier_post_hoc_analysis_of_runs_and_qrels.ipynb`
Parallel preferences	compute preferences for many TREC runs in parallel	`pyterrier_preferences_parallel.ipynb`
Evaluating RAG responses	compare RAG axioms to human and nugget-based judgments	`ictir2025.ipynb`

API versions

The walkthroughs use the current ir_axioms ≥ 1.0 API. Some of the linked notebooks were written for the legacy API (e.g., using ~axiom for caching or ir_axioms.backend.pyterrier imports). See the migration guide for the differences.

Common Setup¶

The retrieval walkthroughs use the passage ranking task of the TREC Deep Learning tracks 2019 and 2020 on MS MARCO, retrieved with PyTerrier.

Install Dependencies¶

pip install "ir_axioms[pyterrier,experiments]"
python -m spacy download en_core_web_sm

Load Datasets¶

We load the datasets with PyTerrier's get_dataset(), which uses ir_datasets under the hood:

from pyterrier.datasets import get_dataset

dataset_name = "msmarco-passage"
dataset = get_dataset(f"irds:{dataset_name}")
dataset_train = get_dataset(f"irds:{dataset_name}/trec-dl-2019/judged")
dataset_test = get_dataset(f"irds:{dataset_name}/trec-dl-2020/judged")

Build the Index¶

If the index does not exist yet, index all MS MARCO passages (this takes a while):

from pathlib import Path
from pyterrier.terrier import IterDictIndexer

cache_dir = Path("cache/")
index_dir = cache_dir / "indices" / dataset_name

if not index_dir.exists():
    indexer = IterDictIndexer(str(index_dir.absolute()))
    indexer.index(dataset.get_corpus_iter(), fields=["text"])

Baseline Retrieval¶

We use BM25 as the baseline retrieval model:

from pyterrier.terrier import Retriever

bm25 = Retriever(str(index_dir.absolute()), wmodel="BM25")

Connect `ir_axioms` to the Index¶

Finally, let ir_axioms use the index for tokenization and collection statistics, and ir_datasets for the query and document texts:

from ir_axioms.integrations import inject_pyterrier

inject_pyterrier(
    index_location=index_dir,
    dataset=dataset_name,
)

Call inject_pyterrier() before instantiating any axioms, as the axioms' dependencies are resolved when they are created.