Skip to content

Axioms

API of ir_axioms.axiom

For descriptions, original formulations, and references of the axioms, see the axiom catalog. For how to use and combine axioms, see the core concepts.

Base Class

ir_axioms.axiom.base.Axiom

Bases: ABC, Generic[Input, Output]

An axiom describes a pairwise constraint between two outputs given the same input, as expressed as a preference.

Subclasses must implement the preference() method that determines the pairwise preference between two outputs, and can optionally also override preference_matrix() to batch-compute a full preference matrix of arbitrarily many outputs more efficiently.

This base class also exposes various operators (i.e., +, -, *, /, %, &, ~) for combining and manipulating axioms, as well as rerank() for KwikSort re-ranking the outputs, and other methods for evaluating rankings of outputs in comparison to the axiom's preferences.

verbose class-attribute instance-attribute 

verbose: bool = False

preference abstractmethod 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference

Compute the pairwise preference between two outputs for the same input. The function should return a positive value if output1 is to be prefered, a negative value if output2 is to be prefered, and zero if there is no preference or if some precondition was not met.

Note that the preference should be independent of the order of output1 and output2, i.e., if the inputs are flipped, the output should be flipped as well.

Parameters:

Name Type Description Default
input Input

Common input for both outputs.

 required
output1 Output

One output for the common input.

 required
output2 Output

Another output for the common input.

 required

Returns:

Type Description
Preference

0 if output1 should be preferred, <0 if output2 should be preferred, or 0 if neither of the outputs should be preferred over the other.
Source code in ir_axioms/axiom/base.py 
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
@abstractmethod
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    """
    Compute the pairwise preference between two outputs for the same input.
    The function should return a positive value if `output1` is to be prefered, a negative value if `output2` is to be prefered, and zero if there is no preference or if some precondition was not met.

    Note that the preference should be independent of the order of ``output1`` and ``output2``, i.e., if the inputs are flipped, the output should be flipped as well.

    :param input: Common input for both outputs.
    :param output1: One output for the common input.
    :param output2: Another output for the common input.
    :return: >0 if ``output1`` should be preferred,
    <0 if ``output2`` should be preferred,
    or 0 if neither of the outputs should be preferred over the other.
    """
    pass

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix

Batch-compute the preferences for a sequence of potential outputs. While the naive default implementation just delegates to preference(), it might make sense to override it to avoid computating certain costly features again and again for the same output.

Parameters:

Name Type Description Default
input Input

Common input for all outputs.

 required
outputs Sequence[Output]

The outputs for the common input.

 required

Returns:

Type Description
PreferenceMatrix

A preference matrix, where the ij-th entry corresponds to the preference between the i-th and j-th output.
Source code in ir_axioms/axiom/base.py 
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    """
    Batch-compute the preferences for a sequence of potential outputs.
    While the naive default implementation just delegates to `preference()`, it might make sense to override it to avoid computating certain costly features again and again for the same output.

    :param input: Common input for all outputs.
    :param outputs: The outputs for the common input.
    :return: A preference matrix, where the ij-th entry corresponds to the preference between the i-th and j-th output.
    """

    return array(
        list(
            tqdm(
                (
                    self.preference(
                        input=input,
                        output1=output1,
                        output2=output2,
                    )
                    for output1 in outputs
                    for output2 in outputs
                ),
                desc="Preferences",
                total=len(outputs) * len(outputs),
                disable=not self.verbose,
            )
        ),
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

batch_preferences 

batch_preferences(
    batch: Sequence[tuple[Input, Sequence[Output]]],
) -> PreferenceMatrices

Batch-compute the preference matrices for multiple inputs, each with its own sequence of outputs. While the naive default implementation just delegates to preferences() for each input-outputs pair, it might make sense to override it to batch costly computations (e.g., model inference) across inputs.

Parameters:

Name Type Description Default
batch Sequence[tuple[Input, Sequence[Output]]]

Pairs of an input and the outputs for that input.

 required

Returns:

Type Description
PreferenceMatrices

A stacked array of preference matrices, where the kij-th entry corresponds to the preference between the i-th and j-th output of the k-th input.
Source code in ir_axioms/axiom/base.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
def batch_preferences(
    self,
    batch: Sequence[tuple[Input, Sequence[Output]]],
) -> PreferenceMatrices:
    """
    Batch-compute the preference matrices for multiple inputs, each with its own sequence of outputs.
    While the naive default implementation just delegates to `preferences()` for each input-outputs pair, it might make sense to override it to batch costly computations (e.g., model inference) across inputs.

    :param batch: Pairs of an input and the outputs for that input.
    :return: A stacked array of preference matrices, where the kij-th entry corresponds to the preference between the i-th and j-th output of the k-th input.
    """

    if len(batch) == 0:
        return empty((0, 0, 0), dtype=float_)
    num_outputs = len(batch[0][1])
    if any(len(outputs) != num_outputs for _, outputs in batch):
        raise ValueError("All sequences of outputs must have the same length.")

    return stack(
        [
            self.preferences(
                input=input,
                outputs=outputs,
            )
            for input, outputs in tqdm(
                batch,
                desc="Batch preferences",
                total=len(batch),
                disable=not self.verbose,
            )
        ]
    )

majority_vote 

majority_vote(
    other: Axiom[Input, Output],
) -> Axiom[Input, Output]
Source code in ir_axioms/axiom/base.py 
160
161
def majority_vote(self, other: "Axiom[Input, Output]") -> "Axiom[Input, Output]":
    return self % other

normalized 

normalized() -> Axiom[Input, Output]

Return the normalized preference of this axiom, replacing positive values with 1 and negative values with -1.

Source code in ir_axioms/axiom/base.py 
193
194
195
196
197
198
def normalized(self) -> "Axiom[Input, Output]":
    """
    Return the normalized preference of this axiom,
    replacing positive values with 1 and negative values with -1.
    """
    return +self

cached 

cached(cache_path: Path) -> Axiom[Input, Output]

Cache this axiom's preferences in the given cache path, meaning the preference() method will only be called once for each query-documents tuple.

Source code in ir_axioms/axiom/base.py 
200
201
202
203
204
205
206
207
208
def cached(self, cache_path: Path) -> "Axiom[Input, Output]":
    """
    Cache this axiom's preferences in the given cache path,
    meaning the ``preference()`` method will only be called once
    for each query-documents tuple.
    """
    from ir_axioms.axiom.cache import CachedAxiom

    return CachedAxiom(axiom=self, cache_path=cache_path)

parallel 

parallel(n_jobs: int | None = None) -> Axiom[Input, Output]

Parallelize preference matrix computation of this axiom.

Source code in ir_axioms/axiom/base.py 
210
211
212
213
214
215
216
def parallel(self, n_jobs: int | None = None) -> "Axiom[Input, Output]":
    """
    Parallelize preference matrix computation of this axiom.
    """
    from ir_axioms.axiom.parallel import ParallelAxiom

    return ParallelAxiom(axiom=self, n_jobs=n_jobs)

as_precondition 

as_precondition(
    expected_sign: Literal[1, 0, -1] = 0,
    strip_preconditions: bool = True,
) -> Precondition[Input, Output]
Source code in ir_axioms/axiom/base.py 
218
219
220
221
222
223
224
225
226
227
228
229
def as_precondition(
    self,
    expected_sign: Literal[1, 0, -1] = 0,
    strip_preconditions: bool = True,
) -> Precondition[Input, Output]:
    from ir_axioms.precondition.axiom import AxiomPrecondition

    return AxiomPrecondition(
        axiom=self,
        expected_sign=expected_sign,
        strip_preconditions=strip_preconditions,
    )

rerank_kwiksort 

rerank_kwiksort(
    input: Input,
    ranking: Sequence[Output],
    pivot_selection: PivotSelection[
        Input, Output
    ] = RandomPivotSelection(),
) -> Sequence[Output]
Source code in ir_axioms/axiom/base.py 
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
@final
def rerank_kwiksort(
    self,
    input: Input,
    ranking: Sequence[Output],
    pivot_selection: PivotSelection[Input, Output] = RandomPivotSelection(),
) -> Sequence[Output]:
    from ir_axioms.algorithms.ranking import kwiksort

    ranking = kwiksort(
        axiom=self,
        input=input,
        vertices=ranking,
        pivot_selection=pivot_selection,
    )
    return ranking

Combinators

See combining axioms and the arithmetic axioms.

ir_axioms.axiom.arithmetic

UniformAxiom dataclass 

UniformAxiom(*, scalar: float)

Bases: Axiom[Any, Any]

scalar instance-attribute 

scalar: float

preference 

preference(
    input: Any, output1: Any, output2: Any
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
19
20
21
22
23
24
25
def preference(
    self,
    input: Any,
    output1: Any,
    output2: Any,
) -> Preference:
    return self.scalar

preferences 

preferences(
    input: Any, outputs: Sequence[Any]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
27
28
29
30
31
32
def preferences(
    self,
    input: Any,
    outputs: Sequence[Any],
) -> PreferenceMatrix:
    return full((len(outputs), len(outputs)), self.scalar)

SumAxiom dataclass 

SumAxiom(*, axioms: Iterable[Axiom[Input, Output]])

Bases: Axiom[Input, Output]

axioms instance-attribute 

axioms: Iterable[Axiom[Input, Output]]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
39
40
41
42
43
44
45
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    return sum(axiom.preference(input, output1, output2) for axiom in self.axioms)

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
47
48
49
50
51
52
53
54
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    return stack([axiom.preferences(input, outputs) for axiom in self.axioms]).sum(
        axis=0
    )

ProductAxiom dataclass 

ProductAxiom(*, axioms: Iterable[Axiom[Input, Output]])

Bases: Axiom[Input, Output]

Multiply the preferences of the given axioms.

Notes: - The product of axiom preferences does not necessarily preserve asymmetry, e.g., for axiom * -axiom, the ij-th and ji-th entries are the same. - The product of axiom preferences does not necessarily preserve transitivity, e.g., for axiom * axiom, any negative preference value's sign is flipped, indicating a > b and b < a at the same time.

axioms instance-attribute 

axioms: Iterable[Axiom[Input, Output]]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
72
73
74
75
76
77
78
79
80
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    return reduce(
        mul, (axiom.preference(input, output1, output2) for axiom in self.axioms), 1
    )

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
82
83
84
85
86
87
88
89
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    return stack([axiom.preferences(input, outputs) for axiom in self.axioms]).prod(
        axis=0
    )

MultiplicativeInverseAxiom dataclass 

MultiplicativeInverseAxiom(*, axiom: Axiom[Input, Output])

Bases: Axiom[Input, Output]

Invert the preferences of the given axiom (i.e., 1 / preference).

Notes: - The multiplicative inverse of axiom preferences does not preserve irreflexivity, e.g., for any previously irreflexive axiom, the multiplicative inverse is undefined along the diagonal (division by zero).

axiom instance-attribute 

axiom: Axiom[Input, Output]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
107
108
109
110
111
112
113
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    return 1 / self.axiom.preference(input, output1, output2)

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
115
116
117
118
119
120
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    return 1 / self.axiom.preferences(input, outputs)

ConjunctionAxiom dataclass 

ConjunctionAxiom(*, axioms: Iterable[Axiom[Input, Output]])

Bases: Axiom[Input, Output]

axioms instance-attribute 

axioms: Iterable[Axiom[Input, Output]]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    preferences = [
        axiom.preference(input, output1, output2) for axiom in self.axioms
    ]
    if all(preference > 0 for preference in preferences):
        return 1
    elif all(preference < 0 for preference in preferences):
        return -1
    else:
        return 0

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
147
148
149
150
151
152
153
154
155
156
157
158
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    preferences = stack(
        [axiom.preferences(input, outputs) for axiom in self.axioms]
    )
    aggregated_preferences = zeros((len(outputs), len(outputs)))
    aggregated_preferences += (preferences > 0).all(axis=0) * 1
    aggregated_preferences += (preferences < 0).all(axis=0) * -1
    return aggregated_preferences

VoteAxiom dataclass 

VoteAxiom(
    *,
    axioms: Iterable[Axiom[Input, Output]],
    minimum_votes: float = 0.5,
)

Bases: Axiom[Input, Output]

axioms instance-attribute 

axioms: Iterable[Axiom[Input, Output]]

minimum_votes class-attribute instance-attribute 

minimum_votes: float = 0.5

Minimum portion of votes in favor or against either document, to be considered a majority, for example, 0.5 for absolute majority, 0.6 for qualified majority, 0 for relative majority, or 1 for consensus.

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    preferences = [
        axiom.preference(input, output1, output2) for axiom in self.axioms
    ]

    # Total count of possible votes.
    count: int = len(preferences)

    # Minimum (absolute) number of votes to reach a majority.
    minimum_votes: int = ceil(self.minimum_votes * count)

    # Number of observed positive votes.
    positive_votes: int = sum(1 for preference in preferences if preference > 0)
    # Number of observed negative votes.
    negative_votes: int = sum(1 for preference in preferences if preference < 0)

    if positive_votes > negative_votes and positive_votes >= minimum_votes:
        return 1
    elif negative_votes > positive_votes and negative_votes >= minimum_votes:
        return -1
    else:
        # Draw.
        return 0

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    preferences = stack(
        [
            axiom.preferences(input, outputs)
            for axiom in tqdm(
                self.axioms,
                desc="Compute preferences",
                disable=not self.verbose,
            )
        ]
    )

    # Total count of possible votes.
    count: int = preferences.shape[0]

    # Minimum (absolute) number of votes to reach a majority.
    minimum_votes: int = ceil(self.minimum_votes * count)

    # Number of observed positive votes.
    positive_votes = (preferences > 0).sum(axis=0)
    # Number of observed negative votes.
    negative_votes = (preferences < 0).sum(axis=0)

    mask_positive = (positive_votes > negative_votes) & (
        positive_votes >= minimum_votes
    )
    mask_negative = (negative_votes > positive_votes) & (
        negative_votes >= minimum_votes
    )

    aggregated_preferences = zeros((len(outputs), len(outputs)))
    aggregated_preferences += mask_positive * 1
    aggregated_preferences += mask_negative * -1

    return aggregated_preferences

MajorityVoteAxiom dataclass 

MajorityVoteAxiom(
    *,
    axioms: Iterable[Axiom[Input, Output]],
    minimum_votes: float = 0.5,
)

Bases: VoteAxiom[Input, Output]

CascadeAxiom dataclass 

CascadeAxiom(*, axioms: Iterable[Axiom[Input, Output]])

Bases: Axiom[Input, Output]

axioms instance-attribute 

axioms: Iterable[Axiom[Input, Output]]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
262
263
264
265
266
267
268
269
270
271
272
273
274
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    preferences = (
        axiom.preference(input, output1, output2) for axiom in self.axioms
    )
    decisive_preferences = (
        preference for preference in preferences if preference != 0
    )
    return next(decisive_preferences, 0)

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    preferences = zeros((len(outputs), len(outputs)))
    for axiom in self.axioms:
        undecided = preferences == 0
        fill_diagonal(undecided, False)
        if not undecided.any():
            # Early terminate if all non-recursive elements of the preference matrix have been filled.
            break
        # Fill undecided preferences with the next axiom's preferences.
        preferences[undecided] = axiom.preferences(input, outputs)[undecided]
    return preferences

NegatedAxiom dataclass 

NegatedAxiom(*, axiom: Axiom[Input, Output])

Bases: Axiom[Input, Output]

axiom instance-attribute 

axiom: Axiom[Input, Output]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
301
302
303
304
305
306
307
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    return -self.axiom.preference(input, output1, output2)

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
309
310
311
312
313
314
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    return -self.axiom.preferences(input, outputs)

NormalizedAxiom dataclass 

NormalizedAxiom(*, axiom: Axiom[Input, Output])

Bases: Axiom[Input, Output]

axiom instance-attribute 

axiom: Axiom[Input, Output]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/arithmetic.py 
325
326
327
328
329
330
331
332
333
334
335
336
337
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    preference = self.axiom.preference(input, output1, output2)
    if preference > 0:
        return 1
    elif preference < 0:
        return -1
    else:
        return 0

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/arithmetic.py 
339
340
341
342
343
344
345
346
347
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    preferences = self.axiom.preferences(input, outputs)
    preferences[preferences > 0] = 1
    preferences[preferences < 0] = -1
    return preferences

Preconditions

ir_axioms.axiom.precondition.PreconditionMixin dataclass 

PreconditionMixin(
    *, precondition: Precondition[Input, Output]
)

Bases: Axiom[Input, Output], ABC

precondition instance-attribute 

precondition: Precondition[Input, Output]

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/precondition.py 
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    if not self.precondition.precondition(
        input=input,
        output1=output1,
        output2=output2,
    ):
        return 0
    return super().preference(  # type: ignore[safe-super]
        input=input,
        output1=output1,
        output2=output2,
    )

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/precondition.py 
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    mask = self.precondition.preconditions(
        input=input,
        outputs=outputs,
    )
    if not mask.any():
        return zeros_like(mask).astype(float)
    # ADDITION: We could try to isolate the non-masked entries here, and only compute preferences where at least one output of the tuple is not masked.
    preferences = super().preferences(
        input=input,
        outputs=outputs,
    )
    # Only keep preferences where the precondition holds.
    return where(mask, preferences, 0)

ir_axioms.precondition

LEN module-attribute 

LEN: Final = lazy_inject(LenPrecondition)

NOP module-attribute 

NOP: Final = lazy_inject(NopPrecondition)

AxiomPrecondition dataclass 

AxiomPrecondition(
    *,
    axiom: Axiom[Input, Output],
    expected_sign: Literal[1, 0, -1] = 0,
    strip_preconditions: bool = True,
)

Bases: Precondition[Input, Output], Generic[Input, Output]

axiom instance-attribute 

axiom: Axiom[Input, Output]

expected_sign class-attribute instance-attribute 

expected_sign: Literal[1, 0, -1] = 0

strip_preconditions class-attribute instance-attribute 

strip_preconditions: bool = True

precondition 

precondition(
    input: Input, output1: Output, output2: Output
) -> Mask
Source code in ir_axioms/precondition/axiom.py 
25
26
27
28
29
30
31
32
def precondition(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Mask:
    preference = self.axiom.preference(input, output1, output2)
    return sign(preference) == self.expected_sign

preconditions 

preconditions(
    input: Input, outputs: Sequence[Output]
) -> MaskMatrix
Source code in ir_axioms/precondition/axiom.py 
34
35
36
37
38
39
40
def preconditions(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> MaskMatrix:
    preferences = self.axiom.preferences(input, outputs)
    return sign(preferences) == self.expected_sign

Precondition

Bases: Generic[Input, Output], Protocol

A precondition can be used to restrict an axiom's preference computation to only those outputs that meet a certain condition (e.g., similar length).

verbose class-attribute instance-attribute 

verbose: bool = False

precondition 

precondition(
    input: Input, output1: Output, output2: Output
) -> Mask

Check if the precondition holds for a pair of outputs given an input.

Parameters:

Name Type Description Default
input Input

Common input for both outputs.

 required
output1 Output

One output for the common input.

 required
output2 Output

Another output for the common input.

 required

Returns:

Type Description
Mask

True if a preference can be computed for the outputs, False otherwise.
Source code in ir_axioms/precondition/base.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
def precondition(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Mask:
    """
    Check if the precondition holds for a pair of outputs given an input.

    :param input: Common input for both outputs.
    :param output1: One output for the common input.
    :param output2: Another output for the common input.
    :return: ``True`` if a preference can be computed for the outputs, ``False`` otherwise.
    """
    ...

preconditions 

preconditions(
    input: Input, outputs: Sequence[Output]
) -> MaskMatrix

Check if the precondition holds for all pairs of outputs given an input.

Parameters:

Name Type Description Default
input Input

Common input for all outputs.

 required
outputs Sequence[Output]

The outputs for the common input.

 required

Returns:

Type Description
MaskMatrix

A mask matrix, where the ij-th entry corresponds to whether the precondition holds for the i-th and j-th output.
Source code in ir_axioms/precondition/base.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
def preconditions(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> MaskMatrix:
    """
    Check if the precondition holds for all pairs of outputs given an input.

    :param input: Common input for all outputs.
    :param outputs: The outputs for the common input.
    :return: A mask matrix, where the ij-th entry corresponds to whether the precondition holds for the i-th and j-th output.
    """
    return array(
        list(
            tqdm(
                (
                    self.precondition(
                        input=input,
                        output1=output1,
                        output2=output2,
                    )
                    for output1 in outputs
                    for output2 in outputs
                ),
                desc="Preconditions",
                total=len(outputs) * len(outputs),
                disable=not self.verbose,
            )
        )
    ).reshape((len(outputs), len(outputs)))

LenPrecondition dataclass 

LenPrecondition(
    *,
    text_contents: TextContents[Document],
    term_tokenizer: TermTokenizer,
    margin_fraction: NoInject[float] = 0.1,
)

Bases: Precondition[Input, Document]

text_contents instance-attribute 

text_contents: TextContents[Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

precondition 

precondition(
    input: Input, output1: Document, output2: Document
) -> Mask
Source code in ir_axioms/precondition/length.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
def precondition(
    self,
    input: Input,
    output1: Document,
    output2: Document,
) -> Mask:
    return isclose(
        len(
            self.term_tokenizer.terms_unordered(
                self.text_contents.contents(output1)
            )
        ),
        len(
            self.term_tokenizer.terms_unordered(
                self.text_contents.contents(output2)
            )
        ),
        rel_tol=self.margin_fraction,
    )

preconditions 

preconditions(
    input: Input, outputs: Sequence[Document]
) -> MaskMatrix
Source code in ir_axioms/precondition/length.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def preconditions(
    self,
    input: Input,
    outputs: Sequence[Document],
) -> MaskMatrix:
    lengths = [
        len(
            self.term_tokenizer.terms_unordered(self.text_contents.contents(output))
        )
        for output in tqdm(
            outputs,
            desc="Lengths",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            isclose(
                length1,
                length2,
                rel_tol=self.margin_fraction,
            )
            for length1, length2 in tqdm(
                product(lengths, repeat=2),
                total=len(lengths) * len(lengths),
                desc="Compare lengths",
                unit="pair",
                disable=not self.verbose,
            )
        ],
        dtype=bool_,
    ).reshape((len(outputs), len(outputs)))

NopPrecondition dataclass 

NopPrecondition()

Bases: Precondition[Input, Document]

precondition 

precondition(
    input: Input, output1: Document, output2: Document
) -> Mask
Source code in ir_axioms/precondition/simple.py 
18
19
20
21
22
23
24
def precondition(
    self,
    input: Input,
    output1: Document,
    output2: Document,
) -> Mask:
    return True

preconditions 

preconditions(
    input: Input, outputs: Sequence[Document]
) -> MaskMatrix
Source code in ir_axioms/precondition/simple.py 
26
27
28
29
30
31
def preconditions(
    self,
    input: Input,
    outputs: Sequence[Document],
) -> MaskMatrix:
    return full((len(outputs), len(outputs)), True, dtype=bool_)

Caching and Parallelization

ir_axioms.axiom.cache

CachedAxiom module-attribute 

CachedAxiom: TypeAlias = DbmCachedAxiom

SupportsRepr

Bases: Protocol

DbmCachedAxiom dataclass 

DbmCachedAxiom(
    *, axiom: Axiom[_Input, _Output], cache_path: Path
)

Bases: Axiom[_Input, _Output]

axiom instance-attribute 

axiom: Axiom[_Input, _Output]

cache_path instance-attribute 

cache_path: Path

preference 

preference(
    input: _Input, output1: _Output, output2: _Output
) -> Preference
Source code in ir_axioms/axiom/cache.py 
77
78
79
80
81
82
83
84
85
86
87
def preference(
    self,
    input: _Input,
    output1: _Output,
    output2: _Output,
) -> Preference:
    return next(
        self._iter_preferences(
            inputs_outputs=((input, output1, output2),),
        )
    )

preferences 

preferences(
    input: _Input, outputs: Sequence[_Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/cache.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
def preferences(
    self,
    input: _Input,
    outputs: Sequence[_Output],
) -> PreferenceMatrix:
    # First, only load the cached dependencies.
    preferences = self._cached_preferences(input, outputs)

    if not (isnan(preferences).any()):
        return preferences

    preferences = self.axiom.preferences(input, outputs)
    with dbm_open(self.cache_path, flag="c") as cache:
        for i1, output1 in enumerate(outputs):
            for i2, output2 in enumerate(outputs):
                key = self._key(input, output1, output2)
                preference = float(preferences[i1, i2])
                if isnan(preference):
                    raise RuntimeError("Missing preferences.")
                preference_bytes = pack("f", preference)
                try:
                    cache[key] = preference_bytes
                except Exception as e:
                    warn(
                        f"Error while writing to axiom cache at: {self.cache_path}"
                    )
                    raise e
    return preferences

cached 

cached(cache_path: Path) -> Axiom[_Input, _Output]
Source code in ir_axioms/axiom/cache.py 
137
138
139
140
141
def cached(self, cache_path: Path) -> Axiom[_Input, _Output]:
    if self.cache_path == cache_path:
        return self
    else:
        return self.axiom.cached(cache_path)

ir_axioms.axiom.parallel

ParallelAxiom dataclass 

ParallelAxiom(
    *,
    axiom: Axiom[Input, Output],
    n_jobs: int | None = None,
)

Bases: Axiom[Input, Output], Generic[Input, Output]

axiom instance-attribute 

axiom: Axiom[Input, Output]

n_jobs class-attribute instance-attribute 

n_jobs: int | None = None

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/parallel.py 
22
23
24
25
26
27
28
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    return self.axiom.preference(input, output1, output2)

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/parallel.py 
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    @delayed
    def _preference(
        output1: Output,
        output2: Output,
    ) -> float:
        return self.preference(
            input=input,
            output1=output1,
            output2=output2,
        )

    with Parallel(n_jobs=self.n_jobs) as parallel:
        preferences: Sequence[float] = parallel(
            _preference(output1, output2)
            for output1 in outputs
            for output2 in outputs
        )

    return array(preferences).reshape(len(outputs), len(outputs))

Learned Axioms

ir_axioms.axiom.estimator

EstimatorAxiom

Bases: Axiom[Input, Output], ABC

fit abstractmethod 

fit(
    target: Axiom,
    inputs_outputs: Iterable[
        tuple[Input, Sequence[Output]]
    ],
) -> None
Source code in ir_axioms/axiom/estimator.py 
15
16
17
18
19
20
21
@abstractmethod
def fit(
    self,
    target: Axiom,
    inputs_outputs: Iterable[tuple[Input, Sequence[Output]]],
) -> None:
    pass

ScikitLearnEstimator

Bases: Protocol

fit 

fit(X: ndarray, y: ndarray) -> Self
Source code in ir_axioms/axiom/estimator.py 
25
def fit(self, X: ndarray, y: ndarray) -> Self: ...

predict 

predict(X: ndarray) -> ndarray
Source code in ir_axioms/axiom/estimator.py 
26
def predict(self, X: ndarray) -> ndarray: ...

ScikitLearnEstimatorAxiom dataclass 

ScikitLearnEstimatorAxiom(
    axioms: Sequence[Axiom[Input, Output]],
    estimator: ScikitLearnEstimator,
)

Bases: EstimatorAxiom[Input, Output], ABC

axioms instance-attribute 

axioms: Sequence[Axiom[Input, Output]]

estimator instance-attribute 

estimator: ScikitLearnEstimator

fit 

fit(
    target: Axiom[Input, Output],
    inputs_outputs: Iterable[
        tuple[Input, Sequence[Output]]
    ],
) -> None
Source code in ir_axioms/axiom/estimator.py 
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
def fit(
    self,
    target: Axiom[Input, Output],
    inputs_outputs: Iterable[tuple[Input, Sequence[Output]]],
) -> None:
    # Compute preferences of every axiom for the inputs and outputs.
    # The individual preference matices for each common input will be flattened, so that, after stacking, the final array contains columns for each axiom and rows for each combination of input and two outputs.
    # Shape: |input+output1+output2 combinations| x |axioms|
    preferences_x = stack(
        [
            concatenate(
                [
                    axiom.preferences(
                        input=input,
                        outputs=outputs,
                    ).reshape(-1)
                    for input, outputs in tqdm(
                        inputs_outputs,
                        desc="Feature preferences",
                        unit="query",
                        disable=not self.verbose,
                    )
                ]
            )
            for axiom in self.axioms
        ],
        axis=-1,
    )

    # If the estimator is a classifier, the targets need to be discretized.
    # By normalizing the target preferences, we ensure that the target values are one of -1, 0, and 1, i.e., the classes for the classifier.
    if is_classifier(self.estimator):
        target = target.normalized()

    # Compute the target preferences for the inputs and outputs.
    # The individual preference matrices for each common input will be flattened, so that, after concatenating, the final array contains the target preferences for each combination of input and two outputs.
    # Shape: |input+output1+output2 combinations|
    preferences_y = concatenate(
        [
            target.preferences(input, outputs).reshape(-1)
            for input, outputs in tqdm(
                inputs_outputs,
                desc="Target preferences",
                unit="query",
                disable=not self.verbose,
            )
        ]
    )

    # With the flattened inputs and flattened targets, now fit the estimator.
    self.estimator.fit(preferences_x, preferences_y)

preference 

preference(
    input: Input, output1: Output, output2: Output
) -> Preference
Source code in ir_axioms/axiom/estimator.py 
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
def preference(
    self,
    input: Input,
    output1: Output,
    output2: Output,
) -> Preference:
    # Compute the preferences of each axiom.
    # Shape: |axioms|
    preferences_x = array(
        [
            axiom.preference(
                input=input,
                output1=output1,
                output2=output2,
            )
            for axiom in self.axioms
        ]
    )
    # Reshape to ensure it is a 2D array with one row.
    # Shape: 1 x |axioms|
    preferences_x = expand_dims(preferences_x, axis=0)

    # Predict the preference using the estimator.
    # Shape: 1
    estimated = self.estimator.predict(preferences_x)

    # Return the estimated preference as a scalar.
    return estimated[0]

preferences 

preferences(
    input: Input, outputs: Sequence[Output]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/estimator.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def preferences(
    self,
    input: Input,
    outputs: Sequence[Output],
) -> PreferenceMatrix:
    # Compute the preferences of each axiom for the inputs and outputs.
    # The individual preference matrices for each common input will be flattened, so that, after stacking, the final array contains columns for each axiom and rows for each combination of input and two outputs.
    # Shape: (|outputs|*|outputs|) x |axioms|
    preferences_x = stack(
        [
            axiom.preferences(
                input=input,
                outputs=outputs,
            ).reshape(-1)
            for axiom in self.axioms
        ],
        axis=-1,
    )

    # Predict the preferences using the estimator.
    # This will aggregate along the last axis, i.e., the axioms.
    # Shape: (|outputs|*|outputs|)
    estimated = self.estimator.predict(preferences_x)

    # Reshape the estimated preferences to ensure it is a 2D preference matrix.
    # Shape: |outputs| x |outputs|
    estimated = estimated.reshape((len(outputs), -1))

    # Return the estimated preferences as a preference matrix.
    return estimated

Simple Axioms

See the utility axioms.

ir_axioms.axiom.simple

NOP module-attribute 

NOP: Final = lazy_inject(NopAxiom)

RANDOM module-attribute 

RANDOM: Final = lazy_inject(RandomAxiom)

GT module-attribute 

GT: Final = lazy_inject(GreaterThanAxiom)

LT module-attribute 

LT: Final = lazy_inject(LessThanAxiom)

DIST module-attribute 

DIST: Final = lazy_inject(DistanceAxiom)

NopAxiom dataclass 

NopAxiom()

Bases: Axiom[Any, Any]

preference 

preference(
    input: Any, output1: Any, output2: Any
) -> Preference
Source code in ir_axioms/axiom/simple.py 
16
17
18
19
20
21
22
def preference(
    self,
    input: Any,
    output1: Any,
    output2: Any,
) -> Preference:
    return 0

preferences 

preferences(
    input: Any, outputs: Sequence[Any]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/simple.py 
24
25
26
27
28
29
def preferences(
    self,
    input: Any,
    outputs: Sequence[Any],
) -> PreferenceMatrix:
    return zeros((len(outputs), len(outputs)))

RandomAxiom dataclass 

RandomAxiom(*, seed: Any | None = None)

Bases: Axiom[Any, Any]

seed class-attribute instance-attribute 

seed: Any | None = None

preference 

preference(
    input: Any, output1: Any, output2: Any
) -> Preference
Source code in ir_axioms/axiom/simple.py 
43
44
45
46
47
48
49
def preference(
    self,
    input: Any,
    output1: Any,
    output2: Any,
) -> Preference:
    return self._generator.integers(-1, 1)

preferences 

preferences(
    input: Any, outputs: Sequence[Any]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/simple.py 
51
52
53
54
55
56
57
58
def preferences(
    self,
    input: Any,
    outputs: Sequence[Any],
) -> PreferenceMatrix:
    return self._generator.integers(-1, 1, (len(outputs), len(outputs))).astype(
        floating
    )

GreaterThanAxiom dataclass 

GreaterThanAxiom()

Bases: Axiom[Any, _SupportsComparisonT]

preference 

preference(
    input: Any,
    output1: _SupportsComparisonT,
    output2: _SupportsComparisonT,
) -> Preference
Source code in ir_axioms/axiom/simple.py 
77
78
79
80
81
82
83
84
85
86
87
def preference(
    self,
    input: Any,
    output1: _SupportsComparisonT,
    output2: _SupportsComparisonT,
) -> Preference:
    if output1 > output2:
        return 1
    elif output1 < output2:
        return -1
    return 0

LessThanAxiom dataclass 

LessThanAxiom()

Bases: Axiom[Any, _SupportsComparisonT]

preference 

preference(
    input: Any,
    output1: _SupportsComparisonT,
    output2: _SupportsComparisonT,
) -> Preference
Source code in ir_axioms/axiom/simple.py 
 95
 96
 97
 98
 99
100
101
102
103
104
105
def preference(
    self,
    input: Any,
    output1: _SupportsComparisonT,
    output2: _SupportsComparisonT,
) -> Preference:
    if output1 < output2:
        return 1
    elif output1 > output2:
        return -1
    return 0

DistanceAxiom dataclass 

DistanceAxiom()

Bases: Axiom[Any, _NumberT]

preference 

preference(
    input: Any, output1: _NumberT, output2: _NumberT
) -> Preference
Source code in ir_axioms/axiom/simple.py 
116
117
118
119
120
121
122
123
def preference(
    self,
    input: Any,
    output1: _NumberT,
    output2: _NumberT,
) -> Preference:
    # Signed distance, so that flipping the outputs flips the preference.
    return output1 - output2

preferences 

preferences(
    input: Any, outputs: Sequence[_NumberT]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/simple.py 
125
126
127
128
129
130
131
def preferences(
    self,
    input: Any,
    outputs: Sequence[_NumberT],
) -> PreferenceMatrix:
    values = array(outputs, dtype=float_)
    return subtract.outer(values, values)

Retrieval Axioms

See the retrieval axioms in the catalog.

ir_axioms.axiom.retrieval.term_frequency

TFC1 module-attribute 

TFC1: Final = lazy_inject(Tfc1Axiom)

TFC3 module-attribute 

TFC3: Final = lazy_inject(Tfc3Axiom)

M_TDC module-attribute 

M_TDC: Final = lazy_inject(ModifiedTdcAxiom)

LEN_M_TDC module-attribute 

LEN_M_TDC: Final = lazy_inject(LenModifiedTdcAxiom)

Tfc1Axiom dataclass 

Tfc1Axiom(
    *,
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    text_statistics: TextStatistics[Document],
    margin_fraction: NoInject[float] = 0.1,
)

Bases: PreconditionMixin[Query, Document], Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/term_frequency.py 
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(input),
    )

    term_frequency_sum1 = sum(
        self.text_statistics.term_frequency(output1, term) for term in query_terms
    )
    term_frequency_sum2 = sum(
        self.text_statistics.term_frequency(output2, term) for term in query_terms
    )

    return self._preference(term_frequency_sum1, term_frequency_sum2)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/term_frequency.py 
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(input),
    )

    term_frequency_sums = [
        sum(
            self.text_statistics.term_frequency(output, term)
            for term in query_terms
        )
        for output in tqdm(
            outputs,
            desc="Sum term frequencies",
            unit="document",
            disable=not self.verbose,
        )
    ]

    return array(
        [
            self._preference(term_frequency_sum1, term_frequency_sum2)
            for term_frequency_sum1 in term_frequency_sums
            for term_frequency_sum2 in term_frequency_sums
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

Tfc3Axiom dataclass 

Tfc3Axiom(
    *,
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    index_statistics: IndexStatistics,
    text_statistics: TextStatistics[Document],
    margin_fraction: NoInject[float] = 0.1,
)

Bases: PreconditionMixin[Query, Document], Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

index_statistics instance-attribute 

index_statistics: IndexStatistics

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/term_frequency.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )

    sum_document1 = 0
    sum_document2 = 0
    for query_term1, query_term2 in combinations(query_unique_terms, 2):
        term_discrimination1 = self.index_statistics.inverse_document_frequency(
            query_term1
        )
        term_discrimination2 = self.index_statistics.inverse_document_frequency(
            query_term2
        )

        if isclose(
            term_discrimination1,
            term_discrimination2,
            rel_tol=self.margin_fraction,
        ):
            d1q1 = self.text_statistics.term_frequency(output1, query_term1)
            d2q1 = self.text_statistics.term_frequency(output2, query_term1)
            d1q2 = self.text_statistics.term_frequency(output1, query_term2)
            d2q2 = self.text_statistics.term_frequency(output2, query_term2)

            if d1q1 != 0 and d1q2 != 0:
                if isclose(d2q1, d1q1 + d1q2) and d2q2 == 0:
                    sum_document1 += 1
                if isclose(d2q2, d1q2 + d1q1) and d2q1 == 0:
                    sum_document1 += 1
            if d2q1 != 0 and d2q2 != 0:
                if isclose(d1q1, d2q1 + d2q2) and d1q2 == 0:
                    sum_document2 += 1
                if isclose(d1q2, d2q2 + d2q1) and d1q1 == 0:
                    sum_document2 += 1

    return strictly_greater(sum_document1, sum_document2)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/term_frequency.py 
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    query_term_pairs = [
        (query_term1, query_term2)
        for query_term1, query_term2 in tqdm(
            combinations(query_unique_terms, 2),
            desc="Query term pairs",
            unit="pair",
            disable=not self.verbose,
        )
        if isclose(
            self.index_statistics.inverse_document_frequency(query_term1),
            self.index_statistics.inverse_document_frequency(query_term2),
            rel_tol=self.margin_fraction,
        )
    ]
    considered_query_terms = set(chain.from_iterable(query_term_pairs))
    term_frequencies = [
        {
            query_term: self.text_statistics.term_frequency(output, query_term)
            for query_term in considered_query_terms
        }
        for output in tqdm(
            outputs,
            desc="Term frequencies",
            unit="document",
            disable=not self.verbose,
        )
    ]

    return array(
        list(
            tqdm(
                (
                    strictly_greater(
                        sum(
                            1
                            for query_term1, query_term2 in query_term_pairs
                            if term_frequencies[i1][query_term1] != 0
                            and term_frequencies[i1][query_term2] != 0
                            and (
                                (
                                    isclose(
                                        term_frequencies[i2][query_term1],
                                        term_frequencies[i1][query_term1]
                                        + term_frequencies[i1][query_term2],
                                    )
                                    and term_frequencies[i2][query_term2] == 0
                                )
                                or (
                                    isclose(
                                        term_frequencies[i2][query_term2],
                                        term_frequencies[i1][query_term2]
                                        + term_frequencies[i1][query_term1],
                                    )
                                    and term_frequencies[i2][query_term1] == 0
                                )
                            )
                        ),
                        sum(
                            1
                            for query_term1, query_term2 in query_term_pairs
                            if term_frequencies[i2][query_term1] != 0
                            and term_frequencies[i2][query_term2] != 0
                            and (
                                (
                                    isclose(
                                        term_frequencies[i1][query_term1],
                                        term_frequencies[i2][query_term1]
                                        + term_frequencies[i2][query_term2],
                                    )
                                    and term_frequencies[i1][query_term2] == 0
                                )
                                or (
                                    isclose(
                                        term_frequencies[i1][query_term2],
                                        term_frequencies[i2][query_term2]
                                        + term_frequencies[i2][query_term1],
                                    )
                                    and term_frequencies[i1][query_term1] == 0
                                )
                            )
                        ),
                    )
                    for i1 in range(len(outputs))
                    for i2 in range(len(outputs))
                ),
                desc="Compare",
                unit="pair",
                disable=not self.verbose,
            )
        ),
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ModifiedTdcAxiom dataclass 

ModifiedTdcAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    index_statistics: IndexStatistics,
    text_statistics: TextStatistics[Query | Document],
)

Bases: Axiom[Query, Document]

Modified TDC as in:

Shi, S., Wen, J.R., Yu, Q., Song, R., Ma, W.Y.: Gravitation-based model for information retrieval. In: SIGIR ’05.

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

index_statistics instance-attribute 

index_statistics: IndexStatistics

text_statistics instance-attribute 

text_statistics: TextStatistics[Query | Document]

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/term_frequency.py 
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    term_frequencies1 = self.text_statistics.term_frequencies(output1)
    term_frequencies2 = self.text_statistics.term_frequencies(output2)

    if not _single_different_term_frequency(
        query_unique_terms=query_unique_terms,
        term_frequencies1=term_frequencies1,
        term_frequencies2=term_frequencies2,
    ):
        return 0

    return _modified_tdc_preference(
        query_term_pairs=self._query_term_pairs(query_unique_terms),
        query_term_frequencies=self.text_statistics.term_frequencies(input),
        term_frequencies1=term_frequencies1,
        term_frequencies2=term_frequencies2,
    )

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/term_frequency.py 
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    query_term_pairs = self._query_term_pairs(query_unique_terms)
    query_term_frequencies = self.text_statistics.term_frequencies(input)
    term_frequencies = [
        self.text_statistics.term_frequencies(output)
        for output in tqdm(
            outputs,
            desc="Term frequencies",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            (
                _modified_tdc_preference(
                    query_term_pairs=query_term_pairs,
                    query_term_frequencies=query_term_frequencies,
                    term_frequencies1=term_frequencies1,
                    term_frequencies2=term_frequencies2,
                )
                if _single_different_term_frequency(
                    query_unique_terms=query_unique_terms,
                    term_frequencies1=term_frequencies1,
                    term_frequencies2=term_frequencies2,
                )
                else 0
            )
            for term_frequencies1 in term_frequencies
            for term_frequencies2 in term_frequencies
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

LenModifiedTdcAxiom dataclass 

LenModifiedTdcAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    index_statistics: IndexStatistics,
    text_statistics: TextStatistics[Query | Document],
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
)

Bases: PreconditionMixin[Query, Document], ModifiedTdcAxiom

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

ir_axioms.axiom.retrieval.length_norm

LNC1 module-attribute 

LNC1: Final = lazy_inject(Lnc1Axiom)

TF_LNC module-attribute 

TF_LNC: Final = lazy_inject(TfLncAxiom)

Lnc1Axiom dataclass 

Lnc1Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    text_statistics: TextStatistics[Document],
    margin_fraction: NoInject[float] = 0.1,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/length_norm.py 
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    term_frequencies1 = self.text_statistics.term_frequencies(output1)
    term_frequencies2 = self.text_statistics.term_frequencies(output2)
    if not _similar_query_term_frequencies(
        query_unique_terms=query_unique_terms,
        term_frequencies1=term_frequencies1,
        term_frequencies2=term_frequencies2,
        margin_fraction=self.margin_fraction,
    ):
        return 0

    document1_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(output1),
    )
    document2_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(output2),
    )

    # Prefer the shorter document.
    return strictly_fewer(document1_terms, document2_terms)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/length_norm.py 
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    term_frequencies = [
        self.text_statistics.term_frequencies(output) for output in outputs
    ]
    document_terms = list(
        self.term_tokenizer.iter_terms_unordered(
            self.text_contents.iter_contents(outputs),
        )
    )
    return array(
        [
            (
                strictly_fewer(document1_terms, document2_terms)
                if _similar_query_term_frequencies(
                    query_unique_terms=query_unique_terms,
                    term_frequencies1=term_frequencies1,
                    term_frequencies2=term_frequencies2,
                    margin_fraction=self.margin_fraction,
                )
                else 0
            )
            for term_frequencies1, document1_terms in zip(
                term_frequencies, document_terms
            )
            for term_frequencies2, document2_terms in zip(
                term_frequencies, document_terms
            )
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

TfLncAxiom dataclass 

TfLncAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    text_statistics: TextStatistics[Document],
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/length_norm.py 
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(output1),
    )
    document2_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(output2),
    )
    document1_term_frequencies = {
        query_term: self.text_statistics.term_frequency(output1, query_term)
        for query_term in query_unique_terms
    }
    document2_term_frequencies = {
        query_term: self.text_statistics.term_frequency(output2, query_term)
        for query_term in query_unique_terms
    }
    return self._preference(
        query_unique_terms=query_unique_terms,
        document1_terms=document1_terms,
        document2_terms=document2_terms,
        document1_term_frequencies=document1_term_frequencies,
        document2_term_frequencies=document2_term_frequencies,
    )

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/length_norm.py 
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_terms = [
        self.term_tokenizer.terms_unordered(
            self.text_contents.contents(output),
        )
        for output in tqdm(
            outputs,
            total=len(outputs),
            desc="Tokenize",
            unit="document",
            disable=not self.verbose,
        )
    ]
    document_term_frequencies = [
        {
            query_term: self.text_statistics.term_frequency(output, query_term)
            for query_term in query_unique_terms
        }
        for output in tqdm(
            outputs,
            total=len(outputs),
            desc="Term frequencies",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            self._preference(
                query_unique_terms=query_unique_terms,
                document1_terms=document1_terms,
                document2_terms=document2_terms,
                document1_term_frequencies=document1_term_frequencies,
                document2_term_frequencies=document2_term_frequencies,
            )
            for document1_terms, document1_term_frequencies in zip(
                document_terms, document_term_frequencies
            )
            for document2_terms, document2_term_frequencies in zip(
                document_terms, document_term_frequencies
            )
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.retrieval.lower_bound

LB1 module-attribute 

LB1: Final = lazy_inject(Lb1Axiom)

Lb1Axiom dataclass 

Lb1Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    text_statistics: TextStatistics[Document],
    margin_fraction: NoInject[float] = 0.1,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/lower_bound.py 
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    if not isclose(
        _require_score(output1),
        _require_score(output2),
        rel_tol=self.margin_fraction,
    ):
        return 0

    query_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )

    document1_tf = self.text_statistics.term_frequencies(output1)
    document2_tf = self.text_statistics.term_frequencies(output2)

    query_term_only_in_document1 = any(
        document1_tf.get(term, 0) > 0 and document2_tf.get(term, 0) <= 0
        for term in query_terms
    )
    query_term_only_in_document2 = any(
        document2_tf.get(term, 0) > 0 and document1_tf.get(term, 0) <= 0
        for term in query_terms
    )

    if query_term_only_in_document1 and not query_term_only_in_document2:
        return 1
    elif not query_term_only_in_document1 and query_term_only_in_document2:
        return -1
    else:
        return 0

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/lower_bound.py 
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    # Determine which document pairs have similar scores (documents × documents).
    similar_scores = _similar_scores(
        scores=[_require_score(output) for output in outputs],
        margin_fraction=self.margin_fraction,
    )

    # Get the unique query terms and wrap as list for later deterministic access.
    query_terms = list(
        self.term_tokenizer.unique_terms(
            self.text_contents.contents(input),
        )
    )

    # Compute term frequencies for documents (only with at least one similarly scored document).
    has_similar_scores = similar_scores.sum(axis=1) > 1
    term_frequencies = [
        self.text_statistics.term_frequencies(output) if has_similar_score else {}
        for output, has_similar_score in zip(outputs, has_similar_scores)
    ]

    # Determine which query terms occur in which document (documents × query terms).
    contains_query_term = array(
        [
            document_tf.get(term, 0) > 0
            for document_tf in term_frequencies
            for term in query_terms
        ],
        dtype=bool_,
    ).reshape((len(outputs), len(query_terms)))

    # Find query terms that occur in the first but not the second document (documents × documents).
    # Note: For boolean matrices, the matrix product (documents × query terms) @ (query terms × documents) computes any(contains[i, t] and not contains[j, t] for t in query terms) for every document pair (i, j) at once.
    query_term_only_in_document1 = contains_query_term @ ~contains_query_term.T
    query_term_only_in_document2 = query_term_only_in_document1.T

    # If only the first document has an exclusive query term then return 1, -1 if only the second has one, 0 otherwise or if the scores are not similar.
    return where(
        similar_scores,
        query_term_only_in_document1.astype(float_)
        - query_term_only_in_document2.astype(float_),
        0,
    )

ir_axioms.axiom.retrieval.query_aspects

REG module-attribute 

REG: Final = lazy_inject(RegAxiom)

ANTI_REG module-attribute 

ANTI_REG: Final = lazy_inject(AntiRegAxiom)

ASPECT_REG module-attribute 

ASPECT_REG: Final = lazy_inject(AspectRegAxiom)

AND module-attribute 

AND: Final = lazy_inject(AndAxiom)

LEN_AND module-attribute 

LEN_AND: Final = lazy_inject(LenAndAxiom)

M_AND module-attribute 

M_AND: Final = lazy_inject(ModifiedAndAxiom)

LEN_M_AND module-attribute 

LEN_M_AND: Final = lazy_inject(LenModifiedAndAxiom)

DIV module-attribute 

DIV: Final = lazy_inject(DivAxiom)

LEN_DIV module-attribute 

LEN_DIV: Final = lazy_inject(LenDivAxiom)

RegAxiom dataclass 

RegAxiom(
    *,
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    text_statistics: TextStatistics[Document],
    term_similarity: TermSimilarity,
)

Bases: PreconditionMixin[Query, Document], Axiom[Query, Document]

Reference: Zheng, W., Fang, H.: Query aspect based term weighting regularization in information retrieval. In: Gurrin, C., et al. (eds.) ECIR 2010.

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

term_similarity instance-attribute 

term_similarity: TermSimilarity

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input)
    )

    min_average_similarity_terms = (
        self.term_similarity.min_average_similarity_terms(query_unique_terms)
    )
    if len(min_average_similarity_terms) == 0:
        return 0

    term_frequencies1 = {
        term: self.text_statistics.term_frequency(output1, term)
        for term in min_average_similarity_terms
    }
    term_frequencies2 = {
        term: self.text_statistics.term_frequency(output2, term)
        for term in min_average_similarity_terms
    }

    if all(
        term_frequencies1[term] > term_frequencies2[term]
        for term in min_average_similarity_terms
    ):
        return 1
    elif all(
        term_frequencies1[term] < term_frequencies2[term]
        for term in min_average_similarity_terms
    ):
        return -1
    else:
        return 0

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )

    min_average_similarity_terms = (
        self.term_similarity.min_average_similarity_terms(query_unique_terms)
    )
    if len(min_average_similarity_terms) == 0:
        return zeros((len(outputs), len(outputs)), dtype=float_)

    term_frequencies = [
        {
            term: self.text_statistics.term_frequency(output, term)
            for term in min_average_similarity_terms
        }
        for output in outputs
    ]

    return array(
        [
            (
                1
                if all(
                    term_frequencies1[term] > term_frequencies2[term]
                    for term in min_average_similarity_terms
                )
                else (
                    -1
                    if all(
                        term_frequencies1[term] < term_frequencies2[term]
                        for term in min_average_similarity_terms
                    )
                    else 0
                )
            )
            for term_frequencies1 in term_frequencies
            for term_frequencies2 in term_frequencies
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

AntiRegAxiom dataclass 

AntiRegAxiom(
    *,
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    text_statistics: TextStatistics[Document],
    term_similarity: TermSimilarity,
)

Bases: PreconditionMixin[Query, Document], Axiom[Query, Document]

Reference: Zheng, W., Fang, H.: Query aspect based term weighting regularization in information retrieval. In: Gurrin, C., et al. (eds.) ECIR 2010.

Modified to use maximum similarity instead of minimum similarity.

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

term_similarity instance-attribute 

term_similarity: TermSimilarity

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )

    max_average_similarity_terms = (
        self.term_similarity.max_average_similarity_terms(query_unique_terms)
    )
    if len(max_average_similarity_terms) == 0:
        return 0

    term_frequencies1 = {
        term: self.text_statistics.term_frequency(output1, term)
        for term in max_average_similarity_terms
    }
    term_frequencies2 = {
        term: self.text_statistics.term_frequency(output2, term)
        for term in max_average_similarity_terms
    }

    if all(
        term_frequencies1[term] > term_frequencies2[term]
        for term in max_average_similarity_terms
    ):
        return 1
    elif all(
        term_frequencies1[term] < term_frequencies2[term]
        for term in max_average_similarity_terms
    ):
        return -1
    else:
        return 0

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )

    max_average_similarity_terms = (
        self.term_similarity.max_average_similarity_terms(query_unique_terms)
    )
    if len(max_average_similarity_terms) == 0:
        return zeros((len(outputs), len(outputs)), dtype=float_)

    term_frequencies = [
        {
            term: self.text_statistics.term_frequency(output, term)
            for term in max_average_similarity_terms
        }
        for output in outputs
    ]

    return array(
        [
            (
                1
                if all(
                    term_frequencies1[term] > term_frequencies2[term]
                    for term in max_average_similarity_terms
                )
                else (
                    -1
                    if all(
                        term_frequencies1[term] < term_frequencies2[term]
                        for term in max_average_similarity_terms
                    )
                    else 0
                )
            )
            for term_frequencies1 in term_frequencies
            for term_frequencies2 in term_frequencies
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

AspectRegAxiom dataclass 

AspectRegAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    index_statistics: IndexStatistics,
    term_similarity: TermSimilarity,
    term_discriminator_margin_fraction: NoInject[
        float
    ] = 0.1,
)

Bases: Axiom[Query, Document]

Similar to REG but follows the query aspect clustering from the paper and then counts the number of aspects covered in each document.

Reference: Zheng, W., Fang, H.: Query aspect based term weighting regularization in information retrieval. In: Gurrin, C., et al. (eds.) ECIR 2010.

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

index_statistics instance-attribute 

index_statistics: IndexStatistics

term_similarity instance-attribute 

term_similarity: TermSimilarity

term_discriminator_margin_fraction class-attribute instance-attribute 

term_discriminator_margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    query_aspects = self._query_aspects(query_unique_terms)
    if query_aspects is None:
        return 0

    document1_unique_terms, document2_unique_terms = (
        self.term_tokenizer.iter_unique_terms(
            self.text_contents.iter_contents((output1, output2)),
        )
    )
    return strictly_greater(
        _covers_any_aspect(query_aspects, document1_unique_terms),
        _covers_any_aspect(query_aspects, document2_unique_terms),
    )

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    query_aspects = self._query_aspects(query_unique_terms)
    if query_aspects is None:
        return zeros((len(outputs), len(outputs)), dtype=float_)

    covers_any_aspect = [
        _covers_any_aspect(query_aspects, document_unique_terms)
        for document_unique_terms in self.term_tokenizer.iter_unique_terms(
            self.text_contents.iter_contents(outputs),
        )
    ]
    return array(
        [
            strictly_greater(covers_any_aspect1, covers_any_aspect2)
            for covers_any_aspect1 in covers_any_aspect
            for covers_any_aspect2 in covers_any_aspect
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

AndAxiom dataclass 

AndAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output1),
    )
    document2_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output2),
    )

    all_query_terms1 = query_unique_terms <= document1_unique_terms
    all_query_terms2 = query_unique_terms <= document2_unique_terms
    return strictly_greater(all_query_terms1, all_query_terms2)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_unique_terms = (
        self.term_tokenizer.unique_terms(
            self.text_contents.contents(output),
        )
        for output in outputs
    )
    all_query_terms = [
        query_unique_terms <= terms
        for terms in tqdm(
            document_unique_terms,
            total=len(outputs),
            desc="Query term overlap",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            strictly_greater(all_query_terms1, all_query_terms2)
            for all_query_terms1 in all_query_terms
            for all_query_terms2 in all_query_terms
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

LenAndAxiom dataclass 

LenAndAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
)

Bases: PreconditionMixin[Query, Document], AndAxiom

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

ModifiedAndAxiom dataclass 

ModifiedAndAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

Modified AND: One document contains a larger subset of query terms.

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output1),
    )
    document2_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output2),
    )

    query_terms1 = query_unique_terms & document1_unique_terms
    query_terms2 = query_unique_terms & document2_unique_terms
    return strictly_greater(len(query_terms1), len(query_terms2))

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_unique_terms = (
        self.term_tokenizer.unique_terms(
            self.text_contents.contents(output),
        )
        for output in outputs
    )
    num_query_terms = [
        len(query_unique_terms & terms)
        for terms in tqdm(
            document_unique_terms,
            total=len(outputs),
            desc="Query term overlap",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            strictly_greater(num_query_terms1, num_query_terms2)
            for num_query_terms1 in num_query_terms
            for num_query_terms2 in num_query_terms
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

LenModifiedAndAxiom dataclass 

LenModifiedAndAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
)

Bases: PreconditionMixin[Query, Document], ModifiedAndAxiom

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

DivAxiom dataclass 

DivAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output1),
    )
    document2_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output2),
    )

    overlap1 = _vocabulary_overlap(
        vocabulary1=query_unique_terms,
        vocabulary2=document1_unique_terms,
    )
    overlap2 = _vocabulary_overlap(
        vocabulary1=query_unique_terms,
        vocabulary2=document2_unique_terms,
    )

    return strictly_greater(overlap2, overlap1)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/query_aspects.py 
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_unique_terms = (
        self.term_tokenizer.unique_terms(
            self.text_contents.contents(output),
        )
        for output in outputs
    )
    overlaps = [
        _vocabulary_overlap(
            vocabulary1=query_unique_terms,
            vocabulary2=terms,
        )
        for terms in tqdm(
            document_unique_terms,
            total=len(outputs),
            desc="Vocabulary overlap",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            strictly_greater(overlap2, overlap1)
            for overlap1 in overlaps
            for overlap2 in overlaps
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

LenDivAxiom dataclass 

LenDivAxiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    precondition: NoInject[
        Precondition[Query, Document]
    ] = LEN(),
)

Bases: PreconditionMixin[Query, Document], DivAxiom

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Query, Document]] = (
    field(default_factory=LEN)
)

ir_axioms.axiom.retrieval.term_similarity

STMC1 module-attribute 

STMC1: Final = lazy_inject(Stmc1Axiom)

STMC2 module-attribute 

STMC2: Final = lazy_inject(Stmc2Axiom)

Stmc1Axiom dataclass 

Stmc1Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    term_similarity: TermSimilarity,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

term_similarity instance-attribute 

term_similarity: TermSimilarity

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/term_similarity.py 
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output1),
    )
    document2_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(output2),
    )
    average_similarity1 = self.term_similarity.average_similarity(
        document1_unique_terms, query_unique_terms
    )
    average_similarity2 = self.term_similarity.average_similarity(
        document2_unique_terms, query_unique_terms
    )
    return strictly_greater(average_similarity1, average_similarity2)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/term_similarity.py 
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_unique_terms = (
        self.term_tokenizer.unique_terms(self.text_contents.contents(output))
        for output in outputs
    )
    average_similarities = [
        self.term_similarity.average_similarity(document_terms, query_unique_terms)
        for document_terms in tqdm(
            document_unique_terms,
            total=len(outputs),
            desc="Compute avg. similarities",
            unit="document",
            disable=not self.verbose,
        )
    ]

    return array(
        [
            strictly_greater(average_similarities1, average_similarities2)
            for average_similarities1 in average_similarities
            for average_similarities2 in average_similarities
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

Stmc2Axiom dataclass 

Stmc2Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    term_similarity: TermSimilarity,
    text_statistics: TextStatistics[Document],
    margin_fraction: NoInject[float] = 0.2,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

term_similarity instance-attribute 

term_similarity: TermSimilarity

text_statistics instance-attribute 

text_statistics: TextStatistics[Document]

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.2

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference

Given the most similar pair of a query term and non-query term, prefer the first document if the second document's non-query term frequency compared to the first document's query term frequency is similar to the second document's length compared to the first document's length.

Source code in ir_axioms/axiom/retrieval/term_similarity.py 
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    """
    Given the most similar pair of a query term and non-query term,
    prefer the first document if the second document's non-query term frequency
    compared to the first document's query term frequency
    is similar to the second document's length
    compared to the first document's length.
    """

    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(output1),
    )
    document2_terms = self.term_tokenizer.terms_unordered(
        self.text_contents.contents(output2),
    )

    document1_length = len(document1_terms)
    document2_length = len(document2_terms)

    document_unique_terms = set(document1_terms) | set(document2_terms)
    non_query_terms = document_unique_terms - query_unique_terms

    _, max_similarity_pairs = self.term_similarity.max_similarity_pairs(
        query_unique_terms,
        non_query_terms,
    )
    if len(max_similarity_pairs) == 0:
        return 0

    term_frequencies1 = self.text_statistics.term_frequencies(output1)
    term_frequencies2 = self.text_statistics.term_frequencies(output2)

    return _stmc2_preference(
        max_similarity_pairs=max_similarity_pairs,
        document1_length=document1_length,
        document2_length=document2_length,
        term_frequencies1=term_frequencies1,
        term_frequencies2=term_frequencies2,
        margin_fraction=self.margin_fraction,
    )

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/term_similarity.py 
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_terms = list(
        self.term_tokenizer.iter_terms_unordered(
            self.text_contents.iter_contents(outputs),
        )
    )
    term_frequencies = [
        self.text_statistics.term_frequencies(output) for output in outputs
    ]
    # The most similar pairs over the union of two documents' non-query terms are the most similar pairs of whichever document has the higher maximum.
    max_similarity_pairs = [
        self._document_max_similarity_pairs(query_unique_terms, terms)
        for terms in tqdm(
            document_terms,
            desc="Max. similarity pairs",
            unit="document",
            disable=not self.verbose,
        )
    ]

    def pair_preference(i1: int, i2: int) -> Preference:
        max_similarity1, pairs1 = max_similarity_pairs[i1]
        max_similarity2, pairs2 = max_similarity_pairs[i2]
        max_similarity = max(max_similarity1, max_similarity2)
        pairs = {
            # Include maximum similiarity pairs from each document whose max similarity matches the global maximum similarity.
            *(pairs1 if max_similarity1 >= max_similarity else set()),
            *(pairs2 if max_similarity2 >= max_similarity else set()),
        }
        return _stmc2_preference(
            max_similarity_pairs=pairs,
            document1_length=len(document_terms[i1]),
            document2_length=len(document_terms[i2]),
            term_frequencies1=term_frequencies[i1],
            term_frequencies2=term_frequencies[i2],
            margin_fraction=self.margin_fraction,
        )

    return array(
        [
            pair_preference(i1, i2)
            for i1 in range(len(outputs))
            for i2 in range(len(outputs))
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.retrieval.proximity

PROX1 module-attribute 

PROX1: Final = lazy_inject(Prox1Axiom)

PROX2 module-attribute 

PROX2: Final = lazy_inject(Prox2Axiom)

PROX3 module-attribute 

PROX3: Final = lazy_inject(Prox3Axiom)

PROX4 module-attribute 

PROX4: Final = lazy_inject(Prox4Axiom)

PROX5 module-attribute 

PROX5: Final = lazy_inject(Prox5Axiom)

Prox1Axiom dataclass 

Prox1Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/proximity.py 
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output1),
    )
    document1_unique_terms = set(document1_terms)
    document2_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output2),
    )
    document2_unique_terms = set(document2_terms)

    if not _same_query_term_subset(
        query_terms=query_unique_terms,
        document1_terms=document1_unique_terms,
        document2_terms=document2_unique_terms,
    ):
        return 0

    overlapping_terms = (
        query_unique_terms & document1_unique_terms & document2_unique_terms
    )

    average1 = _average_between_query_terms(overlapping_terms, document1_terms)
    average2 = _average_between_query_terms(overlapping_terms, document2_terms)

    return strictly_greater(average2, average1)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/proximity.py 
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_terms, document_unique_terms = _tokenize_documents(
        self.text_contents, self.term_tokenizer, outputs, self.verbose
    )
    # If both documents contain the same query term subset,
    # the overlapping terms are just the query terms in each document.
    averages = [
        _average_between_query_terms(query_unique_terms & unique_terms, terms)
        for terms, unique_terms in zip(document_terms, document_unique_terms)
    ]
    return array(
        [
            (
                strictly_greater(averages[i2], averages[i1])
                if _same_query_term_subset(
                    query_terms=query_unique_terms,
                    document1_terms=document_unique_terms[i1],
                    document2_terms=document_unique_terms[i2],
                )
                else 0
            )
            for i1 in range(len(outputs))
            for i2 in range(len(outputs))
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

Prox2Axiom dataclass 

Prox2Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/proximity.py 
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output1),
    )
    document1_unique_terms = set(document1_terms)
    document2_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output2),
    )
    document2_unique_terms = set(document2_terms)

    if not _same_query_term_subset(
        query_terms=query_unique_terms,
        document1_terms=document1_unique_terms,
        document2_terms=document2_unique_terms,
    ):
        return 0

    common_terms = (
        query_unique_terms & document1_unique_terms & document2_unique_terms
    )

    first_position_sum1 = sum(document1_terms.index(term) for term in common_terms)
    first_position_sum2 = sum(document2_terms.index(term) for term in common_terms)
    return strictly_greater(first_position_sum2, first_position_sum1)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/proximity.py 
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_terms = [
        self.term_tokenizer.terms(self.text_contents.contents(output))
        for output in tqdm(
            outputs,
            desc="Tokenize documents",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            (
                # Prefer the document where query terms appear earlier.
                strictly_greater(
                    sum(
                        document_terms2.index(term)
                        for term in query_unique_terms
                        & set(document_terms1)
                        & set(document_terms2)
                    ),
                    sum(
                        document_terms1.index(term)
                        for term in query_unique_terms
                        & set(document_terms1)
                        & set(document_terms2)
                    ),
                )
                if _same_query_term_subset(
                    query_terms=query_unique_terms,
                    document1_terms=set(document_terms1),
                    document2_terms=set(document_terms2),
                )
                else 0
            )
            for document_terms1 in document_terms
            for document_terms2 in document_terms
        ],
        dtype=float_,
    ).reshape(len(outputs), len(outputs))

Prox3Axiom dataclass 

Prox3Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/proximity.py 
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_terms = self.term_tokenizer.terms(
        self.text_contents.contents(input),
    )
    query_unique_terms = set(query_terms)
    document1_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output1),
    )
    document1_unique_terms = set(document1_terms)
    document2_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output2),
    )
    document2_unique_terms = set(document2_terms)

    if not _same_query_term_subset(
        query_terms=query_unique_terms,
        document1_terms=document1_unique_terms,
        document2_terms=document2_unique_terms,
    ):
        return 0

    return strictly_less(
        _find_index(
            query_terms=query_terms,
            document_terms=document1_terms,
        ),
        _find_index(
            query_terms=query_terms,
            document_terms=document2_terms,
        ),
    )

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/proximity.py 
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_terms = self.term_tokenizer.terms(
        self.text_contents.contents(input),
    )
    query_unique_terms = set(query_terms)
    document_terms, document_unique_terms = _tokenize_documents(
        self.text_contents, self.term_tokenizer, outputs, self.verbose
    )
    indices = [
        _find_index(query_terms=query_terms, document_terms=terms)
        for terms in document_terms
    ]
    return array(
        [
            (
                strictly_less(indices[i1], indices[i2])
                if _same_query_term_subset(
                    query_terms=query_unique_terms,
                    document1_terms=document_unique_terms[i1],
                    document2_terms=document_unique_terms[i2],
                )
                else 0
            )
            for i1 in range(len(outputs))
            for i2 in range(len(outputs))
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

Prox4Axiom dataclass 

Prox4Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/proximity.py 
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output1),
    )
    document1_unique_terms = set(document1_terms)
    document2_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output2),
    )
    document2_unique_terms = set(document2_terms)

    if not _all_query_terms_in_documents(
        query_terms=query_unique_terms,
        document1_terms=document1_unique_terms,
        document2_terms=document2_unique_terms,
    ):
        return 0

    occurrences1, count1 = _closest_grouping_size_and_count(
        query_terms=query_unique_terms,
        document_terms=document1_terms,
    )
    occurrences2, count2 = _closest_grouping_size_and_count(
        query_terms=query_unique_terms,
        document_terms=document2_terms,
    )

    if occurrences1 != occurrences2:
        return strictly_less(occurrences1, occurrences2)
    else:
        return strictly_greater(count1, count2)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/proximity.py 
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_terms, document_unique_terms = _tokenize_documents(
        self.text_contents, self.term_tokenizer, outputs, self.verbose
    )
    # Only needed for documents containing all query terms.
    groupings = [
        (
            _closest_grouping_size_and_count(
                query_terms=query_unique_terms,
                document_terms=terms,
            )
            if query_unique_terms <= unique_terms
            else (0, 0)
        )
        for terms, unique_terms in zip(document_terms, document_unique_terms)
    ]
    return array(
        [
            (
                (
                    strictly_less(occurrences1, occurrences2)
                    if occurrences1 != occurrences2
                    else strictly_greater(count1, count2)
                )
                if _all_query_terms_in_documents(
                    query_terms=query_unique_terms,
                    document1_terms=unique_terms1,
                    document2_terms=unique_terms2,
                )
                else 0
            )
            for unique_terms1, (occurrences1, count1) in zip(
                document_unique_terms, groupings
            )
            for unique_terms2, (occurrences2, count2) in zip(
                document_unique_terms, groupings
            )
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

Prox5Axiom dataclass 

Prox5Axiom(
    *,
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: Axiom[Query, Document]

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/proximity.py 
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> Preference:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document1_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output1),
    )
    document1_unique_terms = set(document1_terms)
    document2_terms = self.term_tokenizer.terms(
        self.text_contents.contents(output2),
    )
    document2_unique_terms = set(document2_terms)

    if not _all_query_terms_in_documents(
        query_terms=query_unique_terms,
        document1_terms=document1_unique_terms,
        document2_terms=document2_unique_terms,
    ):
        return 0

    smallest_span1 = _average_smallest_span(
        query_terms=query_unique_terms,
        document_terms=document1_terms,
    )
    smallest_span2 = _average_smallest_span(
        query_terms=query_unique_terms,
        document_terms=document2_terms,
    )

    return strictly_less(smallest_span1, smallest_span2)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/proximity.py 
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    query_unique_terms = self.term_tokenizer.unique_terms(
        self.text_contents.contents(input),
    )
    document_terms, document_unique_terms = _tokenize_documents(
        self.text_contents, self.term_tokenizer, outputs, self.verbose
    )
    # Only needed for documents containing all query terms.
    smallest_spans = [
        (
            _average_smallest_span(
                query_terms=query_unique_terms,
                document_terms=terms,
            )
            if query_unique_terms <= unique_terms
            else inf
        )
        for terms, unique_terms in zip(document_terms, document_unique_terms)
    ]
    return array(
        [
            (
                strictly_less(smallest_span1, smallest_span2)
                if _all_query_terms_in_documents(
                    query_terms=query_unique_terms,
                    document1_terms=unique_terms1,
                    document2_terms=unique_terms2,
                )
                else 0
            )
            for unique_terms1, smallest_span1 in zip(
                document_unique_terms, smallest_spans
            )
            for unique_terms2, smallest_span2 in zip(
                document_unique_terms, smallest_spans
            )
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.retrieval.argumentative

ArgUC module-attribute 

ArgUC: Final = lazy_inject(ArgumentativeUnitsCountAxiom)

QTArg module-attribute 

QTArg: Final = lazy_inject(
    QueryTermOccurrenceInArgumentativeUnitsAxiom
)

QTPArg module-attribute 

QTPArg: Final = lazy_inject(
    QueryTermPositionInArgumentativeUnitsAxiom
)

aSLDoc module-attribute 

aSLDoc: Final = lazy_inject(AverageSentenceLengthAxiom)

aSL module-attribute 

aSL: Final = aSLDoc

ArgumentativeUnitsCountAxiom dataclass 

ArgumentativeUnitsCountAxiom(
    *,
    models: NoInject[Iterable[str]] = DEFAULT_TARGER_MODELS,
    api_url: NoInject[str] = DEFAULT_TARGER_API_URL,
    cache_dir: NoInject[Path | None] = None,
    precondition: NoInject[
        Precondition[Any, Document]
    ] = LEN(),
    text_contents: TextContents[Document],
)

Bases: PreconditionMixin[Any, Document], _TargerMixin, Axiom[Any, Document]

Favor documents with more argumentative units.

text_contents instance-attribute 

text_contents: TextContents[Document]

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Any, Document]] = field(
    default_factory=LEN
)

preference 

preference(
    input: Any, output1: Document, output2: Document
)
Source code in ir_axioms/axiom/retrieval/argumentative.py 
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
def preference(
    self,
    input: Any,
    output1: Document,
    output2: Document,
):
    arguments1 = self.analyze_text(self.text_contents, output1)
    arguments2 = self.analyze_text(self.text_contents, output2)

    count1 = sum(
        _count_argumentative_units(sentences) for _, sentences in arguments1.items()
    )
    count2 = sum(
        _count_argumentative_units(sentences) for _, sentences in arguments2.items()
    )

    return strictly_greater(count1, count2)

QueryTermOccurrenceInArgumentativeUnitsAxiom dataclass 

QueryTermOccurrenceInArgumentativeUnitsAxiom(
    *,
    models: NoInject[Iterable[str]] = DEFAULT_TARGER_MODELS,
    api_url: NoInject[str] = DEFAULT_TARGER_API_URL,
    cache_dir: NoInject[Path | None] = None,
    precondition: NoInject[
        Precondition[Any, Document]
    ] = LEN(),
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
)

Bases: PreconditionMixin[Query, Document], _TargerMixin, Axiom[Query, Document]

Favor documents with more query terms in argumentative units.

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Any, Document]] = field(
    default_factory=LEN
)

preference 

preference(
    input: Query, output1: Document, output2: Document
)
Source code in ir_axioms/axiom/retrieval/argumentative.py 
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
):
    arguments1 = self.analyze_text(
        text_contents=self.text_contents,
        document=output1,
    )
    arguments2 = self.analyze_text(
        text_contents=self.text_contents,
        document=output2,
    )

    count1 = sum(
        _count_query_terms(
            text_contents=self.text_contents,
            term_tokenizer=self.term_tokenizer,
            sentences=sentences,
            query=input,
        )
        for _, sentences in arguments1.items()
    )
    count2 = sum(
        _count_query_terms(
            text_contents=self.text_contents,
            term_tokenizer=self.term_tokenizer,
            sentences=sentences,
            query=input,
        )
        for _, sentences in arguments2.items()
    )

    return strictly_greater(count1, count2)

QueryTermPositionInArgumentativeUnitsAxiom dataclass 

QueryTermPositionInArgumentativeUnitsAxiom(
    *,
    models: NoInject[Iterable[str]] = DEFAULT_TARGER_MODELS,
    api_url: NoInject[str] = DEFAULT_TARGER_API_URL,
    cache_dir: NoInject[Path | None] = None,
    precondition: NoInject[
        Precondition[Any, Document]
    ] = LEN(),
    text_contents: TextContents[Query | Document],
    term_tokenizer: TermTokenizer,
    penalty: NoInject[int | None] = 100000,
)

Bases: PreconditionMixin[Query, Document], _TargerMixin, Axiom[Query, Document]

Favor documents where the first occurrence of a query term in an argumentative unit is closer to the beginning of the document.

This axiom is based on the general observation that query terms occur “earlier” in relevant documents.

References: Troy, A.D., Zhang, G.: Enhancing Relevance Scoring with Chronological Term Rank. In: Proceedings of SIGIR 2007. pp. 599–606. ACM. Mitra, B., Diaz, F., Craswell, N.: Learning to Match Using Local and Distributed Representations of Text for Web Search. In: Proceedings of WWW 2017. pp. 1291–1299. ACM.

text_contents instance-attribute 

text_contents: TextContents[Query | Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

penalty class-attribute instance-attribute 

penalty: NoInject[int | None] = 100000

Penalty for the average query term position, if a query term is not found in any argumentative unit for a document. Set to None to use the maximum length of the two compared documents.

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Any, Document]] = field(
    default_factory=LEN
)

preference 

preference(
    input: Query, output1: Document, output2: Document
)
Source code in ir_axioms/axiom/retrieval/argumentative.py 
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
):
    arguments1 = self.analyze_text(
        text_contents=self.text_contents,
        document=output1,
    )
    arguments2 = self.analyze_text(
        text_contents=self.text_contents,
        document=output1,
    )

    if len(arguments1) == 0 or len(arguments2) == 0:
        return 0

    penalty = self.penalty
    if penalty is None:
        penalty = (
            max(
                len(
                    self.term_tokenizer.terms_unordered(
                        self.text_contents.contents(output1),
                    ),
                ),
                len(
                    self.term_tokenizer.terms_unordered(
                        self.text_contents.contents(output1),
                    ),
                ),
            )
            + 1
        )

    position1 = mean(
        _query_term_position_in_argument(
            text_contents=self.text_contents,
            term_tokenizer=self.term_tokenizer,
            sentences=sentences,
            query=input,
            penalty=penalty,
        )
        for _, sentences in arguments1.items()
    )
    position2 = mean(
        _query_term_position_in_argument(
            text_contents=self.text_contents,
            term_tokenizer=self.term_tokenizer,
            sentences=sentences,
            query=input,
            penalty=penalty,
        )
        for _, sentences in arguments2.items()
    )

    return strictly_less(position1, position2)

AverageSentenceLengthAxiom dataclass 

AverageSentenceLengthAxiom(
    *,
    precondition: NoInject[
        Precondition[Any, Document]
    ] = LEN(),
    text_contents: TextContents[Document],
    term_tokenizer: TermTokenizer,
    sentence_tokenizer: SentenceTokenizer,
    min_sentence_length: NoInject[int] = 12,
    max_sentence_length: NoInject[int] = 20,
)

Bases: PreconditionMixin[Any, Document], Axiom[Any, Document]

Favor documents with an average sentence length between a minimum (default: 12) and a maximum (default: 20) number of words.

This axiom is based on the general observation for text readability / good writing style [8, 10].

References: Markel, M.: Technical Communication. 9th ed. Bedford/St Martin’s (2010) Newell, C.: Editing Tip: Sentence Length (2014)

text_contents instance-attribute 

text_contents: TextContents[Document]

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

sentence_tokenizer instance-attribute 

sentence_tokenizer: SentenceTokenizer

min_sentence_length class-attribute instance-attribute 

min_sentence_length: NoInject[int] = 12

max_sentence_length class-attribute instance-attribute 

max_sentence_length: NoInject[int] = 20

precondition class-attribute instance-attribute 

precondition: NoInject[Precondition[Any, Document]] = field(
    default_factory=LEN
)

preference 

preference(
    input: Any, output1: Document, output2: Document
) -> Preference
Source code in ir_axioms/axiom/retrieval/argumentative.py 
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
def preference(
    self,
    input: Any,
    output1: Document,
    output2: Document,
) -> Preference:
    sentence_length1 = _average_sentence_length(
        text_contents=self.text_contents,
        term_tokenizer=self.term_tokenizer,
        sentence_tokenizer=self.sentence_tokenizer,
        document=output1,
    )
    sentence_length2 = _average_sentence_length(
        text_contents=self.text_contents,
        term_tokenizer=self.term_tokenizer,
        sentence_tokenizer=self.sentence_tokenizer,
        document=output2,
    )

    min_length = self.min_sentence_length
    max_length = self.max_sentence_length

    length_in_range1 = min_length <= sentence_length1 <= max_length
    length_in_range2 = min_length <= sentence_length2 <= max_length

    return strictly_greater(length_in_range1, length_in_range2)

preferences 

preferences(
    input: Any, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/argumentative.py 
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
def preferences(
    self,
    input: Any,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    lengths = (
        _average_sentence_length(
            text_contents=self.text_contents,
            term_tokenizer=self.term_tokenizer,
            sentence_tokenizer=self.sentence_tokenizer,
            document=document,
        )
        for document in outputs
    )
    lengths_in_range = [
        self.min_sentence_length <= length <= self.max_sentence_length
        for length in tqdm(
            lengths,
            total=len(outputs),
            desc="Sentence lengths",
            unit="document",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            strictly_greater(length_in_range1, length_in_range2)
            for length_in_range1 in lengths_in_range
            for length_in_range2 in lengths_in_range
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.retrieval.simple

ORIG module-attribute 

ORIG = lazy_inject(OriginalAxiom)

ORACLE module-attribute 

ORACLE = lazy_inject(OracleAxiom)

OriginalAxiom dataclass 

OriginalAxiom()

Bases: Axiom[Any, Document]

preference 

preference(
    input: Any, output1: Document, output2: Document
) -> float
Source code in ir_axioms/axiom/retrieval/simple.py 
12
13
14
15
16
17
18
19
20
21
22
23
def preference(
    self,
    input: Any,
    output1: Document,
    output2: Document,
) -> float:
    if output1.rank is not None and output2.rank is not None:
        return strictly_less(output1.rank, output2.rank)
    elif output1.score is not None and output2.score is not None:
        return strictly_greater(output1.score, output2.score)
    else:
        raise ValueError("Can only compare ranked and/or scored documents.")

OracleAxiom dataclass 

OracleAxiom()

Bases: Axiom[Any, Document]

preference 

preference(
    input: Any, output1: Document, output2: Document
) -> float
Source code in ir_axioms/axiom/retrieval/simple.py 
31
32
33
34
35
36
37
38
39
40
def preference(
    self,
    input: Any,
    output1: Document,
    output2: Document,
) -> float:
    if output1.relevance is not None and output2.relevance is not None:
        return strictly_greater(output1.relevance, output2.relevance)
    else:
        raise ValueError("Can only compare judged documents.")

ir_axioms.axiom.retrieval.trec

TrecOracleAxiom dataclass 

TrecOracleAxiom(*, qrels: TrecQrel)

Bases: Axiom[Query, Document]

qrels instance-attribute 

qrels: TrecQrel

preference 

preference(
    input: Query, output1: Document, output2: Document
) -> float
Source code in ir_axioms/axiom/retrieval/trec.py 
39
40
41
42
43
44
45
46
47
def preference(
    self,
    input: Query,
    output1: Document,
    output2: Document,
) -> float:
    judgement1 = self._judgement(input.id, output1.id)
    judgement2 = self._judgement(input.id, output2.id)
    return strictly_greater(judgement1, judgement2)

preferences 

preferences(
    input: Query, outputs: Sequence[Document]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/retrieval/trec.py 
49
50
51
52
53
54
55
56
57
58
59
60
61
62
def preferences(
    self,
    input: Query,
    outputs: Sequence[Document],
) -> PreferenceMatrix:
    judgements = [self._judgement(input.id, output.id) for output in outputs]
    return array(
        [
            strictly_greater(judgement1, judgement2)
            for judgement1 in judgements
            for judgement2 in judgements
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

RAG Axioms

See the RAG axioms in the catalog.

ir_axioms.axiom.generation.coherence

Coherence axioms for retrieval augmented text.

For a definition of this utility dimension, see: https://doi.org/10.1145/3626772.3657849

  • Logical coherence: Is the response well-structured?
  • Stylistic coherence: Does the response have a uniform style of speech?

COH1 module-attribute 

COH1: Final = lazy_inject(WordLengthDeviationCoherenceAxiom)

COH2 module-attribute 

COH2: Final = lazy_inject(
    SubjectVerbClosenessCoherenceAxiom
)

WordLengthDeviationCoherenceAxiom dataclass 

WordLengthDeviationCoherenceAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    sentence_tokenizer: SentenceTokenizer,
    term_tokenizer: TermTokenizer,
    margin_fraction: NoInject[float] = 0.0,
)

Bases: Axiom[Any, GenerationResponse]

Prefer text with lower standard deviation of average word lengths across sentences.

text_contents instance-attribute 

text_contents: TextContents[GenerationResponse]

sentence_tokenizer instance-attribute 

sentence_tokenizer: SentenceTokenizer

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.0

preference 

preference(
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/coherence.py 
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
def preference(
    self,
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    average_word_lengths1 = array(
        [
            array(
                [
                    len(word)
                    for word in self.term_tokenizer.terms_unordered(sentence)
                ]
            ).mean()
            for sentence in self.sentence_tokenizer.sentences(
                self.text_contents.contents(output1)
            )
        ]
    )
    average_word_lengths2 = array(
        [
            array(
                [
                    len(word)
                    for word in self.term_tokenizer.terms_unordered(sentence)
                ]
            ).mean()
            for sentence in self.sentence_tokenizer.sentences(
                self.text_contents.contents(output2)
            )
        ]
    )

    average_word_lengths_stdev1 = average_word_lengths1.std()
    average_word_lengths_stdev2 = average_word_lengths2.std()

    if isclose(
        average_word_lengths_stdev1,
        average_word_lengths_stdev2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_less(average_word_lengths_stdev1, average_word_lengths_stdev2)

preferences 

preferences(
    input: Any, outputs: Sequence[GenerationResponse]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/coherence.py 
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
def preferences(
    self,
    input: Any,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    average_word_lengths = (
        array(
            [
                array(
                    [
                        len(word)
                        for word in self.term_tokenizer.terms_unordered(sentence)
                    ]
                ).mean()
                for sentence in self.sentence_tokenizer.sentences(
                    self.text_contents.contents(output)
                )
            ]
        )
        for output in tqdm(
            outputs,
            desc="Calculate average word lengths",
            unit="output",
            disable=not self.verbose,
        )
    )

    average_word_lengths_stdevs = [
        average_word_lengths.std() for average_word_lengths in average_word_lengths
    ]

    return array(
        [
            (
                strictly_less(
                    average_word_lengths_stdev1, average_word_lengths_stdev2
                )
                if not isclose(
                    average_word_lengths_stdev1,
                    average_word_lengths_stdev2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for average_word_lengths_stdev1 in average_word_lengths_stdevs
            for average_word_lengths_stdev2 in average_word_lengths_stdevs
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

SubjectVerbClosenessCoherenceAxiom dataclass 

SubjectVerbClosenessCoherenceAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    language_name: NoInject[str] = "en_core_web_sm",
    margin_fraction: NoInject[float] = 0.1,
)

Bases: Axiom[Any, GenerationResponse]

Prefer text with subjects and verbs that are closer together.

text_contents instance-attribute 

text_contents: TextContents[GenerationResponse]

language_name class-attribute instance-attribute 

language_name: NoInject[str] = 'en_core_web_sm'

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/coherence.py 
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
def preference(
    self,
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    avg_max_sv_distance1 = _average_max_subject_verb_distance(
        self._language(self.text_contents.contents(output1))
    )
    avg_max_sv_distance2 = _average_max_subject_verb_distance(
        self._language(self.text_contents.contents(output2))
    )

    if isclose(
        avg_max_sv_distance1,
        avg_max_sv_distance2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_less(avg_max_sv_distance1, avg_max_sv_distance2)

preferences 

preferences(
    input: Any, outputs: Sequence[GenerationResponse]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/coherence.py 
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
def preferences(
    self,
    input: Any,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    docs = self._language.pipe(
        self.text_contents.contents(output) for output in outputs
    )

    avg_max_sv_distances = [
        _average_max_subject_verb_distance(doc)
        for doc in tqdm(
            docs,
            total=len(outputs),
            desc="Calculate S-V distances",
            unit="output",
            disable=not self.verbose,
        )
    ]

    return array(
        [
            (
                strictly_less(avg_max_sv_distance1, avg_max_sv_distance2)
                if not isclose(
                    avg_max_sv_distance1,
                    avg_max_sv_distance2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for avg_max_sv_distance1 in avg_max_sv_distances
            for avg_max_sv_distance2 in avg_max_sv_distances
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.generation.coverage

Coverage axioms for retrieval-augmented generation.

For a definition of this utility dimension, see: https://doi.org/10.1145/3626772.3657849

  • Broad coverage: Does the response cover diverse information?
  • Deep coverage: Does the response provide in-depth and highly informative content?

COV1 module-attribute 

COV1: Final = lazy_inject(AspectCountCoverageAxiom)

COV2 module-attribute 

COV2: Final = lazy_inject(AspectRedundancyCoverageAxiom)

COV3 module-attribute 

COV3: Final = lazy_inject(
    AspectSimilaritySentenceCountCoverageAxiom
)

AspectCountCoverageAxiom dataclass 

AspectCountCoverageAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    aspect_extraction: AspectExtraction,
    margin_fraction: NoInject[float] = 0.0,
)

Bases: Axiom[Any, GenerationResponse]

Prefer text with more distinct extracted aspects.

text_contents instance-attribute 

text_contents: TextContents[GenerationResponse]

aspect_extraction instance-attribute 

aspect_extraction: AspectExtraction

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.0

preference 

preference(
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/coverage.py 
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
def preference(
    self,
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    aspects1 = self.aspect_extraction.aspects(
        self.text_contents.contents(output1),
    )
    aspects2 = self.aspect_extraction.aspects(
        self.text_contents.contents(output2),
    )

    count1 = len(aspects1)
    count2 = len(aspects2)
    if isclose(
        count1,
        count2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_greater(count1, count2)

preferences 

preferences(
    input: Any, outputs: Sequence[GenerationResponse]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/coverage.py 
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
def preferences(
    self,
    input: Any,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    contents = (self.text_contents.contents(output) for output in outputs)
    aspects = self.aspect_extraction.iter_aspects(contents)

    counts = [
        len(aspects)
        for aspects in tqdm(
            aspects,
            desc="Extract aspects",
            total=len(outputs),
            disable=not self.verbose,
        )
    ]

    return array(
        [
            (
                strictly_greater(count1, count2)
                if not isclose(
                    count1,
                    count2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for count1 in counts
            for count2 in counts
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

AspectRedundancyCoverageAxiom dataclass 

AspectRedundancyCoverageAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    aspect_extraction: AspectExtraction,
    sentence_similarity: SentenceSimilarity,
    margin_fraction: NoInject[float] = 0.2,
)

Bases: Axiom[Any, GenerationResponse]

Prefer text that has less redundant extracted aspects according to sentence similarity between the aspects.

text_contents instance-attribute 

text_contents: TextContents[GenerationResponse]

aspect_extraction instance-attribute 

aspect_extraction: AspectExtraction

sentence_similarity instance-attribute 

sentence_similarity: SentenceSimilarity

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.2

preference 

preference(
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/coverage.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
def preference(
    self,
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    aspects1 = self.aspect_extraction.aspects(
        self.text_contents.contents(output1),
    )
    aspects2 = self.aspect_extraction.aspects(
        self.text_contents.contents(output2),
    )

    similarities1 = self.sentence_similarity.similarities(list(aspects1))
    similarities2 = self.sentence_similarity.similarities(list(aspects2))

    # TODO: Make aggregation configurable.
    aggregate_similarity1 = similarities1.mean()
    aggregate_similarity2 = similarities2.mean()
    if isclose(
        aggregate_similarity1,
        aggregate_similarity2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_less(aggregate_similarity1, aggregate_similarity2)

preferences 

preferences(
    input: Any, outputs: Sequence[GenerationResponse]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/coverage.py 
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
def preferences(
    self,
    input: Any,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    contents = (self.text_contents.contents(output) for output in outputs)
    aspects = self.aspect_extraction.iter_aspects(contents)
    similarities = (
        self.sentence_similarity.similarities(list(aspects))
        for aspects in tqdm(
            aspects,
            desc="Extract aspects",
            total=len(outputs),
            disable=not self.verbose,
        )
    )
    aggregate_similarities = [similarities.mean() for similarities in similarities]
    return array(
        [
            (
                strictly_less(aggregate_similarity1, aggregate_similarity2)
                if not isclose(
                    aggregate_similarity1,
                    aggregate_similarity2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for aggregate_similarity1 in aggregate_similarities
            for aggregate_similarity2 in aggregate_similarities
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

AspectSimilaritySentenceCountCoverageAxiom dataclass 

AspectSimilaritySentenceCountCoverageAxiom(
    *,
    text_contents: TextContents[
        GenerationRequest | GenerationResponse
    ],
    aspect_extraction: AspectExtraction,
    sentence_tokenizer: SentenceTokenizer,
    sentence_similarity: SentenceSimilarity,
    margin_fraction: NoInject[float] = 0.5,
)

Bases: Axiom[GenerationRequest, GenerationResponse]

Prefer text with extracted aspects from the input text mentioned in more sentences of the output, weighing by the sentence's similarity to the aspects.

text_contents instance-attribute 

text_contents: TextContents[
    GenerationRequest | GenerationResponse
]

aspect_extraction instance-attribute 

aspect_extraction: AspectExtraction

sentence_tokenizer instance-attribute 

sentence_tokenizer: SentenceTokenizer

sentence_similarity instance-attribute 

sentence_similarity: SentenceSimilarity

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.5

preference 

preference(
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/coverage.py 
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
def preference(
    self,
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    input_aspects = self.aspect_extraction.aspects(
        self.text_contents.contents(input)
    )
    if len(input_aspects) == 0:
        return 0

    sentences1 = self.sentence_tokenizer.sentences(
        self.text_contents.contents(output1)
    )
    sentences2 = self.sentence_tokenizer.sentences(
        self.text_contents.contents(output2)
    )

    similarities1 = self.sentence_similarity.paired_similarities(
        list(sentences1), list(input_aspects)
    )
    similarities2 = self.sentence_similarity.paired_similarities(
        list(sentences2), list(input_aspects)
    )

    sentence_weights1: NDArray[float_] = similarities1.mean(axis=1)
    sentence_weights2: NDArray[float_] = similarities2.mean(axis=1)

    sentence_counts1 = sentence_weights1.sum()
    sentence_counts2 = sentence_weights2.sum()

    if isclose(
        sentence_counts1,
        sentence_counts2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_greater(sentence_counts1, sentence_counts2)

preferences 

preferences(
    input: GenerationRequest,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/coverage.py 
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
def preferences(
    self,
    input: GenerationRequest,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    input_aspects = self.aspect_extraction.aspects(
        self.text_contents.contents(input)
    )
    if len(input_aspects) == 0:
        return zeros((len(outputs), len(outputs)))

    sentences = (
        self.sentence_tokenizer.sentences(self.text_contents.contents(output))
        for output in outputs
    )

    similarities = (
        self.sentence_similarity.paired_similarities(
            list(sentences), list(input_aspects)
        )
        for sentences in tqdm(
            sentences,
            desc="Aspect-sentence similarities",
            total=len(outputs),
            disable=not self.verbose,
        )
    )

    sentence_weights: Iterable[NDArray[float_]] = (
        weights.mean(axis=1) for weights in similarities
    )

    sentence_counts = [weights.sum() for weights in sentence_weights]

    return array(
        [
            (
                strictly_greater(sentence_count1, sentence_count2)
                if not isclose(
                    sentence_count1,
                    sentence_count2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for sentence_count1 in sentence_counts
            for sentence_count2 in sentence_counts
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.generation.consistency

Consistency axioms for retrieval-augmented generation.

For a definition of this utility dimension, see: https://doi.org/10.1145/3626772.3657849

  • External consistency: Is the statement accurately conveying from the sources?
  • Internal consistency: Does the statement not contradict itself?

CONS1 module-attribute 

CONS1: Final = lazy_inject(
    AspectSimilaritySentenceCountConsistencyAxiom
)

CONS2 module-attribute 

CONS2: Final = lazy_inject(RougeConsistencyAxiom)

CONS3 module-attribute 

CONS3: Final = lazy_inject(
    EntityContradictionConsistencyAxiom
)

AspectSimilaritySentenceCountConsistencyAxiom dataclass 

AspectSimilaritySentenceCountConsistencyAxiom(
    *,
    text_contents: TextContents[
        GenerationRequest | GenerationResponse
    ],
    aspect_extraction: AspectExtraction,
    sentence_tokenizer: SentenceTokenizer,
    sentence_similarity: SentenceSimilarity,
    margin_fraction: NoInject[float] = 0.5,
)

Bases: Axiom[GenerationRequest, GenerationResponse]

Prefer text with extracted aspects from the input context mentioned in more sentences of the output, weighing by the sentence's similarity to the aspects.

text_contents instance-attribute 

text_contents: TextContents[
    GenerationRequest | GenerationResponse
]

aspect_extraction instance-attribute 

aspect_extraction: AspectExtraction

sentence_tokenizer instance-attribute 

sentence_tokenizer: SentenceTokenizer

sentence_similarity instance-attribute 

sentence_similarity: SentenceSimilarity

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.5

preference 

preference(
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/consistency.py 
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
def preference(
    self,
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    if input.context is None:
        return 0
    context_aspects = set(
        chain.from_iterable(self.aspect_extraction.iter_aspects(input.context))
    )
    if len(context_aspects) == 0:
        return 0

    sentences1 = self.sentence_tokenizer.sentences(
        self.text_contents.contents(output1)
    )
    sentences2 = self.sentence_tokenizer.sentences(
        self.text_contents.contents(output2)
    )

    similarities1 = self.sentence_similarity.paired_similarities(
        list(sentences1), list(context_aspects)
    )
    similarities2 = self.sentence_similarity.paired_similarities(
        list(sentences2), list(context_aspects)
    )

    sentence_weights1: NDArray[float_] = similarities1.mean(axis=1)
    sentence_weights2: NDArray[float_] = similarities2.mean(axis=1)

    sentence_counts1 = sentence_weights1.sum()
    sentence_counts2 = sentence_weights2.sum()

    if isclose(
        sentence_counts1,
        sentence_counts2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_greater(sentence_counts1, sentence_counts2)

preferences 

preferences(
    input: GenerationRequest,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/consistency.py 
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
def preferences(
    self,
    input: GenerationRequest,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    if input.context is None:
        return zeros((len(outputs), len(outputs)))
    context_aspects = set(
        chain.from_iterable(self.aspect_extraction.iter_aspects(input.context))
    )
    if len(context_aspects) == 0:
        return zeros((len(outputs), len(outputs)))

    sentences = (
        self.sentence_tokenizer.sentences(self.text_contents.contents(output))
        for output in outputs
    )

    similarities = (
        self.sentence_similarity.paired_similarities(
            list(sentences), list(context_aspects)
        )
        for sentences in tqdm(
            sentences,
            desc="Aspect-sentence similarities",
            total=len(outputs),
            disable=not self.verbose,
        )
    )

    sentence_weights: Iterable[NDArray[float_]] = (
        weights.mean(axis=1) for weights in similarities
    )

    sentence_counts = [weights.sum() for weights in sentence_weights]

    return array(
        [
            (
                strictly_greater(sentence_count1, sentence_count2)
                if not isclose(
                    sentence_count1,
                    sentence_count2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for sentence_count1 in sentence_counts
            for sentence_count2 in sentence_counts
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

RougeConsistencyAxiom dataclass 

RougeConsistencyAxiom(
    *,
    text_contents: TextContents[
        GenerationRequest | GenerationResponse
    ],
    sentence_tokenizer: SentenceTokenizer,
    term_tokenizer: TermTokenizer,
    margin_fraction: NoInject[float] = 0.1,
)

Bases: Axiom[GenerationRequest, GenerationResponse]

Prefer text with higher ROUGE score compared to the input contexts.

text_contents instance-attribute 

text_contents: TextContents[
    GenerationRequest | GenerationResponse
]

sentence_tokenizer instance-attribute 

sentence_tokenizer: SentenceTokenizer

term_tokenizer instance-attribute 

term_tokenizer: TermTokenizer

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/consistency.py 
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
def preference(
    self,
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    if input.context is None:
        return 0

    context = "\n\n".join(input.context)

    contents1 = self.text_contents.contents(output1)
    contents2 = self.text_contents.contents(output2)

    rouge1: dict[str, Score] = self._rouge_scorer.score(
        target=context,
        prediction=contents1,
    )
    rouge2: dict[str, Score] = self._rouge_scorer.score(
        target=context,
        prediction=contents2,
    )
    rouge_l_sum1 = rouge1["rougeLsum"].fmeasure
    rouge_l_sum2 = rouge2["rougeLsum"].fmeasure

    if isclose(
        rouge_l_sum1,
        rouge_l_sum2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_greater(rouge_l_sum1, rouge_l_sum2)

preferences 

preferences(
    input: GenerationRequest,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/consistency.py 
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
def preferences(
    self,
    input: GenerationRequest,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    if input.context is None:
        return zeros((len(outputs), len(outputs)))

    context = "\n\n".join(input.context)

    contents = (self.text_contents.contents(output) for output in outputs)

    rouges: Iterable[dict[str, Score]] = (
        self._rouge_scorer.score(
            target=context,
            prediction=content,
        )
        for content in tqdm(
            contents,
            desc="Compute ROUGE",
            total=len(outputs),
            unit="output",
            disable=not self.verbose,
        )
    )
    rouge_l_sums = [rouge["rougeLsum"].fmeasure for rouge in rouges]

    return array(
        [
            (
                strictly_greater(rouge_l_sum1, rouge_l_sum2)
                if not isclose(
                    rouge_l_sum1,
                    rouge_l_sum2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for rouge_l_sum1 in rouge_l_sums
            for rouge_l_sum2 in rouge_l_sums
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

EntityContradictionConsistencyAxiom dataclass 

EntityContradictionConsistencyAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    language_name: NoInject[str] = "en_core_web_sm",
    margin_fraction: NoInject[float] = 0.2,
)

Bases: Axiom[Any, GenerationResponse]

Prefer text with entities less frequently mentioned in contradictory phrases.

text_contents instance-attribute 

text_contents: TextContents[GenerationResponse]

language_name class-attribute instance-attribute 

language_name: NoInject[str] = 'en_core_web_sm'

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.2

preference 

preference(
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/consistency.py 
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
def preference(
    self,
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    contents1 = self.text_contents.contents(output1)
    contents2 = self.text_contents.contents(output2)
    contradictions1 = self._contradictions_ratio(contents1)
    contradictions2 = self._contradictions_ratio(contents2)
    if isclose(
        contradictions1,
        contradictions2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_less(contradictions1, contradictions2)

preferences 

preferences(
    input: Any, outputs: Sequence[GenerationResponse]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/consistency.py 
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
def preferences(
    self,
    input: Any,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    contents = (
        self.text_contents.contents(output)
        for output in tqdm(
            outputs,
            desc="Count contradictions",
            unit="output",
            disable=not self.verbose,
        )
    )
    contradictions = [self._contradictions_ratio(content) for content in contents]
    return array(
        [
            (
                strictly_less(contradictions1, contradictions2)
                if not isclose(
                    contradictions1,
                    contradictions2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for contradictions1 in contradictions
            for contradictions2 in contradictions
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.generation.correctness

Correctness axioms for retrieval-augmented generation.

For a definition of this utility dimension, see: https://doi.org/10.1145/3626772.3657849

  • Factuality: Does the statement state things that are verifiable?
  • Topical correctness: Does the statement state things within the scope of the user's information need?

CORR1 module-attribute 

CORR1: Final = lazy_inject(CitationSentenceCorrectnessAxiom)

CitationSentenceCorrectnessAxiom dataclass 

CitationSentenceCorrectnessAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    sentence_tokenizer: SentenceTokenizer,
    margin_fraction: NoInject[float] = 0.1,
)

Bases: Axiom[Any, GenerationResponse]

Prefer text with more sentences containing citations.

text_contents instance-attribute 

text_contents: TextContents[GenerationResponse]

sentence_tokenizer instance-attribute 

sentence_tokenizer: SentenceTokenizer

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.1

preference 

preference(
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/correctness.py 
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
def preference(
    self,
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    contains_citation1 = array(
        [
            _PATTERN_CITATION.search(sentence) is not None
            for sentence in self.sentence_tokenizer.sentences(
                self.text_contents.contents(output1)
            )
        ]
    )
    contains_citation2 = array(
        [
            _PATTERN_CITATION.search(sentence) is not None
            for sentence in self.sentence_tokenizer.sentences(
                self.text_contents.contents(output2)
            )
        ]
    )

    citation_coverage1 = contains_citation1.mean()
    citation_coverage2 = contains_citation2.mean()

    if isclose(
        citation_coverage1,
        citation_coverage2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_less(citation_coverage1, citation_coverage2)

preferences 

preferences(
    input: Any, outputs: Sequence[GenerationResponse]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/correctness.py 
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
def preferences(
    self,
    input: Any,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    contains_citation = (
        array(
            [
                _PATTERN_CITATION.search(sentence) is not None
                for sentence in self.sentence_tokenizer.sentences(
                    self.text_contents.contents(output)
                )
            ]
        )
        for output in tqdm(
            outputs,
            desc="Find citations",
            unit="output",
            disable=not self.verbose,
        )
    )
    citation_coverages = [
        contains_citation.mean() for contains_citation in contains_citation
    ]
    return array(
        [
            (
                strictly_less(citation_coverage1, citation_coverage2)
                if not isclose(
                    citation_coverage1,
                    citation_coverage2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for citation_coverage1 in citation_coverages
            for citation_coverage2 in citation_coverages
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.generation.clarity

Clarity axioms for retrieval-augmented generation.

For a definition of this utility dimension, see: https://doi.org/10.1145/3626772.3657849

  • Language clarity: Is the response concise, comprehensible, lexically and grammatically correct, and user-accessible?
  • Content clarity: Does the response clearly communicate the most salient information and where it stems from?

CLAR1 module-attribute 

CLAR1: Final = lazy_inject(
    LanguageToolGrammarErrorProportionClarityAxiom
)

CLAR2 module-attribute 

CLAR2: Final = lazy_inject(FleschReadingEaseClarityAxiom)

LanguageToolGrammarErrorProportionClarityAxiom dataclass 

LanguageToolGrammarErrorProportionClarityAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    language: NoInject[str] = "en-US",
    enabled_categories: NoInject[
        Iterable[_LanguageToolCategory]
    ] = frozenset({"GRAMMAR"}),
    margin_fraction: NoInject[float] = 0.1,
)

Bases: _LanguageToolErrorProportionClarityAxiom

Prefer text with a lower proportion of characters covered by grammar errors.

enabled_categories class-attribute instance-attribute 

enabled_categories: NoInject[
    Iterable[_LanguageToolCategory]
] = frozenset({"GRAMMAR"})

FleschReadingEaseClarityAxiom dataclass 

FleschReadingEaseClarityAxiom(
    *,
    text_contents: TextContents[GenerationResponse],
    language_name: NoInject[str] = "en_core_web_sm",
    margin_fraction: NoInject[float] = 0.0,
)

Bases: Axiom[Any, GenerationResponse]

Prefer text with higher readability as measured by the Flesch reading ease score.

text_contents instance-attribute 

text_contents: TextContents[GenerationResponse]

language_name class-attribute instance-attribute 

language_name: NoInject[str] = 'en_core_web_sm'

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.0

preference 

preference(
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference
Source code in ir_axioms/axiom/generation/clarity.py 
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
def preference(
    self,
    input: Any,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> Preference:
    document1 = self._language(self.text_contents.contents(output1))
    document2 = self._language(self.text_contents.contents(output2))

    flesch_reading_ease1 = flesch_reading_ease(document1)
    flesch_reading_ease2 = flesch_reading_ease(document2)

    if isclose(
        flesch_reading_ease1,
        flesch_reading_ease2,
        rel_tol=self.margin_fraction,
    ):
        return 0
    return strictly_less(flesch_reading_ease1, flesch_reading_ease2)

preferences 

preferences(
    input: Any, outputs: Sequence[GenerationResponse]
) -> PreferenceMatrix
Source code in ir_axioms/axiom/generation/clarity.py 
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
def preferences(
    self,
    input: Any,
    outputs: Sequence[GenerationResponse],
) -> PreferenceMatrix:
    documents = (
        self._language(self.text_contents.contents(output)) for output in outputs
    )
    flesch_reading_eases = [
        flesch_reading_ease(document) if len(document) > 0 else nan
        for document in tqdm(
            documents,
            total=len(outputs),
            desc="Flesch reading eases",
            unit="output",
            disable=not self.verbose,
        )
    ]
    return array(
        [
            (
                strictly_less(flesch_reading_ease1, flesch_reading_ease2)
                if not isclose(
                    flesch_reading_ease1,
                    flesch_reading_ease2,
                    rel_tol=self.margin_fraction,
                )
                else 0
            )
            for flesch_reading_ease1 in flesch_reading_eases
            for flesch_reading_ease2 in flesch_reading_eases
        ],
        dtype=float_,
    ).reshape((len(outputs), len(outputs)))

ir_axioms.axiom.generation.retrieval

GEN_ArgUC module-attribute 

GEN_ArgUC: Final = lazy_inject(
    GenerativeArgumentativeUnitsCountAxiom
)

GEN_QTArg module-attribute 

GEN_QTArg: Final = lazy_inject(
    GenerativeQueryTermOccurrenceInArgumentativeUnitsAxiom
)

GEN_QTPArg module-attribute 

GEN_QTPArg: Final = lazy_inject(
    GenerativeQueryTermPositionInArgumentativeUnitsAxiom
)

GEN_aSL module-attribute 

GEN_aSL: Final = lazy_inject(
    GenerativeAverageSentenceLengthAxiom
)

GEN_LNC1 module-attribute 

GEN_LNC1: Final = lazy_inject(GenerativeLnc1Axiom)

GEN_TF_LNC module-attribute 

GEN_TF_LNC: Final = lazy_inject(GenerativeTfLncAxiom)

GEN_PROX1 module-attribute 

GEN_PROX1: Final = lazy_inject(GenerativeProx1Axiom)

GEN_PROX2 module-attribute 

GEN_PROX2: Final = lazy_inject(GenerativeProx2Axiom)

GEN_PROX3 module-attribute 

GEN_PROX3: Final = lazy_inject(GenerativeProx3Axiom)

GEN_PROX4 module-attribute 

GEN_PROX4: Final = lazy_inject(GenerativeProx4Axiom)

GEN_PROX5 module-attribute 

GEN_PROX5: Final = lazy_inject(GenerativeProx5Axiom)

GEN_REG module-attribute 

GEN_REG: Final = lazy_inject(GenerativeRegAxiom)

GEN_ANTI_REG module-attribute 

GEN_ANTI_REG: Final = lazy_inject(GenerativeAntiRegAxiom)

GEN_AND module-attribute 

GEN_AND: Final = lazy_inject(GenerativeAndAxiom)

GEN_LEN_AND module-attribute 

GEN_LEN_AND: Final = lazy_inject(GenerativeLenAndAxiom)

GEN_M_AND module-attribute 

GEN_M_AND: Final = lazy_inject(GenerativeModifiedAndAxiom)

GEN_DIV module-attribute 

GEN_DIV: Final = lazy_inject(GenerativeDivAxiom)

GEN_LEN_DIV module-attribute 

GEN_LEN_DIV: Final = lazy_inject(GenerativeLenDivAxiom)

GEN_TFC1 module-attribute 

GEN_TFC1: Final = lazy_inject(GenerativeTfc1Axiom)

GEN_STMC1 module-attribute 

GEN_STMC1: Final = lazy_inject(GenerativeStmc1Axiom)

GEN_STMC2 module-attribute 

GEN_STMC2: Final = lazy_inject(GenerativeStmc2Axiom)

GenerativeArgumentativeUnitsCountAxiom dataclass 

GenerativeArgumentativeUnitsCountAxiom(
    *, axiom: ArgumentativeUnitsCountAxiom
)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: ArgumentativeUnitsCountAxiom

GenerativeQueryTermOccurrenceInArgumentativeUnitsAxiom dataclass 

GenerativeQueryTermOccurrenceInArgumentativeUnitsAxiom(
    *, axiom: QueryTermOccurrenceInArgumentativeUnitsAxiom
)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: QueryTermOccurrenceInArgumentativeUnitsAxiom

GenerativeQueryTermPositionInArgumentativeUnitsAxiom dataclass 

GenerativeQueryTermPositionInArgumentativeUnitsAxiom(
    *, axiom: QueryTermPositionInArgumentativeUnitsAxiom
)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: QueryTermPositionInArgumentativeUnitsAxiom

GenerativeAverageSentenceLengthAxiom dataclass 

GenerativeAverageSentenceLengthAxiom(
    *, axiom: AverageSentenceLengthAxiom
)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: AverageSentenceLengthAxiom

GenerativeLnc1Axiom dataclass 

GenerativeLnc1Axiom(*, axiom: Lnc1Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Lnc1Axiom

GenerativeTfLncAxiom dataclass 

GenerativeTfLncAxiom(*, axiom: TfLncAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: TfLncAxiom

GenerativeProx1Axiom dataclass 

GenerativeProx1Axiom(*, axiom: Prox1Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Prox1Axiom

GenerativeProx2Axiom dataclass 

GenerativeProx2Axiom(*, axiom: Prox2Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Prox2Axiom

GenerativeProx3Axiom dataclass 

GenerativeProx3Axiom(*, axiom: Prox3Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Prox3Axiom

GenerativeProx4Axiom dataclass 

GenerativeProx4Axiom(*, axiom: Prox4Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Prox4Axiom

GenerativeProx5Axiom dataclass 

GenerativeProx5Axiom(*, axiom: Prox5Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Prox5Axiom

GenerativeRegAxiom dataclass 

GenerativeRegAxiom(*, axiom: RegAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: RegAxiom

GenerativeAntiRegAxiom dataclass 

GenerativeAntiRegAxiom(*, axiom: AntiRegAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: AntiRegAxiom

GenerativeAndAxiom dataclass 

GenerativeAndAxiom(*, axiom: AndAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: AndAxiom

GenerativeLenAndAxiom dataclass 

GenerativeLenAndAxiom(*, axiom: LenAndAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: LenAndAxiom

GenerativeModifiedAndAxiom dataclass 

GenerativeModifiedAndAxiom(*, axiom: ModifiedAndAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: ModifiedAndAxiom

GenerativeDivAxiom dataclass 

GenerativeDivAxiom(*, axiom: DivAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: DivAxiom

GenerativeLenDivAxiom dataclass 

GenerativeLenDivAxiom(*, axiom: LenDivAxiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: LenDivAxiom

GenerativeTfc1Axiom dataclass 

GenerativeTfc1Axiom(*, axiom: Tfc1Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Tfc1Axiom

GenerativeStmc1Axiom dataclass 

GenerativeStmc1Axiom(*, axiom: Stmc1Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Stmc1Axiom

GenerativeStmc2Axiom dataclass 

GenerativeStmc2Axiom(*, axiom: Stmc2Axiom)

Bases: _RetrievalAxiomWrapper

axiom instance-attribute 

axiom: Stmc2Axiom

ir_axioms.axiom.generation.trec

TrecRagNuggetScoreType module-attribute 

TrecRagNuggetScoreType: TypeAlias = Literal[
    "all", "vital", "weighted"
]

TrecRagNuggetAxiom dataclass 

TrecRagNuggetAxiom(
    *,
    assignments_path: Path = Path(
        "data/nugget_assignment.20241108.jsonl"
    ),
    score_type: TrecRagNuggetScoreType = "all",
    strict: bool = False,
    margin_fraction: float = 0.0,
)

Bases: Axiom[GenerationRequest, GenerationResponse]

assignments_path class-attribute instance-attribute 

assignments_path: Path = Path(
    "data/nugget_assignment.20241108.jsonl"
)

score_type class-attribute instance-attribute 

score_type: TrecRagNuggetScoreType = 'all'

strict class-attribute instance-attribute 

strict: bool = False

margin_fraction class-attribute instance-attribute 

margin_fraction: float = 0.0

preference 

preference(
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> float
Source code in ir_axioms/axiom/generation/trec.py 
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
def preference(
    self,
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> float:
    df = self._assignments
    df = df[df["query_hash"] == hash(input.text)]
    df1 = df[df["answer_text_hash"] == hash(output1.text)]
    df2 = df[df["answer_text_hash"] == hash(output2.text)]
    if len(df1) == 0 or len(df2) == 0:
        return 0

    score_type: TrecRagNuggetScoreType = self.score_type
    if score_type == "all":
        score1 = df1["assignment_score"].sum() / len(df1)
        score2 = df2["assignment_score"].sum() / len(df2)
    elif score_type == "vital":
        df1 = df1[df1["importance"] == "vital"]
        df2 = df2[df2["importance"] == "vital"]
        score1 = df1["assignment_score"].sum() / len(df1)
        score2 = df2["assignment_score"].sum() / len(df2)
    elif score_type == "weighted":
        df1_vital = df1[df1["importance"] == "vital"]
        df2_vital = df2[df2["importance"] == "vital"]
        df1_okay = df1[df1["importance"] != "vital"]
        df2_okay = df2[df2["importance"] != "vital"]
        score1 = (
            df1_vital["assignment_score"].sum()
            + 0.5 * df1_okay["assignment_score"].sum()
        ) / (len(df1_vital) + 0.5 * len(df1_okay))
        score2 = (
            df2_vital["assignment_score"].sum()
            + 0.5 * df2_okay["assignment_score"].sum()
        ) / (len(df2_vital) + 0.5 * len(df2_okay))
    else:
        raise ValueError(f"Unknown score type: {score_type}")

    if isclose(
        score1,
        score2,
        rel_tol=self.margin_fraction,
    ):
        return 0

    return strictly_greater(score1, score2)

ir_axioms.axiom.generation.crowd

TrecRagCrowdUtilityType module-attribute 

TrecRagCrowdUtilityType: TypeAlias = Literal[
    "overall",
    "logical coherence",
    "stylistic coherence",
    "coherence",
    "internal consistency",
    "consistency",
    "topical correctness",
    "correctness",
    "broad coverage",
    "deep coverage",
    "coverage",
]

TrecRagCrowdAxiom dataclass 

TrecRagCrowdAxiom(
    *,
    ratings_path: Path = Path(
        "data/crowd/ratings.jsonl.gz"
    ),
    responses_path: Path = Path(
        "data/crowd/responses.jsonl.gz"
    ),
    utility_type: TrecRagCrowdUtilityType,
    margin_fraction: float = 0.1,
)

Bases: Axiom[GenerationRequest, GenerationResponse]

ratings_path class-attribute instance-attribute 

ratings_path: Path = Path('data/crowd/ratings.jsonl.gz')

responses_path class-attribute instance-attribute 

responses_path: Path = Path("data/crowd/responses.jsonl.gz")

utility_type instance-attribute 

utility_type: TrecRagCrowdUtilityType

margin_fraction class-attribute instance-attribute 

margin_fraction: float = 0.1

preference 

preference(
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> float
Source code in ir_axioms/axiom/generation/crowd.py 
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
def preference(
    self,
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> float:
    input_hash = hash(input.text)
    output1_hash = hash(output1.text)
    output2_hash = hash(output2.text)

    if hash(output1.text) == hash(output2.text):
        return 0

    prob1, prob2 = self._ratings.get(
        (input_hash, output1_hash, output2_hash), (nan, nan)
    )

    if self.margin_fraction > 0 and isclose(
        prob1, prob2, rel_tol=self.margin_fraction
    ):
        return 0
    return strictly_greater(prob1, prob2)

ir_axioms.axiom.generation.llm

TrecRagLlmOrigAxiom dataclass 

TrecRagLlmOrigAxiom(
    *,
    ratings_path: Path = Path(
        "data/ratings-HuggingFaceTB-SmolLM-360M-Instruct.jsonl"
    ),
    margin_fraction: NoInject[float] = 0.0,
)

Bases: Axiom[GenerationRequest, GenerationResponse]

ratings_path class-attribute instance-attribute 

ratings_path: Path = Path(
    "data/ratings-HuggingFaceTB-SmolLM-360M-Instruct.jsonl"
)

margin_fraction class-attribute instance-attribute 

margin_fraction: NoInject[float] = 0.0

preference 

preference(
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> float
Source code in ir_axioms/axiom/generation/llm.py 
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
def preference(
    self,
    input: GenerationRequest,
    output1: GenerationResponse,
    output2: GenerationResponse,
) -> float:
    input_hash = hash(input.text)
    output1_hash = hash(output1.text)
    output2_hash = hash(output2.text)

    if hash(output1.text) == hash(output2.text):
        return 0

    prob1 = self._ratings.get((input_hash, output1_hash), nan)
    prob2 = self._ratings.get((input_hash, output2_hash), nan)
    if isnan(prob1) or isnan(prob2):
        print(f"Missing rating for input: {input.text}")
        print(f"Missing rating for output1: {output1.text}")
        print(f"Missing rating for output2: {output2.text}")

    if self.margin_fraction > 0 and isclose(
        prob1, prob2, rel_tol=self.margin_fraction
    ):
        return 0
    return strictly_greater(prob1, prob2)