Does a definition of a commitment agree with itself?

Last updated 15 October 2026

Run the same definition twice over the same 150 sentences and it disagrees with itself on 1.60% of them. On real email prose the same definition disagreed with itself on 3.0%, which is why nothing smaller than 2% is reported here as a result.

A definition of what counts as a commitment is only worth building on if it gives the same answer twice. This is the measurement of that, with its method and its limits, because a stability figure without its sample is a number rather than a finding.

What was measured, and what was not

This is agreement, not accuracy: how often two runs of one definition over one set of sentences disagree with each other. It says nothing about whether either run was right.

The figures come from the tooling that produced the definition. The detector the product uses is a separate library with its own version, so everything below is a finding about a definition rather than a claim about a product.

How it was done

What two runs are — “The model never sees gold labels, so two runs of the same definition are repeats of an identical task.”
The sentences — “150 synthetic sentences, 113 FLAG / 37 NO_FLAG.”
How many pairs it was pooled over — “Pooled over 15 repeat-pairs from six repeats”
The real-prose sample — “one mailbox, 3,034 files → 1,059 sent messages with an external recipient → 1,689 usable sentences”

What came back

How often it disagreed with itself — “1.60% disagreement, bootstrap 95% CI 0.44–2.93%.”
How many sentences ever moved — “Only 6 of 150 sentences ever flipped”
The same definition on real prose — “Run-to-run disagreement 3.0%, against 0.0% on the synthetic set.”
What we report nothing below — “The noise floor is 2%.”

The useful consequence is a rule rather than a number. A change smaller than the floor is not reported and not acted on; and a net figure is read beside what the change fixed and what it broke, because a near-zero net can be two large movements in opposite directions.

What it does not show

What it is not — “No gold labels, so none of this is an accuracy claim.”
The breadth — “The limitation on the measurement is breadth, not quality: one model, one effort level, one sitting.”
Where the floor does not apply — “The 2% floor does not transfer to real email.”
The real-prose reading's own limit — “One mailbox, so one writer.”
Which version was measured — “it is the last version this tooling produced”

The real-prose reading is the one to dwell on. A definition that never disagreed with itself on synthetic sentences disagreed with itself on 3.0% of real ones, and the sentences it was least sure about were the ones a person would also argue over. A floor measured on clean data does not transfer to messy data, and using it there would call noise a result.

Why this is published

Two of the things this company says are measurements rather than opinions, and this is one of them. Something that can be checked is better published with its limits than quoted without them.