To decide whether a sentence makes a promise, we use a written definition of what counts as one. A definition is only worth building on if it gives the same answer each time it reads the same sentence. Think of a bathroom scale: if you step on twice and get two different weights, you can't trust either. This page is that test for our definition, with how we did it and what it can't tell you. A number without those is easy to misread.
What we measured, and what we did not
We measured agreement. That means how often two runs of the same definition, over the same sentences, gave different answers. It says nothing about whether either run was right. A scale can give the same wrong weight every time.
The figures come from the tools we used to write the definition. The tool the product uses to spot promises is a separate piece of software with its own version number. So everything here is about the definition. It says nothing about how the product performs.
How it was done
- What two runs are — “The model never sees gold labels, so two runs of the same definition are repeats of an identical task.”
- The sentences — “150 synthetic sentences, 113 FLAG / 37 NO_FLAG.”
- How many pairs it was pooled over — “Pooled over 15 repeat-pairs from six repeats”
- The real-prose sample — “one mailbox, 3,034 files → 1,059 sent messages with an external recipient → 1,689 usable sentences”
In plain words: the model doing the reading never sees the right answers, so each run is the same task done again from scratch. The sentences were made up for the test. "FLAG" means the sentence makes a promise and "NO_FLAG" means it does not. The real-email sample came from one person's sent mail, cut down to sentences someone outside the company received.
What came back
- How often it disagreed with itself — “1.60% disagreement, bootstrap 95% CI 0.44–2.93%.”
- How many sentences ever moved — “Only 6 of 150 sentences ever flipped”
- The same definition on real prose — “Run-to-run disagreement 3.0%, against 0.0% on the synthetic set.”
- What we report nothing below — “The noise floor is 2%.”
The range after the first figure shows how sure we can be. The real rate is very likely somewhere between 0.44% and 2.93%.
The "noise floor" is the amount the answers move about even when nothing has changed. Any change smaller than that could be chance. So the useful result is a rule. A change smaller than the floor is not reported and we do not act on it. And when a change fixes some answers and breaks others, we look at both numbers. A total close to zero can hide two big changes that cancel each other out.
What it does not show
- What it is not — “No gold labels, so none of this is an accuracy claim.”
- The breadth — “The limitation on the measurement is breadth, not quality: one model, one effort level, one sitting.”
- Where the floor does not apply — “The 2% floor does not transfer to real email.”
- The real-prose reading's own limit — “One mailbox, so one writer.”
- Which version was measured — “it is the last version this tooling produced”
The real-email result is the one to pay most attention to. In that comparison, on made-up sentences, the definition never changed its answer. On real ones it changed its answer 3.0% of the time. The sentences it was least sure about were the ones people would also argue over. A noise floor measured on tidy test sentences does not carry over to messy real email. Using it there would make chance look like a real result.
Why we publish this
Most of what a company says about itself is opinion. Two things we say are measurements that someone else could check, and this is one of them. We think something that can be checked is better published with its limits than quoted without them.