How the result changed.

1 · What I wanted to measure

Whether answer engines change what they say about a subject as weeks pass. The intuition behind it was commercial rather than academic, for the reason the record opens with. The first ten questions were registered on 4 July 2026, before the first capture. The set grew to eighteen over the next five days, and the full eighteen were registered on 9 July.

2 · What I first built

Twelve questions across six engines at first, eighteen across eight by August, captured in waves, each wave compared against the next. The difference between two waves was to be the reading, and the word for it was drift.

The assumption inside that design was never written down, which is how it survived so long. It assumed that two answers to the same question, weeks apart, differ because weeks passed. Nothing in the first design measured how much two answers differ when no time has passed at all.

3 · What running it produced

Read against the first design, that looked like a large amount of drift. It was also the result the first design was not equipped to interpret.

4 · Where the problems were

Five, and only the last is a bug. The first four are the design being wrong in ways that would have produced a publishable number.

The comparison had no baseline. Without a measurement of what the same question does when simply asked again, a between-wave difference is non-determinism with a date written on it. Nothing separated the two.

The analysis was written after six of the seven waves were captured. The questions were registered between 4 July and 9 July; the analysis that ran over them was written on 22 August. Nothing had been published, so nothing was claimed falsely, but a locked question set with an unlocked analysis leaves open every degree of freedom that matters.

And a field was being read as something it does not record. cleanroom_logged_out is true exactly when the capture mode is an incognito window. It says nothing about whether the session was signed in. I had been inferring sign-in state from the capture mode, reasoning that a temporary-chat surface implies an account. Checked against the live surface on 31 August: chatgpt.com/temporary is reachable while signed out. The inference was simply false.

5 · What changed, and what justified it

The baseline became the measurement. Every cell now carries its own noise floor, the median similarity between two draws of the same question, and a cell counts as moved only when the between-wave similarity falls strictly below that cell’s own floor. The drift number stopped being the reading and became one half of a comparison.

The over-target cells were checked rather than removed. Twelve of 103 cells carried more draws than the frozen design allowed.

The share is now also reported by rank block of the floor, because a single average over the four populations above conceals the thing that makes the criterion work. Corrected 17 September 2026.

An analysis plan now exists and is written before the data. It is the practice’s first, and it exists because of the gap named in station 4.

6 · What survived

What did not survive is the point estimate as a headline.

What follows from the observation alone is narrower and still enough: 85 of 103 is the headline and 33 of 43 is the same count on the better-measured subset of it. The second is a subset with its own uncertainty, not the pessimistic end of a range, and a reader given the headline alone would discount the headline on finding the subset later.

Also gone: any claim about account state. That took two passes and not one. The first replaced the sign-in inference above; the pages went on calling account state “held constant and not known” until 3 September, which says two incompatible things: held constant makes it a control, not known makes it a confound. It now reads as what it is, uncontrolled and unobserved, sitting in the between-wave term where, if it acts at all, it can only inflate the reading, not damp it.

7 · What the next study inherits

Six rules. Two of them were paid for by errors named in station 4: the second by the registration gap, the fifth by the field that was read as what it implies. The other four came from elsewhere in this practice and nothing on this page evidences them. Corrected 3 September 2026.

1. The baseline and the effect must span the same amount of time. A floor measured over thirty seconds cannot be subtracted from an effect measured over an hour.
2. The analysis is registered before the data exists, not before the writing starts.
3. Every criterion states its own chance rate. A threshold set without one can sit so low that chance alone crosses it, and then it measures nothing but its own arithmetic.
4. A control whose result is known in advance runs through the identical code path. A control that gets special handling proves only that the special handling works.
5. A field is read as what it records, never as what it implies.
6. A check that cannot fail is not a check. A filter that matches nothing looks exactly like a filter that found nothing wrong.

The study those rules were written for has a registration and has not been captured. The registration is itself still a draft awaiting ratification, and it is currently on hold because of findings from its own review, which is what the rules are for. Corrected 17 September 2026. The method and the pre-registration format, and the study these numbers come from, both carry the detail this page summarises, and what changed here lists every edit to a stated figure or claim since this record began.