# The floor test, run 08284347

Three files, released under **CC BY 4.0**. Reuse them, including commercially;
attribution required. Attribution line, if you want one ready-made:

> Denes Csaszar. The floor test, run 08284347, read 2026-08-29.

## What is here

`floor-test-cells-08284347.csv`: 111 rows, one per question-by-engine cell.
`floor-test-gap-pairs-08284347.csv`: 957 rows, one per wave PAIR rather than per
cell, across 123 cells and all seven waves. Added 2026-09-03. The cells file
holds one wave pair per cell, which is enough for the floor test and not enough
for the within-cell gap test; this file is what that test is computed from.
`floor-test-summary-08284347.json`: the aggregate figures, with the seeds and
replicate counts, so every number published from these rows can be recomputed.
`SHA256SUMS.txt`: checksums for all three, so a later run can be told from this one.

## The procedure these files come from

Answer engines are non-deterministic: ask the same question twice in the same
hour and the wording already differs. A plain before-and-after comparison
therefore cannot separate change from noise.

**The floor test.** Within one capture wave, the same question is asked of the
same engine several times, and the similarity between those repeat answers is
that cell's own noise floor. Between waves, the same cell's similarity is
measured again. A cell counts as moved only when its between-wave similarity
falls **strictly below its own floor**, and not below a shared threshold. The count
of moved cells is then tested against a permutation null, which is a separate
step and not part of the test itself.

## How to read the columns

- `intent_id`: the question, as a code. The question set itself is not published.
- `engine`: the answer engine.
- `early_wave` / `late_wave`, `early_date` / `late_date`, `gap_days`: the two waves compared.
- `draws_early` / `draws_late`: how many times the question was asked in each wave.
- `within_pairs`: how many repeat pairs the floor rests on. **Median 2, range 0 to 18.**
  A cell with 0 has no floor and is excluded from the test.
- `between_pairs`: pairs compared across the two waves.
- `noise_sim_text`: the cell's own floor: median Jaccard over content tokens between repeats.
- `drift_sim_text`: median Jaccard between waves.
- `noise_sim_domains` / `drift_sim_domains`: the same over the set of cited domains.
- `delta_text`: `drift_sim_text` minus `noise_sim_text`.
- `text_moved_beyond_noise`: `1` when `drift_sim_text < noise_sim_text`, strictly.
  `0` when it is not. **Empty when the cell has no floor**, which is not the same as 0.

The gap-pairs file has its own columns, and they are not the same shape:

- `intent_id` / `engine`: the cell, keyed the same way as in the cells file.
- `wave_a` / `date_a`, `wave_b` / `date_b`, `gap_days`: the two waves of THIS pair.
  A cell appears once per pair of waves in which it has draws, so most cells have
  several rows. A pair whose question text differs between the two waves is not
  in this file at all.
- `pairs`: how many draw-by-draw comparisons the row's similarity is a median over.
- `across_wave_sim_text`: median Jaccard between the draws of `wave_a` and the
  draws of `wave_b`. Never empty in this file.

## Definitions you need to recompute it

**Similarity is Jaccard over content-token sets.** Text is lowercased, URLs are
stripped first, words are matched Unicode-aware, leading and trailing apostrophes
and hyphens are trimmed, tokens of two characters or fewer and stopwords are
dropped, and the rest becomes a set. Domain similarity is Jaccard over the set of
cited domains.

**When both sets are empty the comparison returns nothing, not 1.** Two draws that
cited nothing are not evidence of agreement, so the pair is dropped rather than
counted as perfect agreement.

**Deduplication.** An observation is keyed by run, question, engine, draw index and
a normalised timestamp, then hashed; anything that collides is one observation.
The timestamp normalisation collapses formatting only (separators removed,
trailing sub-second zeros trimmed) and keeps every digit down to the
millisecond. **There is no tolerance window.** Its purpose is that two exports of
one capture do not become two observations; two captures a millisecond apart stay
two.

## What these files do not show

Jaccard over tokens measures wording overlap, not agreement, so this is
answer-text change and not a claim about what the engines recommend. Everything
was captured through a single browser instrument on one account. Whether that
session was signed in was NOT recorded: the field that looks like it records
sign-in state is an alias for one of five capture modes, with no exceptions found
in the archive, so account state and personalisation are neither controlled nor
known and are not separated from changes in the engines. (Corrected 2026-08-31.
This paragraph previously said "a single logged-in browser instrument", which the
archive does not support and which contradicted the capture protocol on file.)
No second scorer has rated anything. And a cell with a high floor is easier to
fall below than one with a low floor, so the criterion carries a known structural
dependency: across quartiles of the floor itself, the share that moved runs 57.7,
84.6, 92.0 and 96.2 percent. (Corrected 2026-08-31 from "56, 84, 92 and 96"; the
first two were rounded down rather than rounded.)

HOW THE FOUR BANDS ARE CUT, added 2026-09-01, because without it these four
shares are not reproducible from this file. Sort the 103 cells with a floor by
noise_sim_text ascending and split into equal rank blocks of 26, 26, 25 and 26.
That is the rule, and it is equivalent to taking quartiles by the inclusive
method and putting a cell in the lowest band whose upper boundary it does not
exceed. Sort on the four decimal places this file carries, not on the three the
site prints: that is the whole of the tie-break, and it is part of the rule. At
four decimals no two of the 103 cells share a floor at any of the three block
boundaries, so the order is already fixed and nothing further has to decide it.
At three decimals two of them do, and the shares move. (Precision added
2026-09-23; the rule named the column and not how many of its digits to read.)

It matters because the plausible alternatives disagree. Quartiles by the
exclusive method give 57.7, 84.6, 88.5 and 100.0. Strict rather than inclusive
comparison at the boundary gives 57.7, 84.0, 92.3 and 96.2. Rounding the column
to three decimals before sorting gives 57.7, 80.8, 96.0 and 96.2: metaai B4 and
metaai R3 both read 0.266 there, they sit either side of the block 2 / block 3
cut, and taking the earlier row first puts the wrong one in block 2. In this
file they are 0.2656 and 0.2655 and no tie-break is needed. A reader doing the
arithmetic correctly under a different convention would land on different numbers and
would reasonably conclude the published shares are wrong.

AND ONE TRAP, stated because it is the obvious thing to try. The three boundary
values printed on the site, 0.200, 0.266 and 0.321, are the column's rounded
quartile VALUES. They are not the cut points that produced the bands, and binning
by them gives 57.7, 81.5, 95.8 and 96.2. Use the rank blocks.

A third of the floors rest on a single repeat pair. 38 of the 103 cells have
exactly one within-wave pair, and 60 draw their floor from one wave only, because
the other wave contributed a single draw and a single draw makes no pair. On the
43 cells whose floor comes from both waves the share that moved is 76.7 percent,
against 82.5 percent overall, and the median difference is -0.032 against -0.065.
The rule for that split: draws_early >= 2 AND draws_late >= 2. (Added 2026-08-31.)

Twelve of the 103 cells used more draws than the draw target frozen in their run
configuration. Removing them raises the headline from 82.5 to 85.7 percent, so the
deviation worked against the reported figure. The run configurations are not
published, so this split cannot be repeated from these files alone.
(Added 2026-08-31.)

HOW THE WITHIN-CELL GAP TEST IS PAIRED, added 2026-09-03, because until that
date neither the rule nor the file it runs on was released and the site's
strongest figure could not be checked. Read `floor-test-gap-pairs-08284347.csv`
in file order. Group rows by `intent_id` and `engine`. Drop a cell with fewer
than two rows. Drop a cell whose rows all carry the same `gap_days`. In each of
the 108 cells that survive, take the row with the smallest `gap_days` and the row
with the largest, and where more than one row ties on either, take THE FIRST SUCH
ROW IN FILE ORDER. The statistic is the larger gap's `across_wave_sim_text` minus
the smaller gap's, rounded to four places. Median -0.0465 over 108 cells, with 87
negative and 21 positive.

THE TIE-BREAK IS PART OF THE RULE AND NOT HOUSEKEEPING. 37 of the 108 cells have
two or more rows tied on the shortest or the longest gap, so the choice is real.
Four other reasonable rules give: last row in file order, median -0.0593 and 89
negative; the most similar of the tied rows, -0.0658 and 93; the least similar,
-0.0423 and 83; the mean of the two tied values, -0.0582 and 90. The direction
holds under all five. The point estimate does not, and the published value is the
second most conservative of them. Treat -0.047 as one defensible point in a range
of -0.042 to -0.066.

CORRECTED 2026-09-03, hours after this section was written. The fifth rule was
first printed as "the median of the tied rows, -0.0658 and 93", and that figure
does not reproduce. Every tie the rule actually touches is exactly two rows: all
37 of the tied groups at a cell's shortest or longest gap hold two, so a median
over them has no middle row, and read one way it repeats "the most similar of the
tied rows" while read the other it is the mean and gives -0.0582 with 90
negative. (Ties elsewhere in the file are not all pairs: of the 119 tied
`gap_days` groups within cells, 20 hold three rows. Those groups sit between a
cell's shortest and longest gap, so the rule never reaches them.) The range
endpoints and the published point estimate are unchanged. Why this happened is at
/changes, not here.

The 95 percent interval around that figure is NOT recomputable from these files.
It is stated in the summary with its seed and its replicate count, and the
resampling code is not released. The same is true of the interval on the median
difference. Both can be believed and neither can be repeated here.

The engines are named in the data. The question set, the scoring anchors and the
capture tool are not published.