The training file is whatever you fine-tuned on — JSONL, CSV, TSV or plain text, gzipped or not.
The eval file is the set you scored. Both are read with File.stream() inside a Web
Worker in this tab. Neither is uploaded, in whole or in part.
It samples the head of each file, lists every leaf field with its fill rate and average length, and
pre-selects the longest text field. Fields that are identical on more than half the rows — a shared
system prompt, an instruction template — are left out by default, because comparing on
one reports near-total overlap between two perfectly clean files. That is the single most likely way
a tool like this misleads you, so it is handled first and shown.
A match is a run of N consecutive words, lowercased, with punctuation removed — GPT-3's definition, which reads: “we define a gram as a lowercase, whitespace delimited word with no punctuation”. N is the 5th-percentile length of your eval rows in words, floored at 8 and capped at 13, which is also GPT-3's rule. A fixed N of 13 reports zero overlap on a short-question benchmark like TruthfulQA, where 84% of rows are shorter than that — a confident false negative.
Your eval rows' word runs go into a hash table. The training file is then streamed past it once — twice when the boilerplate filter is on, the first pass only counting how many distinct training records contain each run so that template text can be dropped. Nothing of the training file is kept except byte counts and, for a bounded number of rows, one short excerpt so you can see the passage in context.
Every number is normalised by the eval row. This is the whole reason evalspill is not built on MinHash or Jaccard similarity: a hundred-word question sitting verbatim inside a thousand-word training document has a Jaccard of 0.089 and a containment of 1.00. Similarity says the two documents are unrelated; containment says the question is in there, which is the thing you asked.
eval.clean.jsonl is the original bytes of the rows you kept — same field order, same
escaping, same line endings — so it drops straight back into your pipeline. Alongside it:
contamination-report.csv with the shared passage per row,
too-short-to-score.csv listing what was never checked, and a self-contained
overlap-report.html that carries its own settings so the number in it stays citable.
File.stream() in a Web Worker in this tab. There is no upload endpoint in this application, and the event log shows every step.localStorage key holds two checkbox settings and nothing else.It will not search for the setting that makes your overlap look worst. That is how you invent a result. N is chosen by a published rule from your own file and is shown; every number is reported at the settings printed beside it; and the exported report carries those settings inside it so the figure stays checkable after it leaves here.
evalspill answers one question: do any of the questions in your evaluation set already appear inside your training data? Not only as identical rows — an exact-match check finds those — but quoted inside a longer document, or with a few words changed around them.
It exists because the two files you need to compare are usually the two you are least able to hand to anyone: the training corpus is customer data and the eval set is unpublished. So the comparison runs on your machine.
The recipe is GPT-3's, and the interface says so wherever it matters. Reworded questions that share no run of words with training are not found by this version — a lexical method cannot see them, and evalspill says that rather than implying coverage it does not have.
Built by Ben Richardson. The rest of the catalogue is at lab.benrichardson.dev. The source is at github.com/ben-gy/evalspill.