A duplicate is not one thing. Two records can be byte-identical, identical except for a header, or the same content in different words, and each of those relations needs a different detector. Near-duplicate detection for training data is the middle case, and the one most pipelines get wrong, either by running only the cheap tier or only the expensive one.
This post defines the three tiers, what each catches, misses and costs, and why a corpus that has had only one of them applied still inflates evaluation scores. It assumes the pipeline described in the stage-by-stage guide to AI data pipelines and goes deeper on one stage.
What counts as a duplicate
An exact duplicate is the same bytes after normalisation: encoding, line endings and trailing whitespace made uniform first. A near-duplicate shares most of its text with another record and differs by small edits: a timestamp, a licence year, a renamed variable, a changed paragraph. A semantic duplicate carries the same information in different words: a paraphrase, a translation, the same question asked twice.
Which relation counts depends on what the data is for. In a pretraining corpus the second copy of a file adds weight and nothing else; in a question-answering set, two phrasings of one question are one item for evaluation but may both be wanted for training. A deduplication policy names the relation, the threshold and the rule for which copy survives, and keeps the counts so someone else can check.
Exact hashing: cheap, and blind to one changed byte
The mechanism is a hash, usually SHA-256, over the normalised bytes of each record, and a set of hashes already seen. The cost is one pass over the corpus and memory for the set. It catches re-crawls, mirrors, vendored copies, unchanged files across forks and repeated uploads.
It misses anything that differs by a single byte: a copyright year, a generated id, a trailing newline the normaliser did not strip. It also misses repetition inside records. Lee et al. (2021) report removing from C4 a single 61-word English sentence that was repeated over 60,000 times; a whole-record hash never sees it, because the sentence sits inside longer documents. Their tool for that case matches repeated substrings with a suffix array, closer to exact hashing than to near-duplicate detection.
Near-duplicate detection for training data with MinHash and LSH
MinHash turns each record into a short signature. The record is shingled into overlapping n-grams of tokens or characters, each of k hash functions is applied to every shingle, and the minimum value per function is kept. The fraction of positions on which two signatures agree estimates the Jaccard similarity of the two shingle sets.
Locality-sensitive hashing makes the comparison tractable. Each signature is cut into bands; two records that agree on every row of any one band become a candidate pair, so most pairs are never compared at all. Candidates are verified, clustered with union-find, and one representative per cluster is kept. The dataset card for The Stack describes this: MinHash with 256 permutations and LSH to find clusters of duplicates across a public code corpus.
This tier catches forks with a few changed lines, re-crawled pages with a new date, templated documents and, depending on the shingle unit, copied code with renamed identifiers. It misses paraphrases and translations, because they share few n-grams, and it is noisy on short records, which have too few shingles for a stable estimate. Its characteristic false positive is boilerplate: two files that share a long licence header and nothing else can look similar, which is why boilerplate is stripped before shingling.
Semantic duplicates: embeddings, and the cost that comes with them
The third tier embeds each record, or each chunk of a long record, with an embedding model, builds an approximate nearest-neighbour index, and treats pairs above a cosine similarity threshold as candidates. It catches what the first two cannot: paraphrases, translations, the same answer written twice, and rephrased copies of benchmark questions.
Its failure mode is the opposite of MinHash. Embeddings are indifferent to the details that make two records different: two financial statements for different quarters, two bug reports with different stack traces, a question and its negation. A high-similarity pair is a candidate for review, not a deletion. Domain text such as code needs an embedding model that has seen code, or the similarities are noise.
The cost is a model call per record, an index, and the human review the false positives force, so it is usually run last, on what survived the first two tiers, and often only on fine-tuning and evaluation sets.
Why you need all three
The tiers target different relations and differ in precision and cost. A later tier would find most of what an earlier one finds, but at a price per record that only makes sense on what survived the cheaper pass, and with hits that need review rather than deletion.
| Tier | Catches | Misses | Cost |
|---|---|---|---|
| Exact hash | Byte-identical copies | Any one-byte edit, repeated substrings | One pass, a hash set |
| MinHash and LSH | Small edits, templates, forks | Paraphrase, translation, short records | Signatures, candidate checks |
| Embeddings | Same meaning, different words | Materially different near-neighbours | A model call per record, review |
Run them in that order: exact hashing shrinks the volume the near tier must sign, and the near tier shrinks what the embedding tier must encode. Each tier should emit a count and a sample of removed pairs, which is what a data quality report should show, so a buyer can see whether "deduplicated" meant one hash pass or three.
Two policy decisions sit outside the algorithms: which representative survives (the earliest, the longest, or the one whose licence is clearest), and what happens when a cluster straddles sources with different terms, where the answer is to keep the record whose provenance and licence permit the use and to record why. For code, where forks dominate, the code domain rules set out how exact and near-duplicate detection are applied across forks, vendored dependencies and generated files.
How duplicates inflate evaluation scores
If a near-copy of an evaluation item sits in the training data, the model has seen the answer, and the score measures memory. Lee et al. (2021) quantify it: train-test overlap affects over 4% of the validation set of standard datasets, and models trained on deduplicated data emit memorised text ten times less frequently.
Duplication inside the evaluation set distorts scores a second way, by weighting the result towards whatever the repeated items test. And semantic duplicates evade n-gram checks entirely, which is why splitting by source rather than by row is the rule that holds up: deduplicate first, split by the unit that generates correlated records, then run near-duplicate detection across the two sides as a check and report what it found.
Runix Pipeline, the tooling Runix Data engagements run on, is designed to run deduplication at these three levels, exact, near and semantic, with counts you can check in the quality report. Runix Pipeline is in development with design partners and Runix Data is in early access; the counts are there to be inspected rather than taken on trust.
Questions this raises
Is MinHash enough for near-duplicate detection?
For edits that keep most of the text, yes: it catches re-crawls, forks and templated documents at a cost most teams can afford. It does not catch paraphrases or translations, which need embedding similarity, and it works best with exact hashing in front of it to cut the volume.
Why does deduplication change evaluation scores?
If a near-copy of an evaluation item is in the training data, the model has seen the answer, so the score measures memory rather than ability. Deduplicating before the split, and checking across the split afterwards, removes that inflation.
Which copy of a duplicate should be kept?
That is a policy decision rather than a technical one: the earliest, the longest, or the one with the clearest licence. Record which rule was applied so the choice can be audited later.