6 Steps to AI-Ready Data Sets

Welcome back to our technical how-to series on AI. In this installment, we’ll go through a step-by-step walkthrough to turn data sources into a clean, deduplicated, well-blended training corpus. In essence, covering the process frontier labs use before training an LLM.

The quality of training data is the single biggest lever for model quality. A smaller model trained on well-curated data routinely outperforms a larger model trained on raw web crawls. The DCLM paper demonstrated that switching from heuristic-only filtering to model-based quality filtering improved downstream task accuracy by 6.6 percentage points, with the same model architecture and training budget.

Every frontier lab—OpenAI, Anthropic, Google, Meta—invests heavily in data pipelines before a single training run begins.

Step 1: Download data sources

We start by pulling raw text from four open, high-quality sources that mirror what frontier labs use. Each source is streamed from HuggingFace using the “datasets” library and saved as a JSONL file (one JSON object per line, with “text,” “source,” and “url” fields).

We download ~1.7M items per source for a Chinchilla-optimal data set.

Step 2: Safety and PII filtering

This runs before quality scoring on purpose. A well-written military manual would score high on quality—we need to remove dangerous content before any quality classifier ever sees it.

Harmful content blocking

Pattern matches for weapon synthesis instructions (e.g., nerve agents, explosives) and CSAM indicators. The check is applied to the first 5,000 characters of each document. Any match means the entire document is discarded.

PII detection and redaction—a three-tier policy

The pipeline applies a tiered approach to personally identifiable information (PII):

1. Hard discard: Documents containing Social Security numbers or credit card numbers are removed entirely—these are too sensitive to redact.
2. Density discard: If the total PII character count exceeds 1% of the document length, the document is discarded. A page that is mostly phone numbers and email addresses is likely a data dump, not useful training text.
3. Redaction: For documents with low-density PII (a stray email address, a phone number in a contact section), the PII is replaced with placeholders: “[EMAIL],” “[PHONE],” “[IP],” etc. The document is kept.

The five PII types detected are: email addresses, US phone numbers, US SSNs, credit card numbers (Visa, Mastercard, Amex, Diners), and IPv4 addresses.

Everything else passes through

The vast majority of documents are clean and go straight to the next step. We log per-file statistics to show how many documents were blocked, discarded, redacted, or passed clean.

Step 3: Heuristic filters

These are the cheapest filters in the pipeline, pure CPU string operations. Their job: Throw away documents that are obviously not useful text before we spend compute on the quality classifier.

Based on the Gopher paper—the same rules used by FineWeb, DCLM, Dolma, and RedPajama:

Text normalization

Before applying filters, every document is normalized:

• Encoding fixes via “ftfy” repairs mojibake, broken Unicode, and HTML entities
• Unicode NFC normalization ensures consistent representation of accented characters
• Whitespace cleanup collapses runs of three or more newlines into double newlines

The normalized text is passed downstream so the following steps work with clean text.

Step 4: Quality classifier (DCLM)

This is the core third-generation technique that separates modern pipelines from the heuristic-only era. We train a fastText classifier to distinguish high-quality text from mediocre web text.

We use the approach from DCLM, the best-performing open quality filter as of 2024.

How it works

1. Build training data. The classifier needs labeled examples:

• Positive class (“__label__hq”): Wikipedia text from the heuristic-filtered output. 
• Negative class (“__label__lq”): Random web text sampled from FineWeb-Edu and DCLM-Baseline files.

Each example is truncated to 500 characters. A 95/5 train/validation split is applied. In our labs, we used 20,000 examples per class.

2. Train a fastText classifier. The model is lightweight and trains on CPU in seconds. Validation precision and recall are logged so you can sanity-check the model before it filters your corpus.

3. Score every document. Each document’s first 512 characters are scored on a 0–1 scale. Documents scoring below 0.5 (the DCLM baseline threshold) are discarded. The score is saved as “dclm_score” in the output for later analysis.

Typical retention: 10%–30% of web text passes. Wikipedia passes almost entirely (it is similar to the positive training data). This is by design—the classifier learns to prefer Wikipedia-like writing quality. The trained classifier model is saved and reused on subsequent runs.

Step 5: Deduplication and contamination removal

Deduplication has the highest ROI of any single pipeline step. A few duplicate copies of a document barely matter, but heavy duplication causes memorization and inflated benchmark scores.

The approach is a streaming, memory-efficient architecture: Documents flow through disk-backed intermediate files rather than being loaded entirely into RAM. Temporary files are cleaned up automatically when the step completes.

Layer 1: Exact hash dedup (SHA-256)

Each document’s text is whitespace-normalized (collapsed to single spaces, lowercased) and hashed with SHA-256. If two documents produce the same hash, only the first is kept. Only the set of hash digests lives in memory; documents themselves stream through disk.

Layer 2: MinHash LSH (near-duplicate detection)

Approximates Jaccard similarity between documents using five-word shingles and 128-hash permutations. Documents sharing more than 70% of their shingles are considered near-duplicates. The LSH index is built per file and freed before processing the next file to manage memory.

In our implementation, we use the “datasketch” library’s “MinHash” and “MinHashLSH” classes. This step catches reworded copies, boilerplate-heavy pages, and syndicated content with minor edits.

Layer 3: Eval set contamination removal

Before finalizing training data, we check for overlap with evaluation benchmarks. We build a set of 13-word n-grams from up to 2,000 examples each from HellaSwag and MMLU (streamed from HuggingFace). If more than 10% of a document’s 13 grams match the eval set, the document is discarded.

Without this step, your benchmark scores are inflated—the model has “seen the test.” This is the standard approach used by GPT-3 and Llama.

Step 6: Blending, Packing, and Corpus Card

The final step transforms clean, deduplicated sources into training-ready data.

Blending

If you concatenate all sources naively, web text dominates at 90%+. Overrepresenting books and Wikipedia—even though they are a tiny fraction of total tokens—measurably improves coherence and factual accuracy. This mirrors Llama 3, Falcon, and Gemma data mixing decisions.

The blend weights (must sum to 1.0):