When building AI systems, data quality means different things at different stages. During pre-training, quality is measured by the number of tokens processed. During post-training, it shifts to the number of well-crafted examples.

Three dimensions matter most when thinking about data:

  • Quality — is the data accurate and clean?
  • Coverage — does it span the problem space?
  • Quantity — is there enough of it?

What makes a dataset high quality?

Relevance — examples need to match the actual task the model is being trained for. Off-topic data adds noise, not signal.

Task alignment — annotations need to reflect what the task actually requires, not just what’s technically correct. The “right” answer depends on context.

Consistency — different annotators should arrive at similar labels. A clear, unambiguous rubric is essential to make this happen.

Correct formatting — every example should follow the exact input/output format the model expects. Inconsistencies here cause silent failures.

Diversity — a dataset full of near-identical examples teaches the model very little. Variety across examples is what drives generalization.

Compliance — data must respect legal and organizational policies. This is easy to overlook and expensive to fix later.

How much data you need depends on:

  • Which finetuning technique you’re using
  • How complex the task is
  • How well the base model already performs on similar tasks

A good first experiment before scaling data collection: check whether even a small set of high-quality examples moves the needle on the metric you care about.

Synthetic Data

When real data is scarce, synthetic data is a practical alternative:

  • For images: affine transformations (rotation, scaling, flipping) can expand a dataset cheaply
  • For text: synonym substitution, AI-generated paraphrases, or token-level perturbations
  • General approach: adding controlled noise to existing examples to create new variations

Some publicly available synthetic datasets worth knowing:

Simple heuristics for filtering bad data:

  • Remove repetitive examples
  • Drop instructions that are unusually short or long
  • Eliminate cases where the same instruction maps to contradictory responses
  • Remove examples where the output simply echoes the input

Known limitations of AI-generated data:

  1. Quality is hard to measure — there’s no simple metric that reliably tells you if synthetic data is actually good.
  2. Surface-level imitation — a model trained on AI-generated data may look capable but fail to generalize beyond what it has seen.
  3. Model collapse risk — iteratively training on AI-generated outputs can cause gradual performance degradation. Biases also tend to get amplified rather than corrected.
  4. Murky data lineage — when many models share the same synthetic training data, tracing what came from where becomes difficult, and data leakage risks increase.

Model Distillation is a related technique where a smaller model (the student) is trained to replicate the behavior of a larger, more capable model (the teacher). One practical approach: fine-tune the student on synthetic instruction data generated by the teacher model.

Data Preprocessing

Before any training, raw data needs to go through several cleaning steps:

  • Analyze token pairs and sequence lengths to understand the data’s structure
  • Deduplicate — near-duplicate examples waste compute and can skew training:
    • Pairwise comparison: compute similarity scores between all pairs (expensive but thorough)
    • Hashing: bucket examples by hash value, then only compare within buckets (faster)
    • Dimensionality reduction: compress examples into lower-dimensional space first, then run pairwise comparison (good balance of speed and accuracy)
  • Clean and filter based on quality heuristics
  • Format everything correctly — typically as (system prompt, user prompt) or (instruction, response) pairs