How to evaluate synthetic data before you train on it

A synthetic dataset can look plausible row by row and still be wrong in ways that matter for training. Evaluation is how you find out before a model does. This article lays out a practical evaluation workflow covering statistical fidelity, downstream utility, and privacy, along with the failure modes each check is designed to catch.

Start with the question you need answered

There is no single quality score for synthetic data. A dataset used to share schema-accurate records with a vendor needs different checks than one used to augment a fraud model. Before running any metrics, write down:

• What the data will be used for: training, augmentation, testing, or sharing.
• Which columns, classes, or conditions the downstream model depends on most.
• What the privacy requirement is, and who will have access to the data.
• Which real, held-out data you will compare against. This set must not have been used to train the generator.

That last point is easy to get wrong. If you compare synthetic data against the same records the generator was trained on, you reward memorization. Keep a real holdout set aside from the start.

Fidelity: does it look like the real data?

Fidelity checks compare the synthetic data to the real reference at increasing levels of structure.

Schema and validity

Start with the basics. Every column should have the expected type and range. Categorical values should come from the real vocabulary. Business rules should hold: end dates after start dates, totals that equal the sum of line items, foreign keys that resolve. These checks are cheap and catch a surprising number of problems.

Marginal distributions

Compare each column on its own. For continuous columns, use the Kolmogorov-Smirnov statistic or the Wasserstein distance alongside overlaid histograms. For categorical columns, compare category frequencies with total variation distance or Jensen-Shannon divergence. Look specifically at:

• Tails and extreme values, which generators often clip or smooth.
• Rare categories, which may be dropped entirely.
• Missing-value rates, which carry information and should be preserved.
• Spikes at specific values, such as zeros or round numbers, which are common in real data.

Pairwise and multivariate structure

Matching marginals is necessary but far from sufficient. A generator that samples each column independently will pass every marginal test and be useless for training. Compare correlation matrices, using Pearson or Spearman for numeric pairs and Cramér's V or mutual information for categorical pairs, and inspect the difference matrix rather than a single averaged number. Then look at conditional distributions for the relationships you know matter: income given age and region, transaction amount given merchant category, and so on.

Discriminator test

Train a classifier to distinguish real rows from synthetic rows. If it separates them easily, the generator has left a detectable signature. The classifier's feature importances point to where: often a single column, an interaction, or a formatting artifact. A result near chance is a good sign, though not proof of quality on its own.

Utility: does it train useful models?

Fidelity metrics describe the data. Utility metrics describe what the data does for your task, which is what you ultimately care about.

Train on synthetic, test on real

The standard check is TSTR: train a model on synthetic data and evaluate it on the real holdout set. Compare it to TRTR, the same model trained on real training data and evaluated on the same holdout. The gap between the two tells you how much task-relevant signal the synthetic data carries.

Run this with the model family you actually intend to use, not only a convenient baseline. Also compare more than headline accuracy:

• Per-class precision and recall, especially for minority classes.
• Performance on the subgroups and slices you care about.
• Calibration, since a model trained on overly clean synthetic data can become overconfident.
• Feature importance rankings, to see whether the model learned the same drivers.

Augmentation experiments

If synthetic data will be mixed with real data, evaluate that directly. Train on real data alone, then on real plus synthetic at several ratios, and evaluate each on the real holdout. This shows whether synthetic data adds value and where adding more stops helping.

Every evaluation should end on real data the generator has never seen. Anything else measures how well the synthetic data agrees with itself.

Privacy: does it leak the source?

Synthetic data is only safer than real data if it does not reproduce identifiable records. Test this explicitly.

Exact and near-duplicate matches

Check for synthetic rows that exactly match training rows. Then check for near matches, which are more common and more telling.

Distance to closest record

For each synthetic row, compute the distance to its nearest neighbour in the generator's training data. Do the same for each row in the real holdout set. If synthetic rows are systematically closer to training records than holdout rows are, the generator is memorizing. The comparison against holdout matters: some closeness is expected simply because both sets come from the same population.

Membership and attribute inference

Membership inference tests ask whether an attacker with the synthetic data can tell if a given record was in the training set. Attribute inference tests ask whether, knowing some fields of a real person, an attacker can predict a sensitive field more accurately from the synthetic data than from general population knowledge. Both are more realistic threat models than duplicate counts alone.

Outliers

Pay special attention to rare records. A person with an unusual combination of attributes is easier to re-identify, and generators are more likely to reproduce such records closely because there is little else to learn from in that region. Differential privacy during training offers a formal bound on this risk, at a cost to fidelity that should itself be measured.

Common failure modes

• Mode collapse: the generator produces a narrow range of typical records and misses real diversity.
• Lost interactions: marginals match, but relationships between columns are weakened or broken.
• Tail truncation: extreme values and rare categories are under-represented.
• Constraint violations: records that are statistically plausible but logically impossible.
• Memorization: close copies of training records, often concentrated among outliers.
• Leakage through evaluation: scores inflated by comparing against the generator's own training data.

Make it a report, not a one-off

Evaluation is most useful when it is repeatable. Fix the holdout set, the metrics, and the thresholds, then produce the same report every time a generator is retrained or its settings change. That turns synthetic data quality into something you can track, review, and sign off on, rather than a judgement made once and forgotten.

SETC's evaluation reports cover fidelity, utility, and privacy against your reference data; the product page describes what they contain. To see them on your own data, book a demo.