The Kolmogorov-Smirnov Test

Suppose you have developed a diagnostic test to screen for a disease from blood results. Before deploying it, you want to know whether it actually separates patients who carry the condition from those who don't. Or suppose the test has been running in a clinic for six months and you want to check whether the patients arriving now look statistically similar to those it was trained on. Or perhaps you are simply checking whether your measurements are roughly normally distributed before applying a method that assumes normality.

These are all, at their core, the same kind of question: are these two distributions the same? The Kolmogorov-Smirnov test11 The one-sample statistic was introduced by Kolmogorov (1933); Smirnov (1948) extended it to the two-sample case used here. Both are collected under Reference 1. is one of the most widely used tools for answering it. It is simple, non-parametric (it makes no assumptions about the shape of the distributions), and it has an intuitive geometric interpretation, which means we can understand it visually.

The Empirical Cumulative Distribution Function

Before we can define the KS statistic, we need to understand the empirical cumulative distribution function, or ECDF. Given a sample of \(n\) observations, the ECDF is a step function that rises by \(1/n\) at each data point. For any value \(x\), it tells you what fraction of your sample falls at or below \(x\).

The ECDF is the simplest possible estimate of a distribution's shape. No binning, no bandwidth choices, no smoothing parameters. Just the data, ordered and counted. This is what makes the KS test distribution-free: it works on the ECDF directly, so it doesn't care whether your data is Gaussian, exponential, or something you've never seen before.22 Formally, the Glivenko–Cantelli theorem guarantees that the ECDF converges uniformly to the true CDF as n → ∞ with probability 1.

The KS Statistic

The KS statistic \(D\) is the maximum vertical distance between two ECDFs:

That's it. You lay two step functions on top of each other and find the point where they're farthest apart. If the two samples come from the same distribution, you'd expect their ECDFs to track each other closely, and \(D\) to be small. If they come from different distributions, the ECDFs will diverge somewhere, and \(D\) will be large.

There's something satisfying about this. The KS statistic doesn't average over the whole distribution, or summarize the difference in means or variances. It finds the single point of maximum disagreement: the worst case. This makes it sensitive to differences in shape, not just location. Two distributions with the same mean but different variances, or the same variance but different skew, will still be caught by the KS test.

In the figure below, try changing the distributions and watch how the gap responds. Pay attention to where the maximum gap occurs: it's often not where you'd intuitively guess.

The p-value

By itself, \(D\) is just a distance. To turn it into a statistical test, we need to ask: if the two samples really did come from the same distribution, how likely is it that we'd observe a gap this large or larger just by chance? That probability is the p-value.

The formula for the approximate p-value of the two-sample KS test is:

The key insight is that the critical value of \(D\) shrinks as sample size grows. With tiny samples, a large gap might just be noise; with thousands of observations, even a small gap can be statistically significant. This is important, and it's why you should never interpret \(D\) in isolation from your sample size.33 The correction factors 0.12 and 0.11 in the p-value formula are from Press et al., Numerical Recipes, 3rd ed. (2007), §14.3.

Interactive Playground

The best way to build intuition is to experiment. Below, you can choose two distributions, tune their parameters, adjust sample sizes, and see the ECDFs, \(D\), and the p-value update in real time. Hit “Resample” to draw fresh random samples with the same settings; notice how \(D\) fluctuates from sample to sample, especially with small \(n\).

Try this: set both distributions to Normal with the same parameters and click Resample a few times. You'll see \(D\) bounce around even though the true distributions are identical. Now change one parameter slightly and watch what happens.

What Makes a Good KS Value?

This is the part that trips people up. There is no universal answer. A KS value of 0.35 can be excellent in one context and alarming in another. The number is the same; what changes is what you're using it for.

The reason is simple: in some settings, a large gap between distributions is exactly what you want (your model successfully separates two groups). In others, a large gap is exactly what you don't want (your production data has drifted from your training data). And in yet others, you're simply checking whether a distributional assumption holds, and any gap at all is bad news.

The lesson: the KS statistic is a pure measure of distance. Whether that distance signals success or raises a warning depends entirely on what the comparison is for.

When to Use the KS Test (and When Not To)

The KS test is most useful when you want a general test for distributional difference, when you don't have a specific alternative hypothesis in mind. Because it's sensitive to any kind of difference (location, scale, shape), it's a good first-pass check.

But this generality comes at a cost. The KS test is often less powerful than tests tailored to a specific alternative. If you specifically want to detect a shift in means, a t-test will generally be more sensitive, particularly when the data is approximately normal or the sample is large. If you want to detect a difference in variance, an F-test is better. The KS test trades sensitivity for breadth: it can detect any difference, but it's not the best at detecting particular differences.

The KS test also assumes continuous distributions. If your data has many tied values (as is common with discrete or ordinal data), the test becomes conservative and will fail to detect real differences more often than it should. And in its one-sample form, where you test your data against a specific theoretical CDF, the p-values are only valid when that distribution's parameters are fully known in advance. If you estimate μ and σ from the same sample and then run the KS test, the p-values are anticonservative, meaning the test rejects the null too often, inflating the Type I error rate (the Lilliefors correction addresses this for the normal case).44 Lilliefors (1967) derived simulation-based critical values for the case where μ and σ are estimated from the same sample, making the standard KS table anticonservative.

Despite these caveats, the KS test remains one of the most intuitive and widely applicable tools in statistics. Its geometric interpretation (the maximum gap between two step functions) is something you can see and feel. And now that you've played with it above, you do.

@article{boixcampos2026kstest,
  author  = {Boix Campos, Javier},
  title   = {The Kolmogorov-Smirnov Test},
  journal = {Vivum},
  year    = {2026},
  url     = {https://vivum-pub.org/editorial/kolmogorov-smirnov-test},
}