Learning roadmap · 6 stages · about 27 h · up to Gold

Statistics for data work

Six stages with Python: describe data, simulate chance, measure uncertainty, test claims, model relationships, then decide and report. Built on free books and docs, with one paid book listed with its price.

Goal Analyse a real dataset end to end and defend the numbers: estimates with intervals, tests with effect sizes and a stated limit.

Links opened on 13 Nov 2026 · 37 sources: 37 opened and kept

  • For self-study
  • Fills the gaps
  • 5 h a week
  • Start 16 Nov 2026
  • Goal date 14 Feb 2027
6 stages over about 27 h, climbing from Bronze to Gold. Times are estimates.

Stage 1 of 6

Describe the data first

Bronze · about 5.5 h · 81% hands-on

Clean a dataset, summarize it and plot it before any model or test. This stage prevents most later mistakes.

  1. Step 1 of 4

    A first look: clean, then summarize

    about 1.5 h · 55 min of it hands-on

    Check the data for impossible values and missing codes, then summarize it. Do this before any model or test.

    Before you open a source

    A column of incomes has a mean of 52,000 and a median of 38,000. What does the gap tell you about the data?

    Sources

    • Main source · Book

      Think Stats 3e, chapter 1: Exploratory Data Analysis · Allen B. Downey

      Reads a real survey into pandas, validates it, transforms it and computes summary statistics. Python code throughout.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Reference

      pandas: DataFrame.describe · pandas team

      Shows what describe() reports: count, mean, std, min, quartiles and max, and how to change the percentiles.

      Free · about 10 min · opened 13 Nov 2026

    • Alternative · Documentation

      10 minutes to pandas · pandas team

      A quick tour of DataFrames, selecting, grouping and plotting, for anyone who needs to refresh pandas first.

      Free · about 20 min · opened 13 Nov 2026

    Do

    Load the dataset from Think Stats chapter 1, or a CSV from your own work, with pandas. Run describe(), look for impossible values and codes that stand for missing, fix them, and write down the mean, median, standard deviation and the 5th, 50th and 95th percentiles of one column.

    Check yourself

    1. Why can the mean of incomes be much larger than the median?

      Show the answer

      A few very high incomes pull the mean up but barely move the median. The distribution is skewed to the right.

    Key points

    • Validate before you summarize: look for impossible values, missing data and codes that stand for missing.
    • Mean, median and mode describe the centre. Standard deviation and percentiles describe the spread.
    • A mean far above the median usually means a right-skewed distribution or a few large outliers.
    • DataFrame.describe() reports count, mean, std, min, quartiles and max in one call.
  2. Step 2 of 4

    Histograms and effect size

    about 1.5 h · 55 min of it hands-on

    A histogram shows the shape of the data. Plot each group, then say how big a difference is, not only whether one exists.

    Before you open a source

    Two groups differ in their mean by 0.1. Is that a large difference? What else do you need to know?

    Sources

    • Main source · Book

      Think Stats 3e, chapter 2: Distributions · Allen B. Downey

      Frequency tables, histograms, outliers and effect size, with the first-babies comparison worked in code.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Documentation

      seaborn: Visualizing distributions of data · seaborn team

      Covers histplot, kdeplot and ecdfplot with the options that change how a distribution looks.

      Free · about 25 min · opened 13 Nov 2026

    Do

    Plot a histogram of one numeric column with three bin widths and pick the one that shows the shape best. Split the data into two groups, overlay their histograms, and compute the difference in means and Cohen's d.

    Check yourself

    1. Why report an effect size next to a difference in means?

      Show the answer

      The raw difference depends on the units and the spread. An effect size such as Cohen's d puts it on a common scale, so you can say whether it is small or large.

    Key points

    • A histogram shows how often values fall into each range. The shape tells you more than the mean does.
    • Outliers can be errors or real rare cases. Look at them before you delete them.
    • Effect size, such as Cohen's d (a difference in means divided by a standard deviation), says how big a difference is.
    • seaborn offers histplot, kdeplot and ecdfplot for plotting distributions.
  3. Step 3 of 4

    PMFs, CDFs and percentiles

    about 1.5 h · 55 min of it hands-on

    A PMF gives each value's probability. A CDF gives the share at or below a value, and makes percentiles and group comparisons easy to read.

    Before you open a source

    You ask 100 students how big their class is, and the school reports its average class size. Which number is larger, and why?

    Sources

    Do

    Compute the PMF and the CDF of one column and plot both. Read the median and the 90th percentile from the CDF. Plot the CDFs of two groups on one chart and describe where they differ.

    Check yourself

    1. Why does asking students give a larger average class size than asking the school?

      Show the answer

      A large class has more students, so it is counted by more students. Small classes are under-represented in the answers.

    Key points

    • A PMF maps each value to its probability. A CDF maps each value to the share of data at or below it.
    • A percentile rank is the share of values at or below a value. You can read it off a CDF.
    • Plotting two CDFs on one chart compares groups without choosing bin widths.
    • Class size paradox: asking students about class size gives a larger average than the school's own average.
  4. Step 4 of 4

    Describe one dataset in ten questions

    about 1.5 h · 1.5 h of it hands-on

    Practice the whole first stage on one dataset of your own.

    Sources

    • Main source · Documentation

      10 minutes to pandas · pandas team

      A quick tour of DataFrames, selecting, grouping and plotting, for anyone who needs to refresh pandas first.

      Free · about 20 min · opened 13 Nov 2026

    Do

    Take a dataset from your work or a CSV you care about. Write ten questions that summaries and plots can answer, answer each with a table or chart, and note one answer that surprised you.

    Key points

    • Start from a question, then choose the table or plot that answers it. Avoid plots without a question.
    • Write down what surprised you. Surprises point to errors in the data or to something worth studying.

Signpost at the end of the stage

Base camp

Show, test or explain each of these. Tick 2 of 3 to pass.

What you can show

  • Show a cleaned dataset with a summary table and two plots of one column.
  • Explain why the mean and median of that column differ.
  • Read a percentile off a CDF you plotted and say what it means.

Camp

Describe a real dataset

about 2 h · Part of the capstone

The first section of your analysis report: what the data contains and what it looks like.

Rungs of the ladder

Climb the ladder: first with the guide, then changed, then your own.

  1. Guided Follow Think Stats chapters 1 and 2 on the book's dataset and reproduce the summary table and the histograms.
  2. Changed Do the same on a dataset of your own and add the effect size of one difference between two groups.

It is done when

  • No impossible values remain
  • Every plot answers a written question
  • One effect size is reported with its units

Stage 2 of 6

Chance you can simulate

Bronze · about 3.5 h · 83% hands-on

Probability as long-run frequency, the binomial and normal distributions, and simulation as the tool for every question you cannot solve by formula.

  1. Step 1 of 3

    Chance and long-run frequency

    about 1 h · 45 min of it hands-on

    Probability is a number between 0 and 1. When you cannot work it out, simulate it.

    Before you open a source

    You flip a fair coin 10 times and get 7 heads. Is the coin biased?

    Sources

    • Main source · Interactive

      Seeing Theory: Basic Probability · Brown University

      A virtual coin to flip 100 times that shows the share of heads settling toward the true probability. The site is marked as archived but still works.

      Free · about 20 min · opened 13 Nov 2026

    Do

    Flip the virtual coin 10, 100 and 1,000 times and watch the share of heads. Then simulate 10,000 runs of 10 flips in Python and count how often you see 7 or more heads.

    Check yourself

    1. A fair coin is flipped 10 times. Is getting 7 or more heads surprising?

      • Yes, very surprising
      • No, it happens often
      Show the answer

      No. About 17 percent of runs of ten fair flips give 7 or more heads, so one such run says little about the coin.

    Key points

    • Probability runs from 0 (impossible) to 1 (certain).
    • With few flips the share of heads can be far from 50 percent. With more flips it settles close to the true probability.
    • A simulation answers a probability question by repeating the random process many times and counting.
    • Seven or more heads in ten fair flips happens in about 17 percent of runs, so it is weak evidence of bias.
  2. Step 2 of 3

    Binomial, normal and the central limit theorem

    about 1.5 h · 55 min of it hands-on

    A few named distributions describe most simple random processes. scipy.stats gives them all the same methods.

    Before you open a source

    Means of samples of 30 from a skewed distribution: what shape do you expect for the histogram of those means?

    Sources

    • Main source · Interactive

      Seeing Theory: Probability Distributions · Brown University

      Lets you view the PMF and CDF of Bernoulli, binomial and other distributions, and has a central limit theorem demo. Archived but online.

      Free · about 25 min · opened 13 Nov 2026

    • Alternative · Documentation

      SciPy tutorial: Statistics (scipy.stats) · SciPy

      Explains probability distributions, the common methods, random number generation and the tests in scipy.stats.

      Free · about 30 min · opened 13 Nov 2026

    • Alternative · Reference

      scipy.stats.binom · SciPy

      Reference for the binomial distribution with pmf, cdf, rvs (random samples) and ppf (percentile).

      Free · about 10 min · opened 13 Nov 2026

    Do

    Explore the binomial distribution in the visual. In Python use scipy.stats.binom to compute P(7 or more heads) for n = 10, p = 0.5 and compare it with your simulation. Then draw 1,000 samples of 30 values from a skewed distribution, take each mean, and plot a histogram of the means.

    Check yourself

    1. Which scipy.stats method draws random samples from a distribution?

      • pmf
      • rvs
      • ppf
      Show the answer

      rvs. For example binom(10, 0.5).rvs(1000) draws 1,000 samples.

    Key points

    • A binomial counts successes in n independent trials that share the same probability p.
    • scipy.stats distributions share methods: pmf, cdf, rvs (random samples) and ppf (percentile).
    • The central limit theorem: means of many samples look roughly normal, even when the data are not.
  3. Step 3 of 3

    Simulate your own questions

    about 1.5 h · 1.5 h of it hands-on

    Turn each question into a random process, repeat it, count.

    Sources

    • Main source · Documentation

      SciPy tutorial: Statistics (scipy.stats) · SciPy

      Explains probability distributions, the common methods, random number generation and the tests in scipy.stats.

      Free · about 30 min · opened 13 Nov 2026

    Do

    Answer three questions by simulation with 10,000 runs each, for example: how often do 5 random people include two with the same birth month? Check one answer with a formula or with scipy.stats.

    Key points

    • Write the random process as code first, repeat it 10,000 times, then count.
    • Check a simulation against a formula for a simple case before you trust it on a hard one.

Signpost at the end of the stage

Chance camp

Show, test or explain each of these. Tick 2 of 3 to pass.

What you can show

  • Simulate a probability in Python and compare it with the exact value from scipy.stats.
  • Say in one sentence why seven heads in ten flips is weak evidence of a biased coin.
  • State what the central limit theorem says about sample means.

Stage 3 of 6

Samples and uncertainty

Silver · about 3.5 h · 73% hands-on

Every number from a sample has a margin. Measure it with standard errors and bootstrap intervals, and read it correctly.

  1. Step 1 of 3

    Sampling distributions and standard error

    about 1.5 h · 55 min of it hands-on

    An estimate changes from sample to sample. The standard error tells you by how much.

    Before you open a source

    You draw 100 different samples of 25 people and compute the mean of each. Will the 100 means be identical?

    Sources

    • Main source · Book

      Think Stats 3e, chapter 8: Estimation · Allen B. Downey

      Sampling distributions, standard error and confidence intervals computed by resampling with penguin weights.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Interactive

      Seeing Theory: Probability Distributions · Brown University

      Lets you view the PMF and CDF of Bernoulli, binomial and other distributions, and has a central limit theorem demo. Archived but online.

      Free · about 25 min · opened 13 Nov 2026

    Do

    Open the central limit theorem demo and change the sample size. In Python draw 1,000 samples of n = 10, 40 and 160 from a skewed population, compute each mean, and plot the three histograms of means. Compute the standard deviation of each set of means.

    Check yourself

    1. You quadruple the sample size. By what factor does the standard error of the mean shrink?

      • 2
      • 4
      • 16
      Show the answer

      By a factor of 2, the square root of 4.

    Key points

    • A sampling distribution is the distribution of a statistic, such as the mean, over many samples.
    • The standard error is the standard deviation of that sampling distribution. It measures how much an estimate varies.
    • Bigger samples have smaller standard errors: the standard error of a mean shrinks with the square root of n.
    • Standard error describes an estimate. Standard deviation describes the data.
  2. Step 2 of 3

    Confidence intervals by bootstrap

    about 1.5 h · 55 min of it hands-on

    Resample your own data many times to see how much an estimate varies, then read an interval off the results.

    Before you open a source

    If you resample your data with replacement, will every resample be identical to the original?

    Sources

    • Main source · Book

      Think Stats 3e, chapter 8: Estimation · Allen B. Downey

      Sampling distributions, standard error and confidence intervals computed by resampling with penguin weights.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Reference

      scipy.stats.bootstrap · SciPy

      Computes a two-sided bootstrap confidence interval for a statistic from one sample.

      Free · about 10 min · opened 13 Nov 2026

    Do

    Pick a numeric column. Resample it with replacement 1,000 times, compute the mean each time, and take the 2.5th and 97.5th percentiles as a 95 percent interval. Compare with scipy.stats.bootstrap.

    Key points

    • Bootstrapping resamples your own data with replacement to see how a statistic would vary.
    • The 2.5th and 97.5th percentiles of the bootstrap statistics give a 95 percent confidence interval.
    • scipy.stats.bootstrap computes a confidence interval from one sample.
    • The bootstrap shows sampling variation. It does not repair a biased or tiny sample.
  3. Step 3 of 3

    What a confidence interval says

    about 45 min · 30 min of it hands-on

    A 95 percent interval is a property of the method, not a 95 percent chance for one interval.

    Before you open a source

    A 95 percent interval for a mean is [2.1, 3.4]. Is there a 95 percent chance that the true mean lies inside?

    Sources

    • Main source · Interactive

      Seeing Theory: Frequentist Inference · Brown University

      Point estimates and confidence intervals, with the confidence level defined as a property of the procedure. Archived but online.

      Free · about 25 min · opened 13 Nov 2026

    Do

    Run the confidence interval demo repeatedly at 95 percent and count how many intervals miss the true value. Then write two sentences about an interval you computed: one correct reading and one wrong one.

    Check yourself

    1. In 100 simulated 95 percent intervals, about how many miss the true value?

      Show the answer

      About 5. That is what 95 percent confidence means for the method.

    Key points

    • The confidence level is the share of intervals, built by the same method over repeated samples, that contain the true value.
    • One computed interval either contains the true value or does not. The 95 percent describes the method.
    • Intervals get wider with smaller samples, more variable data and higher confidence levels.

Signpost at the end of the stage

Uncertainty camp

Show, test or explain each of these. Tick 2 of 3 to pass.

What you can show

  • Show a bootstrap interval for a mean from your data and the code that made it.
  • Explain the difference between standard deviation and standard error.
  • Say in two sentences what a 95 percent interval does and does not mean.

Camp

Put error bars on your numbers

about 1 h · Part of the capstone

Every mean in your report gets an interval.

Rungs of the ladder

Climb the ladder: first with the guide, then changed, then your own.

  1. Guided Add a 95 percent bootstrap interval to every mean in your dataset description.
  2. Changed Compare two groups with both intervals on one plot and say what the overlap does and does not show.

Stage 4 of 6

Testing claims

Silver · about 5 h · 70% hands-on

Test a difference by shuffling, check it with a formula, and learn what a p value cannot tell you.

  1. Step 1 of 5

    A test by shuffling

    about 1.5 h · 55 min of it hands-on

    If the group labels did not matter, shuffling them should give similar differences. How often it does is the p value.

    Before you open a source

    If group labels do not matter, what happens to the difference in means when you shuffle them?

    Sources

    • Main source · Book

      Think Stats 3e, chapter 9: Hypothesis Testing · Allen B. Downey

      Tests by shuffling: a difference in means, other statistics, a correlation and proportions, with the p value as a share of shuffles.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Interactive

      Permutation Test: Visual Explanation · Jared Wilber

      An animated walk through shuffling labels and building the distribution of the test statistic.

      Free · about 15 min · opened 13 Nov 2026

    • Alternative · Reference

      scipy.stats.permutation_test · SciPy

      Runs a permutation test for a statistic you define, so the shuffle you wrote by hand can be checked.

      Free · about 10 min · opened 13 Nov 2026

    Do

    Take two groups from your data. Compute the difference in means, then shuffle the group labels 10,000 times and compute the difference each time. The share of shuffles at least as large as your observed difference is the p value. Check it with scipy.stats.permutation_test.

    Check yourself

    1. In 10,000 shuffles, 240 give a difference at least as large as the observed one. What is the p value?

      Show the answer

      0.024, which is 240 / 10,000.

    Key points

    • A hypothesis test asks: if there were no real effect, how often would chance give a result at least this extreme?
    • A permutation test builds that chance model by shuffling the group labels.
    • The p value is the share of shuffled results at least as extreme as the observed one.
    • The method works for means, medians and other statistics, also those without a formula.
  2. Step 2 of 5

    The t test, checked against your shuffle

    about 1 h · 45 min of it hands-on

    The t test is a shortcut for the same question. Check that it agrees with your permutation test.

    Sources

    • Main source · Reference

      scipy.stats.ttest_ind · SciPy

      The t test for two independent samples. Its permutations argument computes the p value by shuffling.

      Free · about 10 min · opened 13 Nov 2026

    Do

    Run scipy.stats.ttest_ind on the same two groups and compare its p value with your permutation result. Then run it again with the permutations argument set to 10,000 and compare again.

    Key points

    • ttest_ind compares the means of two independent samples and returns a statistic and a p value.
    • Its answer should be close to the permutation test for well-behaved data. If not, ask why before you pick one.
    • The permutations argument makes ttest_ind compute the p value by shuffling instead of by formula.
  3. Step 3 of 5

    What a p value is not

    about 1 h · 35 min of it hands-on

    The p value is one of the most misread numbers in data work. Learn the six statements the ASA made about it.

    Before you open a source

    A study reports p = 0.03. Is there a 97 percent chance that the effect is real?

    Sources

    Do

    Read the six principles in the ASA release and write each in one sentence of your own. Find the one that an analysis you know breaks. Then read the base rate fallacy chapter of Statistics Done Wrong.

    Check yourself

    1. Group A improves with p = 0.03, group B with p = 0.20. Can you say A improved more than B?

      Show the answer

      No. The difference between a significant and a non-significant result is not itself significant. Test the difference between A and B directly.

    Key points

    • A p value does not measure the probability that the hypothesis is true.
    • A p value does not measure the size or the importance of an effect.
    • Conclusions should not rest only on whether p passes a threshold such as 0.05.
    • One group being significant and another not does not show that the two groups differ.
  4. Step 4 of 5

    Power and sample size

    about 1 h · 45 min of it hands-on

    Power is the chance that a test finds a real effect of a given size. Work out how much data you need before you collect it.

    Before you open a source

    How many people per group do you need to detect a medium effect with 80 percent power? A few dozen or a few hundred?

    Sources

    • Main source · Book

      Statistics Done Wrong: Statistical power and underpowered statistics · Alex Reinhart

      Defines power as the likelihood of distinguishing an effect of a given size from luck, and why small studies miss real effects.

      Free · about 15 min · opened 13 Nov 2026

    • Alternative · Documentation

      statsmodels: Statistics (stats) · statsmodels team

      Lists the power classes, including TTestIndPower, for finding the sample size needed for a given effect.

      Free · about 15 min · opened 13 Nov 2026

    Do

    Use statsmodels' TTestIndPower to find the sample size per group needed to detect a medium effect (d = 0.5) with 80 percent power at alpha 0.05. Then repeat for d = 0.2 and compare the two numbers.

    Check yourself

    1. Roughly how many per group for d = 0.5, 80 percent power, alpha 0.05, and about how many for d = 0.2?

      Show the answer

      About 64 per group for d = 0.5 and about 400 per group for d = 0.2.

    Key points

    • Power is the likelihood that a test distinguishes an effect of a given size from luck.
    • Underpowered studies miss real effects.
    • Small effects need much more data: for d = 0.2 you need roughly six times more than for d = 0.5.
    • Decide the sample size before you collect data, not after you see the result.
  5. Step 5 of 5Side trail

    Pseudoreplication

    about 30 min · 20 min of it hands-on

    Many measurements from one unit are not many independent samples.

    Sources

    • Main source · Book

      Statistics Done Wrong: Pseudoreplication · Alex Reinhart

      Shows how treating repeated measurements from one unit as independent inflates the evidence.

      Free · about 10 min · opened 13 Nov 2026

    Do

    Think of a dataset where you have several rows per person, site or machine. Write down how you would count the independent units, and what a test on all rows would claim.

    Key points

    • Repeated measurements from one unit (one person, one site) are not independent samples.
    • Counting them as independent inflates the evidence and makes p values too small.

Signpost at the end of the stage

Testing camp

Show, test or explain each of these. Tick 2 of 3 to pass.

What you can show

  • Run a permutation test and a t test on the same data and compare the two p values.
  • Say in two sentences what a p value is and one thing it is not.
  • Compute the sample size you would need for a given effect size and power.

Stage 5 of 6

Relationships and models

Gold · about 4.5 h · 76% hands-on

Correlation, least squares and multiple regression: relate variables, and know when a model misleads.

  1. Step 1 of 4

    Correlation and causation

    about 1 h · 45 min of it hands-on

    Plot first. Correlation summarizes a straight-line relationship and nothing else.

    Before you open a source

    Ice cream sales and drownings both rise in summer. Does one cause the other?

    Sources

    Do

    Make a scatter plot of two columns and compute Pearson and Spearman correlation. Then open the Anscombe's quartet demo and compare the four datasets: same numbers, different pictures.

    Check yourself

    1. What is a confounder in the ice cream and drowning example?

      Show the answer

      Warm weather. It raises ice cream sales and also raises swimming, which raises drownings.

    Key points

    • Plot the scatter first: a correlation number hides curves, clusters and outliers.
    • Pearson correlation measures a linear relationship from -1 to +1. Rank correlation is less sensitive to outliers.
    • Anscombe's quartet: four datasets with almost the same summary numbers and very different plots.
    • Correlation alone does not show causation. Look for confounders, or run an experiment.
  2. Step 2 of 4

    Fit a line by least squares

    about 1.5 h · 55 min of it hands-on

    Least squares picks the line with the smallest total squared error.

    Sources

    • Main source · Book

      Think Stats 3e, chapter 10: Least Squares · Allen B. Downey

      Fitting a line, the coefficient of determination, estimation and visualizing the uncertainty of a fit.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Interactive

      Seeing Theory: Regression Analysis · Brown University

      Drag points to see how least squares reacts, with Anscombe's quartet and correlation. Archived but online.

      Free · about 25 min · opened 13 Nov 2026

    Do

    Fit a line to two columns, plot it over the data, and compute the residuals and R squared. In the regression demo drag one point far away and watch how the slope moves.

    Check yourself

    1. What does an R squared of 0.3 mean?

      Show the answer

      The model explains about 30 percent of the variance in the outcome. The rest is left unexplained.

    Key points

    • Least squares chooses the line that minimizes the sum of squared errors between observations and predictions.
    • R squared, the coefficient of determination, is the share of the variance in y that the model explains.
    • A single outlier can pull the slope a long way. Look at the residuals.
    • Show the fit with its uncertainty, not as a bare line.
  3. Step 3 of 4

    Multiple regression and control variables

    about 1.5 h · 1 h of it hands-on

    Add a second variable and the first one's effect can change. Learn to read that.

    Before you open a source

    You add a second variable to a regression. Does the coefficient of the first variable stay the same?

    Sources

    • Main source · Book

      Think Stats 3e, chapter 11: Multiple Regression · Allen B. Downey

      Multiple regression with StatsModels, control variables, nonlinear relationships and logistic regression.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Documentation

      statsmodels: Linear Regression · statsmodels team

      Documents OLS and how to read its summary of coefficients, standard errors and R squared.

      Free · about 20 min · opened 13 Nov 2026

    Do

    Use statsmodels' OLS with the formula interface to predict an outcome from one variable, then from the same variable plus a second. Compare the coefficient of the first variable in both models and write down what changed and why.

    Key points

    • Multiple regression estimates the effect of each variable while holding the others fixed.
    • Adding a control variable can shrink, grow or even flip another coefficient.
    • The statsmodels OLS summary shows coefficients, standard errors, p values and R squared.
    • Logistic regression models a yes or no outcome.
  4. Step 4 of 4Side trail

    Regression to the mean

    about 45 min · 40 min of it hands-on

    Extreme results tend to be followed by less extreme ones, just by chance.

    Sources

    Do

    Simulate 1,000 people who each take two noisy tests of the same true ability. Select the top 10 percent on test 1 and compare their average on test 2.

    Key points

    • Extreme measurements are followed on average by less extreme ones, even when nothing changed.
    • Do not credit a treatment for an improvement that regression to the mean explains.

Signpost at the end of the stage

Model camp

Show, test or explain each of these. Tick 2 of 3 to pass.

What you can show

  • Show a scatter plot, a fitted line and a residual plot for two of your variables.
  • Show one coefficient that changes when you add a control variable and explain why.
  • Explain why a correlation alone does not show that one variable causes the other.

Camp

Model one relationship

about 1.5 h · Part of the capstone

The modelling section of your report: one relationship, fitted, plotted and read carefully.

Rungs of the ladder

Climb the ladder: first with the guide, then changed, then your own.

  1. Guided Fit and plot one regression on your dataset, with a 95 percent interval for the slope.
  2. Changed Add one control variable and report how the coefficient changed.

Stage 6 of 6

Decide and report

Gold · about 5.5 h · 75% hands-on

Update beliefs with Bayes' rule, analyse a complete A/B test and write the report a colleague can challenge.

  1. Step 1 of 4

    Update beliefs with Bayes' rule

    about 1.5 h · 50 min of it hands-on

    Bayes' rule combines a prior with evidence. For rare conditions the prior matters more than the test accuracy.

    Before you open a source

    A test for a rare disease is 99 percent accurate and you test positive. Is it 99 percent likely that you have the disease?

    Sources

    • Main source · Interactive

      Seeing Theory: Bayesian Inference · Brown University

      A rare-disease test where you set the prior and the accuracy and see the posterior. Archived but online.

      Free · about 25 min · opened 13 Nov 2026

    • Alternative · Book

      Think Bayes, 2nd edition · Allen B. Downey

      The free Python book on Bayesian statistics. Use it for the same ideas with code.

      Free · about 1 h · opened 13 Nov 2026

    • Alternative · Book

      Bayesian Statistics the Fun Way · Will Kurt

      A paid book on Bayes' theorem, priors and posteriors with worked examples. Think Bayes covers the same ground for free.

      Paid · $39.99, seen on 13 Nov 2026 · opened 13 Nov 2026

    Do

    In the demo set a disease that affects 1 in 1,000 people and a test that is 99 percent accurate for both groups, and read the posterior. Then redo it in Python as a table of 100,000 people.

    Check yourself

    1. 1 in 1,000 people have a disease. The test is 99 percent accurate both ways. You test positive. About how likely is it that you have it?

      Show the answer

      About 9 percent. Of 100,000 people, 100 are ill and 99 test positive; of 99,900 healthy people about 999 test positive. 99 / (99 + 999) is about 9 percent.

    Key points

    • Bayes' rule updates a prior probability with evidence to give a posterior probability.
    • The prior matters: for a rare condition most positives are false positives, even with an accurate test.
    • Counting in a table of 100,000 people is an easy way to do the arithmetic.
    • Think Bayes covers the same ideas in Python for free. The paid book is an alternative explanation.
  2. Step 2 of 4

    Analyse an A/B test end to end

    about 1.5 h · 1.5 h of it hands-on

    Combine the stages: an estimate with an interval, a test, an effect size and a power check.

    Sources

    • Main source · Reference

      scipy.stats.permutation_test · SciPy

      Runs a permutation test for a statistic you define, so the shuffle you wrote by hand can be checked.

      Free · about 10 min · opened 13 Nov 2026

    • Alternative · Reference

      scipy.stats.bootstrap · SciPy

      Computes a two-sided bootstrap confidence interval for a statistic from one sample.

      Free · about 10 min · opened 13 Nov 2026

    • Alternative · Book

      Statistics Done Wrong: Stopping rules and regression to the mean · Alex Reinhart

      Explains why stopping a test when p dips below 0.05 distorts results, and why extreme values drift back toward the mean.

      Free · about 15 min · opened 13 Nov 2026

    Do

    Take or simulate conversion data for versions A and B. Estimate each rate with a bootstrap interval, test the difference with a permutation test, compute the effect size, and check the power you had. Write a three-line conclusion with a number and an interval.

    Key points

    • Report an estimate with an interval, then a test, then whether the size matters in practice.
    • Decide the sample size and the metric before you look at the results.
    • Do not stop a test as soon as p dips below 0.05. Stopping rules change the error rate.
  3. Step 3 of 4

    Write the report

    about 1.5 h · 1 h of it hands-on

    A result nobody can read changes nothing. Write the one-page report your colleague can challenge.

    Sources

    • Main source · Book

      Statistics Done Wrong: the free book · Alex Reinhart

      A free book on common statistical mistakes. Its chapters on p values, power, pseudoreplication and regression match the pitfalls in this roadmap.

      Free · about 40 min · opened 13 Nov 2026

    Do

    Write a one-page report on your dataset: the question, the data and its cleaning, summaries, estimates with intervals, tests with effect sizes, a model if needed, the limits, and the decision. Give it to a colleague and ask which sentence they would challenge.

    Key points

    • Lead with the question and the answer, then show the evidence.
    • Give every number a unit and an interval or a sample size.
    • State the limits: the sample, missing data, and what you did not test.
    • Check the report against the common mistakes in Statistics Done Wrong.
  4. Step 4 of 4Side trail

    Pick a course to go deeper

    about 1 h · 40 min of it hands-on

    Choose one free course or book for the next level and finish a chapter before you open another.

    Sources

    Do

    Choose one of MIT 18.05, Penn State STAT 200 or OpenIntro Statistics. Do the first problem set or the exercises of the first two chapters, and note what you now see differently about your report.

    Key points

    • Pick one resource and finish a chapter before you open another.
    • Go through the exercises, not only the text: the exercises are where it sticks.

Signpost at the end of the stage

Summit camp

Show, test or explain each of these. Tick 3 of 3 to pass.

What you can show

  • Hand over a one-page report with a question, estimates with intervals, a test with an effect size and a stated limit.
  • Show the code that reproduces every number in the report.
  • Explain a base-rate example with a rare condition in three sentences.

Camp

The dataset report

about 4 h · Part of the capstone

One dataset, analysed end to end, written up for a reader who will question it.

Rungs of the ladder

Climb the ladder: first with the guide, then changed, then your own.

  1. Guided Write the report following the structure in the last step, using the dataset from stage one.
  2. Changed Add a power calculation and a permutation test to the report and state what each changes.
  3. Your own Pick a new dataset and write the whole report without a template.

Pick one

  • An A/B test of two page versions
  • A survey with two groups of respondents
  • Sensor readings before and after a change

It is done when

  • Every number has an interval or a sample size
  • The report states its limits
  • A colleague could rerun your code and get the same numbers

Capstone

One dataset, one report

The report grows at every camp: a description, then intervals, then tests, then a model, then a one-page report you can defend.

The camps of every stage build up to this.

This outline and the map are the same roadmap: progress made in one shows in the other.

Times are estimates.