Learning roadmap · 6 stages · about 27 h · up to Gold
Statistics for data work
Six stages with Python: describe data, simulate chance, measure uncertainty, test claims, model relationships, then decide and report. Built on free books and docs, with one paid book listed with its price.
Goal Analyse a real dataset end to end and defend the numbers: estimates with intervals, tests with effect sizes and a stated limit.
Links opened on 13 Nov 2026 · 37 sources: 37 opened and kept
- For self-study
- Fills the gaps
- 5 h a week
- Start 16 Nov 2026
- Goal date 14 Feb 2027
Stage 1 of 6
Describe the data first
Clean a dataset, summarize it and plot it before any model or test. This stage prevents most later mistakes.
Step 1 of 4
A first look: clean, then summarize
Check the data for impossible values and missing codes, then summarize it. Do this before any model or test.
Before you open a source
A column of incomes has a mean of 52,000 and a median of 38,000. What does the gap tell you about the data?
Sources
Main source · Book
Think Stats 3e, chapter 1: Exploratory Data Analysis · Allen B. Downey
Reads a real survey into pandas, validates it, transforms it and computes summary statistics. Python code throughout.
Free · about 1 h · opened 13 Nov 2026
Alternative · Reference
pandas: DataFrame.describe · pandas team
Shows what describe() reports: count, mean, std, min, quartiles and max, and how to change the percentiles.
Free · about 10 min · opened 13 Nov 2026
Alternative · Documentation
10 minutes to pandas · pandas team
A quick tour of DataFrames, selecting, grouping and plotting, for anyone who needs to refresh pandas first.
Free · about 20 min · opened 13 Nov 2026
Do
Load the dataset from Think Stats chapter 1, or a CSV from your own work, with pandas. Run describe(), look for impossible values and codes that stand for missing, fix them, and write down the mean, median, standard deviation and the 5th, 50th and 95th percentiles of one column.
Check yourself
Why can the mean of incomes be much larger than the median?
Show the answer
A few very high incomes pull the mean up but barely move the median. The distribution is skewed to the right.
Key points
- Validate before you summarize: look for impossible values, missing data and codes that stand for missing.
- Mean, median and mode describe the centre. Standard deviation and percentiles describe the spread.
- A mean far above the median usually means a right-skewed distribution or a few large outliers.
- DataFrame.describe() reports count, mean, std, min, quartiles and max in one call.
Step 2 of 4
Histograms and effect size
A histogram shows the shape of the data. Plot each group, then say how big a difference is, not only whether one exists.
Before you open a source
Two groups differ in their mean by 0.1. Is that a large difference? What else do you need to know?
Sources
Main source · Book
Think Stats 3e, chapter 2: Distributions · Allen B. Downey
Frequency tables, histograms, outliers and effect size, with the first-babies comparison worked in code.
Free · about 1 h · opened 13 Nov 2026
Alternative · Documentation
seaborn: Visualizing distributions of data · seaborn team
Covers histplot, kdeplot and ecdfplot with the options that change how a distribution looks.
Free · about 25 min · opened 13 Nov 2026
Do
Plot a histogram of one numeric column with three bin widths and pick the one that shows the shape best. Split the data into two groups, overlay their histograms, and compute the difference in means and Cohen's d.
Check yourself
Why report an effect size next to a difference in means?
Show the answer
The raw difference depends on the units and the spread. An effect size such as Cohen's d puts it on a common scale, so you can say whether it is small or large.
Key points
- A histogram shows how often values fall into each range. The shape tells you more than the mean does.
- Outliers can be errors or real rare cases. Look at them before you delete them.
- Effect size, such as Cohen's d (a difference in means divided by a standard deviation), says how big a difference is.
- seaborn offers histplot, kdeplot and ecdfplot for plotting distributions.
Step 3 of 4
PMFs, CDFs and percentiles
A PMF gives each value's probability. A CDF gives the share at or below a value, and makes percentiles and group comparisons easy to read.
Before you open a source
You ask 100 students how big their class is, and the school reports its average class size. Which number is larger, and why?
Sources
Main source · Book
Think Stats 3e, chapter 3: Probability Mass Functions · Allen B. Downey
Defines the PMF, summarizes it, and works through the class size paradox.
Free · about 45 min · opened 13 Nov 2026
Alternative · Book
Think Stats 3e, chapter 4: Cumulative Distribution Functions · Allen B. Downey
Percentiles, percentile ranks, CDFs and how to compare two groups by their CDFs.
Free · about 45 min · opened 13 Nov 2026
Do
Compute the PMF and the CDF of one column and plot both. Read the median and the 90th percentile from the CDF. Plot the CDFs of two groups on one chart and describe where they differ.
Check yourself
Why does asking students give a larger average class size than asking the school?
Show the answer
A large class has more students, so it is counted by more students. Small classes are under-represented in the answers.
Key points
- A PMF maps each value to its probability. A CDF maps each value to the share of data at or below it.
- A percentile rank is the share of values at or below a value. You can read it off a CDF.
- Plotting two CDFs on one chart compares groups without choosing bin widths.
- Class size paradox: asking students about class size gives a larger average than the school's own average.
Step 4 of 4
Describe one dataset in ten questions
Practice the whole first stage on one dataset of your own.
Sources
Main source · Documentation
10 minutes to pandas · pandas team
A quick tour of DataFrames, selecting, grouping and plotting, for anyone who needs to refresh pandas first.
Free · about 20 min · opened 13 Nov 2026
Do
Take a dataset from your work or a CSV you care about. Write ten questions that summaries and plots can answer, answer each with a table or chart, and note one answer that surprised you.
Key points
- Start from a question, then choose the table or plot that answers it. Avoid plots without a question.
- Write down what surprised you. Surprises point to errors in the data or to something worth studying.
Signpost at the end of the stage
Base camp
What you can show
- Show a cleaned dataset with a summary table and two plots of one column.
- Explain why the mean and median of that column differ.
- Read a percentile off a CDF you plotted and say what it means.
Camp
Describe a real dataset
The first section of your analysis report: what the data contains and what it looks like.
Rungs of the ladder
Climb the ladder: first with the guide, then changed, then your own.
- Guided Follow Think Stats chapters 1 and 2 on the book's dataset and reproduce the summary table and the histograms.
- Changed Do the same on a dataset of your own and add the effect size of one difference between two groups.
It is done when
- No impossible values remain
- Every plot answers a written question
- One effect size is reported with its units
Stage 2 of 6
Chance you can simulate
Probability as long-run frequency, the binomial and normal distributions, and simulation as the tool for every question you cannot solve by formula.
Step 1 of 3
Chance and long-run frequency
Probability is a number between 0 and 1. When you cannot work it out, simulate it.
Before you open a source
You flip a fair coin 10 times and get 7 heads. Is the coin biased?
Sources
Main source · Interactive
Seeing Theory: Basic Probability · Brown University
A virtual coin to flip 100 times that shows the share of heads settling toward the true probability. The site is marked as archived but still works.
Free · about 20 min · opened 13 Nov 2026
Do
Flip the virtual coin 10, 100 and 1,000 times and watch the share of heads. Then simulate 10,000 runs of 10 flips in Python and count how often you see 7 or more heads.
Check yourself
A fair coin is flipped 10 times. Is getting 7 or more heads surprising?
Show the answer
No. About 17 percent of runs of ten fair flips give 7 or more heads, so one such run says little about the coin.
Key points
- Probability runs from 0 (impossible) to 1 (certain).
- With few flips the share of heads can be far from 50 percent. With more flips it settles close to the true probability.
- A simulation answers a probability question by repeating the random process many times and counting.
- Seven or more heads in ten fair flips happens in about 17 percent of runs, so it is weak evidence of bias.
Step 2 of 3
Binomial, normal and the central limit theorem
A few named distributions describe most simple random processes. scipy.stats gives them all the same methods.
Before you open a source
Means of samples of 30 from a skewed distribution: what shape do you expect for the histogram of those means?
Sources
Main source · Interactive
Seeing Theory: Probability Distributions · Brown University
Lets you view the PMF and CDF of Bernoulli, binomial and other distributions, and has a central limit theorem demo. Archived but online.
Free · about 25 min · opened 13 Nov 2026
Alternative · Documentation
SciPy tutorial: Statistics (scipy.stats) · SciPy
Explains probability distributions, the common methods, random number generation and the tests in scipy.stats.
Free · about 30 min · opened 13 Nov 2026
Alternative · Reference
scipy.stats.binom · SciPy
Reference for the binomial distribution with pmf, cdf, rvs (random samples) and ppf (percentile).
Free · about 10 min · opened 13 Nov 2026
Do
Explore the binomial distribution in the visual. In Python use scipy.stats.binom to compute P(7 or more heads) for n = 10, p = 0.5 and compare it with your simulation. Then draw 1,000 samples of 30 values from a skewed distribution, take each mean, and plot a histogram of the means.
Check yourself
Which scipy.stats method draws random samples from a distribution?
Show the answer
rvs. For example binom(10, 0.5).rvs(1000) draws 1,000 samples.
Key points
- A binomial counts successes in n independent trials that share the same probability p.
- scipy.stats distributions share methods: pmf, cdf, rvs (random samples) and ppf (percentile).
- The central limit theorem: means of many samples look roughly normal, even when the data are not.
Step 3 of 3
Simulate your own questions
Turn each question into a random process, repeat it, count.
Sources
Main source · Documentation
SciPy tutorial: Statistics (scipy.stats) · SciPy
Explains probability distributions, the common methods, random number generation and the tests in scipy.stats.
Free · about 30 min · opened 13 Nov 2026
Do
Answer three questions by simulation with 10,000 runs each, for example: how often do 5 random people include two with the same birth month? Check one answer with a formula or with scipy.stats.
Key points
- Write the random process as code first, repeat it 10,000 times, then count.
- Check a simulation against a formula for a simple case before you trust it on a hard one.
Signpost at the end of the stage
Chance camp
What you can show
- Simulate a probability in Python and compare it with the exact value from scipy.stats.
- Say in one sentence why seven heads in ten flips is weak evidence of a biased coin.
- State what the central limit theorem says about sample means.
Stage 3 of 6
Samples and uncertainty
Every number from a sample has a margin. Measure it with standard errors and bootstrap intervals, and read it correctly.
Step 1 of 3
Sampling distributions and standard error
An estimate changes from sample to sample. The standard error tells you by how much.
Before you open a source
You draw 100 different samples of 25 people and compute the mean of each. Will the 100 means be identical?
Sources
Main source · Book
Think Stats 3e, chapter 8: Estimation · Allen B. Downey
Sampling distributions, standard error and confidence intervals computed by resampling with penguin weights.
Free · about 1 h · opened 13 Nov 2026
Alternative · Interactive
Seeing Theory: Probability Distributions · Brown University
Lets you view the PMF and CDF of Bernoulli, binomial and other distributions, and has a central limit theorem demo. Archived but online.
Free · about 25 min · opened 13 Nov 2026
Do
Open the central limit theorem demo and change the sample size. In Python draw 1,000 samples of n = 10, 40 and 160 from a skewed population, compute each mean, and plot the three histograms of means. Compute the standard deviation of each set of means.
Check yourself
You quadruple the sample size. By what factor does the standard error of the mean shrink?
Show the answer
By a factor of 2, the square root of 4.
Key points
- A sampling distribution is the distribution of a statistic, such as the mean, over many samples.
- The standard error is the standard deviation of that sampling distribution. It measures how much an estimate varies.
- Bigger samples have smaller standard errors: the standard error of a mean shrinks with the square root of n.
- Standard error describes an estimate. Standard deviation describes the data.
Step 2 of 3
Confidence intervals by bootstrap
Resample your own data many times to see how much an estimate varies, then read an interval off the results.
Before you open a source
If you resample your data with replacement, will every resample be identical to the original?
Sources
Main source · Book
Think Stats 3e, chapter 8: Estimation · Allen B. Downey
Sampling distributions, standard error and confidence intervals computed by resampling with penguin weights.
Free · about 1 h · opened 13 Nov 2026
Alternative · Reference
scipy.stats.bootstrap · SciPy
Computes a two-sided bootstrap confidence interval for a statistic from one sample.
Free · about 10 min · opened 13 Nov 2026
Do
Pick a numeric column. Resample it with replacement 1,000 times, compute the mean each time, and take the 2.5th and 97.5th percentiles as a 95 percent interval. Compare with scipy.stats.bootstrap.
Key points
- Bootstrapping resamples your own data with replacement to see how a statistic would vary.
- The 2.5th and 97.5th percentiles of the bootstrap statistics give a 95 percent confidence interval.
- scipy.stats.bootstrap computes a confidence interval from one sample.
- The bootstrap shows sampling variation. It does not repair a biased or tiny sample.
Step 3 of 3
What a confidence interval says
A 95 percent interval is a property of the method, not a 95 percent chance for one interval.
Before you open a source
A 95 percent interval for a mean is [2.1, 3.4]. Is there a 95 percent chance that the true mean lies inside?
Sources
Main source · Interactive
Seeing Theory: Frequentist Inference · Brown University
Point estimates and confidence intervals, with the confidence level defined as a property of the procedure. Archived but online.
Free · about 25 min · opened 13 Nov 2026
Do
Run the confidence interval demo repeatedly at 95 percent and count how many intervals miss the true value. Then write two sentences about an interval you computed: one correct reading and one wrong one.
Check yourself
In 100 simulated 95 percent intervals, about how many miss the true value?
Show the answer
About 5. That is what 95 percent confidence means for the method.
Key points
- The confidence level is the share of intervals, built by the same method over repeated samples, that contain the true value.
- One computed interval either contains the true value or does not. The 95 percent describes the method.
- Intervals get wider with smaller samples, more variable data and higher confidence levels.
Signpost at the end of the stage
Uncertainty camp
What you can show
- Show a bootstrap interval for a mean from your data and the code that made it.
- Explain the difference between standard deviation and standard error.
- Say in two sentences what a 95 percent interval does and does not mean.
Camp
Put error bars on your numbers
Every mean in your report gets an interval.
Rungs of the ladder
Climb the ladder: first with the guide, then changed, then your own.
- Guided Add a 95 percent bootstrap interval to every mean in your dataset description.
- Changed Compare two groups with both intervals on one plot and say what the overlap does and does not show.
Stage 4 of 6
Testing claims
Test a difference by shuffling, check it with a formula, and learn what a p value cannot tell you.
Step 1 of 5
A test by shuffling
If the group labels did not matter, shuffling them should give similar differences. How often it does is the p value.
Before you open a source
If group labels do not matter, what happens to the difference in means when you shuffle them?
Sources
Main source · Book
Think Stats 3e, chapter 9: Hypothesis Testing · Allen B. Downey
Tests by shuffling: a difference in means, other statistics, a correlation and proportions, with the p value as a share of shuffles.
Free · about 1 h · opened 13 Nov 2026
Alternative · Interactive
Permutation Test: Visual Explanation · Jared Wilber
An animated walk through shuffling labels and building the distribution of the test statistic.
Free · about 15 min · opened 13 Nov 2026
Alternative · Reference
scipy.stats.permutation_test · SciPy
Runs a permutation test for a statistic you define, so the shuffle you wrote by hand can be checked.
Free · about 10 min · opened 13 Nov 2026
Do
Take two groups from your data. Compute the difference in means, then shuffle the group labels 10,000 times and compute the difference each time. The share of shuffles at least as large as your observed difference is the p value. Check it with scipy.stats.permutation_test.
Check yourself
In 10,000 shuffles, 240 give a difference at least as large as the observed one. What is the p value?
Show the answer
0.024, which is 240 / 10,000.
Key points
- A hypothesis test asks: if there were no real effect, how often would chance give a result at least this extreme?
- A permutation test builds that chance model by shuffling the group labels.
- The p value is the share of shuffled results at least as extreme as the observed one.
- The method works for means, medians and other statistics, also those without a formula.
Step 2 of 5
The t test, checked against your shuffle
The t test is a shortcut for the same question. Check that it agrees with your permutation test.
Sources
Main source · Reference
scipy.stats.ttest_ind · SciPy
The t test for two independent samples. Its permutations argument computes the p value by shuffling.
Free · about 10 min · opened 13 Nov 2026
Do
Run scipy.stats.ttest_ind on the same two groups and compare its p value with your permutation result. Then run it again with the permutations argument set to 10,000 and compare again.
Key points
- ttest_ind compares the means of two independent samples and returns a statistic and a p value.
- Its answer should be close to the permutation test for well-behaved data. If not, ask why before you pick one.
- The permutations argument makes ttest_ind compute the p value by shuffling instead of by formula.
Step 3 of 5
What a p value is not
The p value is one of the most misread numbers in data work. Learn the six statements the ASA made about it.
Before you open a source
A study reports p = 0.03. Is there a 97 percent chance that the effect is real?
Sources
Main source · Article
ASA statement on statistical significance and p-values (press release) · American Statistical Association
The ASA release lists six principles for using p values, such as that a p value does not measure the probability that a hypothesis is true.
Free · about 15 min · opened 13 Nov 2026
Alternative · Book
Statistics Done Wrong: The p value and the base rate fallacy · Alex Reinhart
Shows why a small p value does not make a finding 95 percent likely to be true, using the base rate fallacy.
Free · about 15 min · opened 13 Nov 2026
Alternative · Book
Statistics Done Wrong: When differences in significance aren't significant differences · Alex Reinhart
Explains why one group being significant and another not does not show that the two groups differ.
Free · about 15 min · opened 13 Nov 2026
Do
Read the six principles in the ASA release and write each in one sentence of your own. Find the one that an analysis you know breaks. Then read the base rate fallacy chapter of Statistics Done Wrong.
Check yourself
Group A improves with p = 0.03, group B with p = 0.20. Can you say A improved more than B?
Show the answer
No. The difference between a significant and a non-significant result is not itself significant. Test the difference between A and B directly.
Key points
- A p value does not measure the probability that the hypothesis is true.
- A p value does not measure the size or the importance of an effect.
- Conclusions should not rest only on whether p passes a threshold such as 0.05.
- One group being significant and another not does not show that the two groups differ.
Step 4 of 5
Power and sample size
Power is the chance that a test finds a real effect of a given size. Work out how much data you need before you collect it.
Before you open a source
How many people per group do you need to detect a medium effect with 80 percent power? A few dozen or a few hundred?
Sources
Main source · Book
Statistics Done Wrong: Statistical power and underpowered statistics · Alex Reinhart
Defines power as the likelihood of distinguishing an effect of a given size from luck, and why small studies miss real effects.
Free · about 15 min · opened 13 Nov 2026
Alternative · Documentation
statsmodels: Statistics (stats) · statsmodels team
Lists the power classes, including TTestIndPower, for finding the sample size needed for a given effect.
Free · about 15 min · opened 13 Nov 2026
Do
Use statsmodels' TTestIndPower to find the sample size per group needed to detect a medium effect (d = 0.5) with 80 percent power at alpha 0.05. Then repeat for d = 0.2 and compare the two numbers.
Check yourself
Roughly how many per group for d = 0.5, 80 percent power, alpha 0.05, and about how many for d = 0.2?
Show the answer
About 64 per group for d = 0.5 and about 400 per group for d = 0.2.
Key points
- Power is the likelihood that a test distinguishes an effect of a given size from luck.
- Underpowered studies miss real effects.
- Small effects need much more data: for d = 0.2 you need roughly six times more than for d = 0.5.
- Decide the sample size before you collect data, not after you see the result.
Step 5 of 5Side trail
Pseudoreplication
Many measurements from one unit are not many independent samples.
Sources
Main source · Book
Statistics Done Wrong: Pseudoreplication · Alex Reinhart
Shows how treating repeated measurements from one unit as independent inflates the evidence.
Free · about 10 min · opened 13 Nov 2026
Do
Think of a dataset where you have several rows per person, site or machine. Write down how you would count the independent units, and what a test on all rows would claim.
Key points
- Repeated measurements from one unit (one person, one site) are not independent samples.
- Counting them as independent inflates the evidence and makes p values too small.
Signpost at the end of the stage
Testing camp
What you can show
- Run a permutation test and a t test on the same data and compare the two p values.
- Say in two sentences what a p value is and one thing it is not.
- Compute the sample size you would need for a given effect size and power.
Stage 5 of 6
Relationships and models
Correlation, least squares and multiple regression: relate variables, and know when a model misleads.
Step 1 of 4
Correlation and causation
Plot first. Correlation summarizes a straight-line relationship and nothing else.
Before you open a source
Ice cream sales and drownings both rise in summer. Does one cause the other?
Sources
Main source · Book
Think Stats 3e, chapter 7: Relationships between variables · Allen B. Downey
Scatter plots, correlation, rank correlation and a section on correlation and causation.
Free · about 1 h · opened 13 Nov 2026
Alternative · Interactive
Seeing Theory: Regression Analysis · Brown University
Drag points to see how least squares reacts, with Anscombe's quartet and correlation. Archived but online.
Free · about 25 min · opened 13 Nov 2026
Do
Make a scatter plot of two columns and compute Pearson and Spearman correlation. Then open the Anscombe's quartet demo and compare the four datasets: same numbers, different pictures.
Check yourself
What is a confounder in the ice cream and drowning example?
Show the answer
Warm weather. It raises ice cream sales and also raises swimming, which raises drownings.
Key points
- Plot the scatter first: a correlation number hides curves, clusters and outliers.
- Pearson correlation measures a linear relationship from -1 to +1. Rank correlation is less sensitive to outliers.
- Anscombe's quartet: four datasets with almost the same summary numbers and very different plots.
- Correlation alone does not show causation. Look for confounders, or run an experiment.
Step 2 of 4
Fit a line by least squares
Least squares picks the line with the smallest total squared error.
Sources
Main source · Book
Think Stats 3e, chapter 10: Least Squares · Allen B. Downey
Fitting a line, the coefficient of determination, estimation and visualizing the uncertainty of a fit.
Free · about 1 h · opened 13 Nov 2026
Alternative · Interactive
Seeing Theory: Regression Analysis · Brown University
Drag points to see how least squares reacts, with Anscombe's quartet and correlation. Archived but online.
Free · about 25 min · opened 13 Nov 2026
Do
Fit a line to two columns, plot it over the data, and compute the residuals and R squared. In the regression demo drag one point far away and watch how the slope moves.
Check yourself
What does an R squared of 0.3 mean?
Show the answer
The model explains about 30 percent of the variance in the outcome. The rest is left unexplained.
Key points
- Least squares chooses the line that minimizes the sum of squared errors between observations and predictions.
- R squared, the coefficient of determination, is the share of the variance in y that the model explains.
- A single outlier can pull the slope a long way. Look at the residuals.
- Show the fit with its uncertainty, not as a bare line.
Step 3 of 4
Multiple regression and control variables
Add a second variable and the first one's effect can change. Learn to read that.
Before you open a source
You add a second variable to a regression. Does the coefficient of the first variable stay the same?
Sources
Main source · Book
Think Stats 3e, chapter 11: Multiple Regression · Allen B. Downey
Multiple regression with StatsModels, control variables, nonlinear relationships and logistic regression.
Free · about 1 h · opened 13 Nov 2026
Alternative · Documentation
statsmodels: Linear Regression · statsmodels team
Documents OLS and how to read its summary of coefficients, standard errors and R squared.
Free · about 20 min · opened 13 Nov 2026
Do
Use statsmodels' OLS with the formula interface to predict an outcome from one variable, then from the same variable plus a second. Compare the coefficient of the first variable in both models and write down what changed and why.
Key points
- Multiple regression estimates the effect of each variable while holding the others fixed.
- Adding a control variable can shrink, grow or even flip another coefficient.
- The statsmodels OLS summary shows coefficients, standard errors, p values and R squared.
- Logistic regression models a yes or no outcome.
Step 4 of 4Side trail
Regression to the mean
Extreme results tend to be followed by less extreme ones, just by chance.
Sources
Main source · Book
Statistics Done Wrong: Stopping rules and regression to the mean · Alex Reinhart
Explains why stopping a test when p dips below 0.05 distorts results, and why extreme values drift back toward the mean.
Free · about 15 min · opened 13 Nov 2026
Do
Simulate 1,000 people who each take two noisy tests of the same true ability. Select the top 10 percent on test 1 and compare their average on test 2.
Key points
- Extreme measurements are followed on average by less extreme ones, even when nothing changed.
- Do not credit a treatment for an improvement that regression to the mean explains.
Signpost at the end of the stage
Model camp
What you can show
- Show a scatter plot, a fitted line and a residual plot for two of your variables.
- Show one coefficient that changes when you add a control variable and explain why.
- Explain why a correlation alone does not show that one variable causes the other.
Camp
Model one relationship
The modelling section of your report: one relationship, fitted, plotted and read carefully.
Rungs of the ladder
Climb the ladder: first with the guide, then changed, then your own.
- Guided Fit and plot one regression on your dataset, with a 95 percent interval for the slope.
- Changed Add one control variable and report how the coefficient changed.
Stage 6 of 6
Decide and report
Update beliefs with Bayes' rule, analyse a complete A/B test and write the report a colleague can challenge.
Step 1 of 4
Update beliefs with Bayes' rule
Bayes' rule combines a prior with evidence. For rare conditions the prior matters more than the test accuracy.
Before you open a source
A test for a rare disease is 99 percent accurate and you test positive. Is it 99 percent likely that you have the disease?
Sources
Main source · Interactive
Seeing Theory: Bayesian Inference · Brown University
A rare-disease test where you set the prior and the accuracy and see the posterior. Archived but online.
Free · about 25 min · opened 13 Nov 2026
Alternative · Book
Think Bayes, 2nd edition · Allen B. Downey
The free Python book on Bayesian statistics. Use it for the same ideas with code.
Free · about 1 h · opened 13 Nov 2026
Alternative · Book
Bayesian Statistics the Fun Way · Will Kurt
A paid book on Bayes' theorem, priors and posteriors with worked examples. Think Bayes covers the same ground for free.
Paid · $39.99, seen on 13 Nov 2026 · opened 13 Nov 2026
Do
In the demo set a disease that affects 1 in 1,000 people and a test that is 99 percent accurate for both groups, and read the posterior. Then redo it in Python as a table of 100,000 people.
Check yourself
1 in 1,000 people have a disease. The test is 99 percent accurate both ways. You test positive. About how likely is it that you have it?
Show the answer
About 9 percent. Of 100,000 people, 100 are ill and 99 test positive; of 99,900 healthy people about 999 test positive. 99 / (99 + 999) is about 9 percent.
Key points
- Bayes' rule updates a prior probability with evidence to give a posterior probability.
- The prior matters: for a rare condition most positives are false positives, even with an accurate test.
- Counting in a table of 100,000 people is an easy way to do the arithmetic.
- Think Bayes covers the same ideas in Python for free. The paid book is an alternative explanation.
Step 2 of 4
Analyse an A/B test end to end
Combine the stages: an estimate with an interval, a test, an effect size and a power check.
Sources
Main source · Reference
scipy.stats.permutation_test · SciPy
Runs a permutation test for a statistic you define, so the shuffle you wrote by hand can be checked.
Free · about 10 min · opened 13 Nov 2026
Alternative · Reference
scipy.stats.bootstrap · SciPy
Computes a two-sided bootstrap confidence interval for a statistic from one sample.
Free · about 10 min · opened 13 Nov 2026
Alternative · Book
Statistics Done Wrong: Stopping rules and regression to the mean · Alex Reinhart
Explains why stopping a test when p dips below 0.05 distorts results, and why extreme values drift back toward the mean.
Free · about 15 min · opened 13 Nov 2026
Do
Take or simulate conversion data for versions A and B. Estimate each rate with a bootstrap interval, test the difference with a permutation test, compute the effect size, and check the power you had. Write a three-line conclusion with a number and an interval.
Key points
- Report an estimate with an interval, then a test, then whether the size matters in practice.
- Decide the sample size and the metric before you look at the results.
- Do not stop a test as soon as p dips below 0.05. Stopping rules change the error rate.
Step 3 of 4
Write the report
A result nobody can read changes nothing. Write the one-page report your colleague can challenge.
Sources
Main source · Book
Statistics Done Wrong: the free book · Alex Reinhart
A free book on common statistical mistakes. Its chapters on p values, power, pseudoreplication and regression match the pitfalls in this roadmap.
Free · about 40 min · opened 13 Nov 2026
Do
Write a one-page report on your dataset: the question, the data and its cleaning, summaries, estimates with intervals, tests with effect sizes, a model if needed, the limits, and the decision. Give it to a colleague and ask which sentence they would challenge.
Key points
- Lead with the question and the answer, then show the evidence.
- Give every number a unit and an interval or a sample size.
- State the limits: the sample, missing data, and what you did not test.
- Check the report against the common mistakes in Statistics Done Wrong.
Step 4 of 4Side trail
Pick a course to go deeper
Choose one free course or book for the next level and finish a chapter before you open another.
Sources
Main source · Course
MIT OpenCourseWare 18.05: Introduction to Probability and Statistics · MIT
A full university course on probability and statistics with free materials. A route to go deeper after this roadmap.
Free · opened 13 Nov 2026
Alternative · Course
Penn State STAT 200: Elementary Statistics · Penn State
Free online lesson notes for an introductory statistics course. An alternative way to go deeper.
Free · opened 13 Nov 2026
Alternative · Book
OpenIntro Statistics · OpenIntro
A free textbook of introductory statistics. A third way to go deeper, with exercises.
Free · opened 13 Nov 2026
Do
Choose one of MIT 18.05, Penn State STAT 200 or OpenIntro Statistics. Do the first problem set or the exercises of the first two chapters, and note what you now see differently about your report.
Key points
- Pick one resource and finish a chapter before you open another.
- Go through the exercises, not only the text: the exercises are where it sticks.
Signpost at the end of the stage
Summit camp
What you can show
- Hand over a one-page report with a question, estimates with intervals, a test with an effect size and a stated limit.
- Show the code that reproduces every number in the report.
- Explain a base-rate example with a rare condition in three sentences.
Camp
The dataset report
One dataset, analysed end to end, written up for a reader who will question it.
Rungs of the ladder
Climb the ladder: first with the guide, then changed, then your own.
- Guided Write the report following the structure in the last step, using the dataset from stage one.
- Changed Add a power calculation and a permutation test to the report and state what each changes.
- Your own Pick a new dataset and write the whole report without a template.
Pick one
- An A/B test of two page versions
- A survey with two groups of respondents
- Sensor readings before and after a change
It is done when
- Every number has an interval or a sample size
- The report states its limits
- A colleague could rerun your code and get the same numbers
Capstone
One dataset, one report
The report grows at every camp: a description, then intervals, then tests, then a model, then a one-page report you can defend.
The camps of every stage build up to this.