Comparison of two means

You measure a numeric outcome in two groups and want to know if the group means really differ, or if the gap you see is just sampling noise. Birth weight in babies of smokers versus non-smokers. Systolic blood pressure before and after a salt-reduction program. Length of stay for two dengue fluid protocols. The tool for all of these is the t-test, and the first decision is which t-test. This lesson works that decision end to end: independent groups, paired measurements, the assumptions behind both, and what to do when those assumptions break.

Here is the running case for the first half. A district clinic in Johor ran a 12-week trial. Fifteen hypertensive adults joined a structured diet program, and a separate fifteen stayed on usual care. The outcome is systolic blood pressure (SBP) in mmHg at week 12. The two groups are different people, so the measurements are independent.

The two-sample (independent) t-test

The estimate of interest is the difference in means, . Here that is 132.0 minus 140.0, so −8.0 mmHg. The diet group sits 8 mmHg lower on average. The question is whether a difference that size is more than chance would produce when the two population means are equal.

To judge that, you need the standard error of the difference. When you are willing to assume both groups share one underlying spread, you pool the two sample standard deviations into a single estimate, giving more weight to the larger sample.

With the numbers above, . The standard error of the difference then combines the uncertainty from both group means.

That gives mmHg. The test statistic is the difference measured in standard errors, and it is compared against a t distribution.

So on 28 degrees of freedom, which gives a two-sided P-value of about 0.04. The data give moderate evidence against the null hypothesis of no difference.

A confidence interval for the difference

The P-value tells you whether zero is plausible. The confidence interval tells you the range of differences the data support, which is the more useful answer for a clinician. It uses , the 5% point of the t distribution on the same degrees of freedom (2.048 for 28 d.f.).

Here the margin is , so the 95% CI is −8.0 ± 7.52, that is −15.5 to −0.5 mmHg. With 95% confidence, the diet program lowers mean SBP by between 0.5 and 15.5 mmHg. The interval excludes zero, which agrees with the P-value below 0.05. It also shows the effect could be trivial (half a mmHg) or large (15 mmHg). A small trial buys a wide interval.

n1 <- 15; m1 <- 132; s1 <- 9
n0 <- 15; m0 <- 140; s0 <- 11

# pool the SDs, get the s.e., then t and its P-value
sp <- sqrt(((n1-1)*s1^2 + (n0-1)*s0^2) / (n1+n0-2))

sp <- sqrt(((n1-1)*s1^2 + (n0-1)*s0^2) / (n1+n0-2))
se <- sp * sqrt(1/n1 + 1/n0)
tval <- (m1 - m0) / se
df <- n1 + n0 - 2
p <- 2 * pt(-abs(tval), df)
tcrit <- qt(0.975, df)
ci <- (m1 - m0) + c(-1, 1) * tcrit * se
round(c(sp=sp, se=se, t=tval, df=df, p=p, lo=ci[1], hi=ci[2]), 3)
# sp 10.05, se 3.670, t -2.180, df 28, p 0.038, CI -15.52 to -0.48

import numpy as np
from scipy import stats
n1, m1, s1 = 15, 132, 9
n0, m0, s0 = 15, 140, 11

# pool the SDs, get the s.e., then t and its P-value
sp = np.sqrt(((n1-1)*s1**2 + (n0-1)*s0**2) / (n1+n0-2))

sp = np.sqrt(((n1-1)*s1**2 + (n0-1)*s0**2) / (n1+n0-2))
se = sp * np.sqrt(1/n1 + 1/n0)
t = (m1 - m0) / se
df = n1 + n0 - 2
p = 2 * stats.t.cdf(-abs(t), df)
tcrit = stats.t.ppf(0.975, df)
lo, hi = (m1 - m0) - tcrit*se, (m1 - m0) + tcrit*se
print(round(sp,3), round(se,3), round(t,3), df, round(p,3), round(lo,2), round(hi,2))
# 10.05 3.67 -2.18 28 0.038 -15.52 -0.48

Assumptions: normality and equal variance

The two-sample t-test rests on two assumptions. First, the outcome is roughly normal within each group. Second, the two groups have equal population variances, which is what lets you pool the SDs. The t-test holds up well against mild non-normality, especially as samples grow, because the sampling distribution of the mean tends toward normal anyway. It is less forgiving of unequal variances when the group sizes also differ.

Check normality with a histogram or a normal quantile plot, not a significance test. With small groups a formal normality test has too little power to catch real skew and too much fuss over tiny departures. For equal variance, the practical rule from the original texts is simple: if one sample SD is more than about twice the other, do not assume equal variance.

The Welch correction for unequal variances

When the variances are not equal, the pooled SD is the wrong summary, because it averages two spreads that are genuinely different. The Welch correction fixes this. It skips pooling and builds the standard error straight from the two separate variances, then adjusts the degrees of freedom downward with the Welch–Satterthwaite formula.

The Welch degrees of freedom land somewhere between the smaller group's d.f. and the pooled d.f. When the two groups are the same size, the Welch and pooled standard errors are identical and only the d.f. shift slightly, so the choice barely matters. The gap opens up when the sample sizes and the variances both differ. Take a smaller diet group of 8 with a tight SD of 6, against a larger usual-care group of 20 with a loose SD of 14.

n1 <- 8;  m1 <- 132; s1 <- 6
n0 <- 20; m0 <- 140; s0 <- 14

# compute the pooled t and the Welch t, and compare their P-values

# pooled (assumes equal variance)
sp <- sqrt(((n1-1)*s1^2 + (n0-1)*s0^2) / (n1+n0-2))
se_p <- sp * sqrt(1/n1 + 1/n0)
t_p <- (m1-m0)/se_p; df_p <- n1+n0-2
# Welch (allows unequal variance)
v1 <- s1^2/n1; v0 <- s0^2/n0
se_w <- sqrt(v1 + v0)
t_w <- (m1-m0)/se_w
df_w <- (v1+v0)^2 / (v1^2/(n1-1) + v0^2/(n0-1))
round(c(t_pooled=t_p, p_pooled=2*pt(-abs(t_p),df_p),
        t_welch=t_w, df_welch=df_w, p_welch=2*pt(-abs(t_w),df_w)), 3)
# pooled t -1.546, p 0.134  |  Welch t -2.116, df 25.7, p 0.044

from scipy import stats

# ttest_ind_from_stats(mean1, std1, n1, mean2, std2, n2, equal_var=...)

pooled = stats.ttest_ind_from_stats(132, 6, 8, 140, 14, 20, equal_var=True)
welch  = stats.ttest_ind_from_stats(132, 6, 8, 140, 14, 20, equal_var=False)
print("pooled  t=%.3f  p=%.3f" % (pooled.statistic, pooled.pvalue))
print("welch   t=%.3f  p=%.3f" % (welch.statistic,  welch.pvalue))
# pooled  t=-1.546  p=0.134
# welch   t=-2.116  p=0.044

Same data, two conclusions. The pooled test says no clear effect (P = 0.13). The Welch test says there is one (P = 0.04). The pooled test was misled because it forced the large group's wide variance onto the small group. Welch is the honest answer here.

The paired t-test, and when to use it

Sometimes the two measurements are not on two separate groups but on the same unit twice, or on two units deliberately matched. Blood pressure in one patient before and after treatment. A lab assay on a sample split and run on two machines. A case and its age-and-sex-matched control. These are paired data, and treating them as two independent samples throws away the design.

The fix is to collapse each pair into a single difference, then test whether the mean of those differences is zero. This is a one-sample t-test on the differences. It cancels the variation between units, which is often the largest source of noise.

Consider 10 hypertensive patients at a Klang clinic, with SBP measured before and after 8 weeks of a salt-reduction program. The difference is before minus after, so a positive value is a drop in pressure.

The mean difference is mmHg, with SD of the differences . The standard error is , so on 9 degrees of freedom, giving P = 0.003. The 95% CI uses : it is , that is 3.5 to 12.5 mmHg. Clear evidence that the program lowers SBP.

before <- c(150,148,162,138,168,154,158,144,160,166)
after  <- c(142,146,148,140,152,148,148,144,148,152)

t.test(before, after, paired = TRUE)

t.test(before, after, paired = TRUE)
# t = 4, df = 9, p-value = 0.003, mean diff 8, 95% CI 3.48 to 12.52

# what you would WRONGLY get ignoring the pairing:
t.test(before, after, var.equal = TRUE)$statistic   # t = 2.40

import numpy as np
from scipy import stats
before = np.array([150,148,162,138,168,154,158,144,160,166])
after  = np.array([142,146,148,140,152,148,148,144,148,152])

res = stats.ttest_rel(before, after)

d = before - after
n = len(d)
res = stats.ttest_rel(before, after)
se = d.std(ddof=1) / np.sqrt(n)
tcrit = stats.t.ppf(0.975, n-1)
ci = (d.mean() - tcrit*se, d.mean() + tcrit*se)
print("mean diff", d.mean(), "t", round(res.statistic,3), "p", round(res.pvalue,4))
print("95% CI", round(ci[0],2), round(ci[1],2))
# mean diff 8.0 t 4.0 p 0.0031
# 95% CI 3.48 12.52

A picture makes the pairing obvious. Draw a line for each patient from their before value to their after value. Almost every line slopes down, and the few that do not are the weak responders.

before <- c(150,148,162,138,168,154,158,144,160,166)
after  <- c(142,146,148,140,152,148,148,144,148,152)

# one grey line per patient, points at each visit

plot(NA, xlim = c(0.8, 2.2), ylim = range(c(before, after)),
     xaxt = "n", xlab = "", ylab = "Systolic BP (mmHg)",
     main = "Each patient: before vs after")
axis(1, at = c(1, 2), labels = c("Before", "After"))
for (i in seq_along(before)) lines(c(1, 2), c(before[i], after[i]), col = "grey")
points(rep(1, 10), before, pch = 19)
points(rep(2, 10), after,  pch = 19)

import numpy as np
import matplotlib.pyplot as plt
before = np.array([150,148,162,138,168,154,158,144,160,166])
after  = np.array([142,146,148,140,152,148,148,144,148,152])

# one grey line per patient, points at each visit

fig, ax = plt.subplots()
for b, a in zip(before, after):
    ax.plot([1, 2], [b, a], color="grey")
ax.scatter(np.ones(10), before, color="black")
ax.scatter(np.full(10, 2), after, color="black")
ax.set_xticks([1, 2]); ax.set_xticklabels(["Before", "After"])
ax.set_ylabel("Systolic BP (mmHg)")
ax.set_title("Each patient: before vs after")
plt.show()

Reporting the result

A good report leads with the effect size and its interval, then gives the test in brackets. The reader should be able to judge clinical importance without rerunning anything. For the diet trial:

For the paired trial you would write the mean within-patient change, not two separate group means: "Mean SBP fell by 8.0 mmHg after 8 weeks (95% CI 3.5 to 12.5; paired t-test, t = 4.0, d.f. = 9, P = 0.003)." Always name the design in the test, so a reader knows the pairing was respected.

Common mistakes

• Analysing paired data as independent. Before/after, matched case-control, split samples: all paired. Running an independent-samples test wastes the design and usually hides a real effect behind inflated noise.
• Assuming equal variance without looking. If one SD is more than twice the other, the pooled test can mislead, badly so when group sizes also differ. Default to Welch and you rarely go wrong.
• Reading the P-value and skipping the interval. P = 0.04 with a CI of 0.5 to 15.5 mmHg means the effect could be clinically trivial. The interval carries the information a clinician needs; the P-value alone does not.
• Testing skewed data on the raw scale. Viral loads, costs, and enzyme levels are often right-skewed. The means get dragged by outliers. Transform first (usually logs), test, then back-transform.
• Confusing the two-sample t-test with a paired test because both compare "two means". The question is whether the two columns come from the same units. Same units means paired.

Tips

• Decide independent versus paired from the study design, before you open the data. The design, not the spreadsheet layout, tells you which test fits.
• Let Welch be your default for independent groups. It matches the pooled test when variances are equal and protects you when they are not.
• Plot before you test. A histogram per group checks normality; a before-after line plot shows whether pairing is doing real work.
• Report the difference and its 95% CI first, the P-value second. Editors and clinicians read effect sizes, not just stars.
• For heavily skewed outcomes that logs do not fix, reach for a rank-based test (Mann-Whitney for independent, Wilcoxon signed-rank for paired) instead of forcing a t-test.