Means, standard deviations and standard errors

This lesson is about summarising a column of measurements with two numbers: one for where the values sit, one for how far they spread. Get those two right and you can describe a sample of dengue platelet counts, newborn weights, or blood pressures in a single line of a results table. It then adds the number that trips up most clinicians: the standard error, which is not the spread of the data at all.

You should already be comfortable reading a frequency distribution. Here you reduce one to a centre and a spread, then ask a different question: how precisely does this sample pin down the mean of the wider population it came from?

What is the centre of the data? Mean, median and mode

The mean is the sum of the values divided by how many there are. It is the everyday average, written and spoken "x bar".

Here stands for each value, means "add them up", and is the number of observations. The mean uses every value, so it feeds naturally into the rest of statistics.

The median is the middle value once you sort the data from smallest to largest. With an odd count it is the single middle observation. With an even count you average the two middle ones. Half the values fall below the median and half above. Because it ignores how extreme the tails are, the median barely moves when one patient has a freak result.

The mode is the value that occurs most often. For counts it is easy to read: in a clinic where parity (number of previous pregnancies) runs 0, 1, 1, 2, 2, 2, 3, the mode is 2. For continuous measurements where every reading differs, there may be no useful mode at all.

Which one you report depends on shape. For roughly symmetric data such as blood pressure or haemoglobin, quote the mean. For skewed data such as income, hospital length of stay, or triceps skinfold, a few large values drag the mean upward and the median better represents a typical person. Skinfold and income are right-skewed, so their mean sits above their median.

For strongly right-skewed measurements such as antibody titres or parasite counts, even the arithmetic mean of the raw values misleads, because it is pulled by a long upper tail. The right average there is often the geometric mean, which averages the values on a log scale and then transforms back. You will meet it properly in the transformations module; for now, note that the arithmetic mean is the wrong summary for heavily skewed data, and a geometric mean or median fits better.

What is spread, and how do we measure it?

The simplest spread is the range, the largest value minus the smallest. It is easy but fragile: it rests on the two most extreme readings and tends to grow just because you measured more people. The interquartile range, the gap between the upper and lower quartiles, covers the middle 50% and ignores the tails, which makes it steadier.

For most analysis the working measure of spread is the variance. It is built from how far each value sits from the mean, the deviation . You cannot just average those deviations, because the positive and negative ones cancel to zero by construction. Squaring them removes the sign, so we average the squared deviations instead.

What are degrees of freedom, and why divide by n minus 1?

Notice the denominator is , not . That count is the degrees of freedom. The reason is that the deviations from the sample mean must add to zero, so once you know any of them, the last one is fixed. Only of the deviations are free to vary. Dividing by rather than corrects a bias and gives a better estimate of the true variance in the population. With small samples the difference is large: at , dividing by 4 instead of 5 raises the variance by a quarter.

What does the standard deviation tell you?

Variance has an awkward unit. If you measure birth weight in kilograms, the variance is in kilograms squared, which means nothing to a clinician. Take the square root and you are back in the original units. That square root is the standard deviation, written .

The standard deviation, or SD, is the headline measure of how spread out the individual values are. There is a useful rule of thumb for data that follow a bell shape. About 70% of observations fall within one SD of the mean, and about 95% fall within two SDs. So if adult diastolic blood pressure has a mean of 78 mmHg and an SD of 9.4 mmHg, roughly 95% of people sit between 78 minus 18.8 and 78 plus 18.8, that is about 59 to 97 mmHg.

Worked example 1: blood pressure of five patients

A nurse records the diastolic blood pressure (mmHg) of five patients in a Klang Valley clinic: 70, 81, 74, 88, 77.

1. Mean: mmHg.
2. Median: sorted the values are 70, 74, 77, 81, 88, so the middle value is 77 mmHg.
3. Mode: every reading is different, so there is no mode.
4. Spread: subtract the mean from each value, square, and add up the squares.

The deviations sum to zero, as they must. The squared deviations sum to 190. So the variance is , and the standard deviation is mmHg. The SD is about 6.9 mmHg, a believable spread for five patients.

bp <- c(70, 81, 74, 88, 77)

m <- # the mean
s <- # the standard deviation

m   <- mean(bp)
med <- median(bp)
v   <- var(bp)              # var() divides by n-1
s   <- sd(bp)
se  <- s / sqrt(length(bp))
round(c(mean = m, median = med, var = v, sd = s, se = se), 3)
# mean 78, median 77, var 47.5, sd 6.892, se 3.082

import numpy as np
bp = np.array([70, 81, 74, 88, 77])

m = # the mean
s = # the standard deviation

m   = bp.mean()
med = np.median(bp)
v   = bp.var(ddof=1)        # ddof=1 -> divide by n-1
s   = bp.std(ddof=1)
se  = s / np.sqrt(bp.size)
print(m, med, round(v, 1), round(s, 3), round(se, 3))
# 78.0 77.0 47.5 6.892 3.082

You can see the spread rather than just compute it. A histogram shows the shape and a boxplot shows the quartiles and any outliers.

set.seed(1)
bp <- round(rnorm(200, mean = 78, sd = 9.4))

# plot a histogram and a boxplot side by side

par(mfrow = c(1, 2))
hist(bp, col = "grey", main = "Diastolic BP", xlab = "mmHg")
boxplot(bp, main = "Spread", ylab = "mmHg")
round(c(mean = mean(bp), sd = sd(bp)), 2)
# mean ~ 78, sd ~ 9.x

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(1)
bp = rng.normal(78, 9.4, 200)

# plot a histogram and a boxplot side by side

fig, ax = plt.subplots(1, 2, figsize=(8, 3))
ax[0].hist(bp, color="grey")
ax[0].set_title("Diastolic BP"); ax[0].set_xlabel("mmHg")
ax[1].boxplot(bp)
ax[1].set_title("Spread"); ax[1].set_ylabel("mmHg")
plt.tight_layout(); plt.show()
print(round(bp.mean(), 1), round(bp.std(ddof=1), 1))

What is the coefficient of variation?

The SD carries the units of the data, so a weight SD in kilograms and a height SD in centimetres cannot be compared directly. The coefficient of variation (CV) fixes that. It expresses the SD as a fraction, or percentage, of the mean.

The CV is unit-free, so it measures relative spread: how big the variation is compared with the size of the thing being measured. That makes it useful when you compare variability across measurements on different scales, or across groups with widely different means. The same weights measured in pounds instead of kilograms give a different SD but the same CV, because the unit cancels out.

weight_kg <- c(2.9, 3.4, 2.6, 3.8, 3.1)

# cv = sd / mean * 100, in kg and again in pounds

weight_lb <- weight_kg * 2.20462
cv_kg <- sd(weight_kg) / mean(weight_kg) * 100
cv_lb <- sd(weight_lb) / mean(weight_lb) * 100
round(c(sd_kg = sd(weight_kg), sd_lb = sd(weight_lb),
        cv_kg = cv_kg, cv_lb = cv_lb), 3)
# the SD changes with the unit, the CV does not

import numpy as np
weight_kg = np.array([2.9, 3.4, 2.6, 3.8, 3.1])

# cv = sd / mean * 100, in kg and again in pounds

weight_lb = weight_kg * 2.20462
cv_kg = weight_kg.std(ddof=1) / weight_kg.mean() * 100
cv_lb = weight_lb.std(ddof=1) / weight_lb.mean() * 100
print(round(weight_kg.std(ddof=1), 3), round(weight_lb.std(ddof=1), 3))
print(round(cv_kg, 3), round(cv_lb, 3))
# the SD changes with the unit, the CV does not

What happens when you change the units?

You often need to shift or rescale a column: subtract a baseline, convert grams to kilograms, or turn Fahrenheit into centigrade. Two rules cover every case. Adding or subtracting a constant shifts the mean by that amount but leaves the SD unchanged, because sliding every value along the axis does not change how spread out they are. Multiplying or dividing by a constant scales both the mean and the SD by the same factor.

bp <- c(70, 81, 74, 88, 77)

# shift: bp + 10   scale: bp * 2
# compare mean and sd before and after

shifted <- bp + 10
scaled  <- bp * 2
round(c(mean = mean(bp),      sd = sd(bp),
        mean_shift = mean(shifted), sd_shift = sd(shifted),
        mean_scale = mean(scaled),  sd_scale = sd(scaled)), 3)
# mean rises by 10, sd unchanged; doubling doubles both mean and sd

import numpy as np
bp = np.array([70, 81, 74, 88, 77])

# shift: bp + 10   scale: bp * 2
# compare mean and sd before and after

shifted = bp + 10
scaled  = bp * 2
print(bp.mean(), round(bp.std(ddof=1), 3))
print(shifted.mean(), round(shifted.std(ddof=1), 3))
print(scaled.mean(), round(scaled.std(ddof=1), 3))
# mean rises by 10, sd unchanged; doubling doubles both mean and sd

How do you find the mean and SD from a frequency distribution?

Sometimes you do not have the raw list, only a table of counts: how many people had each value. You can still get the mean and SD without writing the value out once per person. The trick is to weight each value by its frequency , the number of times it occurs.

The table below records the number of previous pregnancies (parity) for 100 women at an antenatal clinic.

The total number of pregnancies is , and the mean is that total divided by , the number of women.

Here , so the mean is pregnancies. Using gives an SD of about 1.13.

x <- 0:4
f <- c(18, 27, 31, 19, 5)

# n = sum(f); mean = sum(f*x)/n
# var = (sum(f*x^2) - sum(f*x)^2/n) / (n-1)

n      <- sum(f)
mean_x <- sum(f * x) / n
var_x  <- (sum(f * x^2) - sum(f * x)^2 / n) / (n - 1)
sd_x   <- sqrt(var_x)
round(c(n = n, mean = mean_x, sd = sd_x), 3)
# n 100, mean 1.66, sd 1.13
barplot(f, names.arg = x, col = "grey",
        xlab = "previous pregnancies", ylab = "number of women",
        main = "Parity of 100 women")

import numpy as np
import matplotlib.pyplot as plt
x = np.array([0, 1, 2, 3, 4])
f = np.array([18, 27, 31, 19, 5])

# n = f.sum(); mean = (f*x).sum()/n
# var = ((f*x**2).sum() - (f*x).sum()**2/n) / (n-1)

n      = f.sum()
mean_x = (f * x).sum() / n
var_x  = ((f * x**2).sum() - (f * x).sum()**2 / n) / (n - 1)
sd_x   = np.sqrt(var_x)
print(n, round(mean_x, 3), round(sd_x, 3))
# 100 1.66 1.13
plt.bar(x, f, color="grey")
plt.xlabel("previous pregnancies"); plt.ylabel("number of women")
plt.title("Parity of 100 women"); plt.show()

What is the standard error?

The sample is rarely interesting for its own sake. You measure 64 pregnant women to learn about all pregnant women in the district. The sample mean estimates the population mean, written (mu). The sample SD estimates the population SD, written (sigma).

Your sample mean will not land exactly on . A different sample of 64 women would give a slightly different mean. That wobble from one sample to the next is sampling variation. Imagine taking many independent samples of the same size and writing down each sample mean. Those means form their own distribution, the sampling distribution. Its centre is the population mean, and its spread is the standard error of the mean, written SE.

The SE measures how precisely the sample mean estimates the population mean. Two things shrink it: less variation in the data (smaller ) and a larger sample (bigger ). Because sits under a square root, you must quadruple the sample to halve the standard error. Going from 64 to 256 women cuts the SE in half, not to a quarter.

The key idea is easier to see than to read. The picture below puts the spread of individual people next to the spread of the sample mean. Individuals scatter widely, and that width is the SD. The average of a sample sits in a much narrower band, and that width is the SE. The band tightens further as the sample grows from 16 to 64.

set.seed(7)
pop <- rnorm(100000, mean = 78, sd = 9.4)

# density of individuals vs sample means for n = 16 and n = 64

means16 <- replicate(3000, mean(sample(pop, 16)))
means64 <- replicate(3000, mean(sample(pop, 64)))
plot(density(pop), lwd = 2, xlim = c(60, 96), ylim = c(0, 0.4),
     main = "Individuals spread wide; means cluster tight",
     xlab = "diastolic BP (mmHg)")
lines(density(means16), col = "blue", lwd = 2)
lines(density(means64), col = "red",  lwd = 2)
abline(v = 78, lty = 2)
legend("topright", c("individuals (SD)", "mean of 16 (SE)", "mean of 64 (SE)"),
       col = c("black", "blue", "red"), lwd = 2, bty = "n")
round(c(sd_indiv = sd(pop), se_16 = 9.4 / sqrt(16), se_64 = 9.4 / sqrt(64)), 3)
# sd_indiv ~ 9.4, se_16 ~ 2.35, se_64 ~ 1.175

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(7)
pop = rng.normal(78, 9.4, 100000)

# histogram of individuals vs sample means for n = 16 and n = 64

means16 = np.array([rng.choice(pop, 16).mean() for _ in range(3000)])
means64 = np.array([rng.choice(pop, 64).mean() for _ in range(3000)])
plt.hist(pop,     bins=60, density=True, alpha=0.4, label="individuals (SD)")
plt.hist(means16, bins=40, density=True, alpha=0.5, label="mean of 16 (SE)")
plt.hist(means64, bins=40, density=True, alpha=0.5, label="mean of 64 (SE)")
plt.axvline(78, ls="--", color="k")
plt.xlim(60, 96); plt.legend(); plt.xlabel("diastolic BP (mmHg)")
plt.title("Individuals spread wide; means cluster tight")
plt.show()
print(round(pop.std(ddof=1), 3),
      round(9.4 / np.sqrt(16), 3), round(9.4 / np.sqrt(64), 3))
# ~9.4  ~2.35  ~1.175

Worked example 2: the same SD, two different jobs

Stay with those 64 women: mean haemoglobin 11.2 g/dL, SD 1.6 g/dL, SE 0.2 g/dL. The SD and the SE answer two different questions about this one sample.

1. How much do individual women differ? Use the SD. About 95% of women fall within two SDs of the mean: , that is 8.0 to 14.4 g/dL. This is a wide clinical range, and it should be, because women genuinely differ.
2. How well do we know the average? Use the SE. The population mean is likely within two SEs of our sample mean: , that is 10.8 to 11.6 g/dL. This is a narrow band, because averaging 64 women cancels out a lot of the individual noise.
3. Now quadruple the sample. With 256 women and the same SD, g/dL. The SD is unchanged at 1.6, because women have not become more alike. Only the precision of the mean improved.

That third point is the heart of the matter. Collecting more data does not narrow the spread of the population. It narrows your uncertainty about the average. The SD describes people; the SE describes your estimate.

set.seed(42)
pop <- rnorm(5000, mean = 78, sd = 9.4)   # the population

# take 2000 samples of size 10 and of size 40
# compare sd of the sample means with 9.4 / sqrt(n)

means10 <- replicate(2000, mean(sample(pop, 10)))
means40 <- replicate(2000, mean(sample(pop, 40)))
round(c(se_theory_10 = 9.4 / sqrt(10), sd_means_10 = sd(means10),
        se_theory_40 = 9.4 / sqrt(40), sd_means_40 = sd(means40)), 3)
# se_theory_10 ~ 2.973, sd_means_10 ~ 2.9x
# se_theory_40 ~ 1.486, sd_means_40 ~ 1.4x
hist(means10, col = rgb(0, 0, 1, 0.4), breaks = 20, xlim = c(70, 86),
     main = "Sampling distribution of the mean", xlab = "mmHg")
hist(means40, col = rgb(1, 0, 0, 0.4), breaks = 20, add = TRUE)

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(42)
pop = rng.normal(78, 9.4, 5000)   # the population

# take 2000 samples of size 10 and of size 40
# compare std of the sample means with 9.4 / sqrt(n)

means10 = np.array([rng.choice(pop, 10).mean() for _ in range(2000)])
means40 = np.array([rng.choice(pop, 40).mean() for _ in range(2000)])
print(round(9.4 / np.sqrt(10), 3), round(means10.std(ddof=1), 3))
print(round(9.4 / np.sqrt(40), 3), round(means40.std(ddof=1), 3))
# ~2.973  ~2.9x
# ~1.486  ~1.4x
plt.hist(means10, bins=20, alpha=0.4, label="n = 10")
plt.hist(means40, bins=20, alpha=0.4, label="n = 40")
plt.legend(); plt.xlabel("mmHg")
plt.title("Sampling distribution of the mean")
plt.show()

What is the difference between SD and SE?

This is the confusion that fills published tables with the wrong number. The SD and the SE are linked by , so the SE is always the smaller of the two. Reporting the SE when you meant the SD makes your data look far tighter than it is.

The rule for which to report follows from what you want to communicate.

• Quote the SD when you are describing variability between people: reference ranges, how patients differ, the spread of birth weights or blood pressures in your sample. The SD answers "how different are individuals?"
• Quote the SE (or better, a confidence interval built from it) when you are describing how well you have estimated a mean. Typical uses are the precision of a treatment effect, the mean reduction in fasting glucose, or any result you want to generalise to the population. The SE answers "how sure am I about the average?"

A confidence interval is just the sample mean reaching out about two standard errors on each side, which captures the population mean about 95% of the time. You build it formally in a later chapter once the normal distribution is in hand. For now, hold the distinction: SD for spread, SE for precision.

Common mistakes

• Reporting the SE when you meant the SD. The SE is smaller by a factor of , so this makes the data look much tighter than it is. With 100 patients the SE is one tenth of the SD. Always label which one a bracket holds.
• Dividing by instead of . This is the population variance, which understates the spread of a sample. In NumPy, set ddof=1; the default divides by .
• Quoting a mean for skewed data. For hospital length of stay or income, a few large values pull the mean above the typical case. Report the median, and consider a geometric mean for right-skewed data.
• Thinking a bigger sample reduces the SD. It does not. More data sharpens the estimate of the mean (smaller SE) but leaves the spread of individuals (the SD) about where it was. The population does not become more uniform because you measured more of it.
• Comparing SDs across different units. An SD in kilograms and an SD in pounds are not comparable. Use the coefficient of variation when you want relative spread that does not depend on the unit.
• Forgetting units on the variance. Variance is in squared units (g/dL squared), which is uninterpretable to a clinician. Take the square root and report the SD in the original units.

Tips

• Before computing anything, decide whether you are describing people or estimating an average. That choice tells you SD or SE before you touch a formula.
• For roughly bell-shaped data, sanity-check the SD with the rule of thumb: mean plus or minus two SDs should cover about 95% of your values. If that range is impossible (a negative blood pressure, a birth weight above 6 kg), recheck the calculation.
• Sketch the data first. A histogram or boxplot tells you whether the mean and SD are even the right summaries, or whether skew means you should switch to median and interquartile range.
• When you want a more precise mean, remember the square root. Halving the SE means roughly four times the sample, which has a real budget cost in a field study.
• State the sample size next to any mean. Without a reader cannot move between SD and SE or judge how much to trust the estimate.