Using P-values and confidence intervals

A statistical analysis hands you two numbers: a confidence interval and a P-value. Most reports lead with the P-value, decide "significant or not", and stop there. That habit throws away most of what the data are telling you. This lesson shows what a P-value really measures, where it misleads, and why you read it next to the confidence interval rather than instead of it.

The running example is a trial run at a Klang Valley clinic. A new antihypertensive is compared against standard care in adults with high blood pressure. The outcome is the drop in systolic blood pressure (SBP) after twelve weeks, measured in millimetres of mercury (mmHg). We compare the mean drop in the two groups, and the whole inference rests on one estimate and its standard error.

Null and alternative hypotheses

Statistics looks for evidence against a sharp, specific claim called the null hypothesis: that there is no difference between the groups, or no association between exposure and outcome, in the population. For our trial the null hypothesis is that the new drug and standard care lower SBP by the same average amount, so the true difference in mean drop is exactly zero.

The alternative hypothesis is what you are left with if the null is wrong. By default it is two-sided: the true difference is not zero, in either direction. The drug could be better or worse than standard care. We collect data, then ask how strongly they argue against the null.

What a P-value is, and what it is not

The P-value is the probability of getting a difference at least as large as the one you observed, if the null hypothesis were true. It runs from 0 to 1. A small P-value means your data would be surprising under "no effect", so the data argue against the null. A large P-value means your data sit comfortably with "no effect", so they give you no reason to abandon it.

The smaller the P-value, the stronger the evidence against the null. There is no magic line. A value of 0.04 and a value of 0.06 carry almost the same weight of evidence, even though one sits on each side of the old 0.05 convention. Report the exact P-value, not just whether it cleared a threshold.

Now the harder half. A P-value is widely misread, so fix in your mind what it is not:

• It is not the probability that the null hypothesis is true. The calculation already assumes the null is true, so it cannot also tell you the chance of that.
• It is not the probability that your result is "due to chance".
• It is not a measure of effect size. A tiny, clinically useless difference can have a tiny P-value in a large study.
• A large P-value does not prove the null. It often just means the study was too small to detect a real effect. Absence of evidence is not evidence of absence.

The test statistic, the P-value, and the confidence interval

Almost every test you will meet has the same skeleton. You take your estimate (a mean, a difference in means, a log risk ratio), and you divide it by its standard error. That ratio is the test statistic, often called z when the sample is large enough to use the normal distribution.

The test statistic counts how many standard errors the estimate sits away from the null value of zero. The further from zero, the smaller the P-value. For a two-sided test we read off both tails of the standard normal distribution.

The confidence interval is built from the same two ingredients. The 95% interval stretches 1.96 standard errors either side of the estimate, because 1.96 is the point that leaves 2.5% of the normal curve in each tail.

Worked example 1: the blood-pressure trial

Suppose the trial finds that the new drug lowers SBP by 6 mmHg more than standard care, with a standard error of 2 mmHg. Work it through.

1. Test statistic. . The estimate is three standard errors from zero.
2. Two-sided P-value. . Strong evidence against the null.
3. 95% confidence interval. mmHg.

Read both together. The P-value of 0.0027 says the data conflict sharply with "no difference". The interval says the true extra drop is plausibly anywhere from about 2 to 10 mmHg. A 2 mmHg edge would barely matter to a patient; a 10 mmHg edge would be a meaningful gain. The single estimate of 6 hides that spread, and the interval is what puts it back.

delta <- 6     # mmHg: extra SBP drop on the new drug
se    <- 2     # standard error of the difference

z <- delta / se
# fill in p_two, p_one, and the 95% CI

z     <- delta / se                    # 3.0
p_two <- 2 * (1 - pnorm(abs(z)))       # 0.0027
p_one <- 1 - pnorm(abs(z))             # 0.00135
ci    <- delta + c(-1, 1) * 1.96 * se  # 2.08 to 9.92
c(z = z, p_two = p_two, p_one = p_one, lower = ci[1], upper = ci[2])

import numpy as np
from scipy.stats import norm
delta, se = 6.0, 2.0   # mmHg and its standard error

z = delta / se
# fill in p_two, p_one, and the 95% CI

z     = delta / se                       # 3.0
p_two = 2 * (1 - norm.cdf(abs(z)))       # 0.0027
p_one = 1 - norm.cdf(abs(z))             # 0.00135
ci    = delta + np.array([-1, 1]) * 1.96 * se   # 2.08 to 9.92
print(dict(z=z, p_two=p_two, p_one=p_one, lower=ci[0], upper=ci[1]))

One-sided versus two-sided tests

A two-sided test counts surprising results in both directions, so it doubles the single-tail probability. In the example the two-sided P-value is 0.0027 and the one-sided value is half of that, 0.00135. A one-sided test only counts evidence in the direction you picked in advance.

Use two-sided tests by default. A one-sided test is only honest when a difference in the other direction is genuinely impossible or would be acted on exactly like no difference, and you committed to that before seeing the data. Switching to one-sided after the fact, just to push a P-value under 0.05, is a way to fool yourself. Most journals and reviewers expect two-sided P-values unless you justify otherwise.

Type I and type II errors, and power

Treating a P-value as a yes/no decision exposes you to two kinds of mistake. A type I error is a false positive: you reject the null when it is actually true. Its probability is the significance level, , usually set at 0.05. So even when a treatment truly does nothing, about 1 in 20 such tests will cross the 0.05 line by chance alone.

A type II error is a false negative: the null is false, a real effect exists, but your study fails to detect it. Its probability is . The power of a study is , the chance of detecting an effect of a given size when it is really there.

Power rises with the true effect size and with the sample size, because both push the test statistic away from zero. For a two-sided test at level with a known standard error, the power to detect a true difference is approximately:

For our trial, if the true difference were 6 mmHg with a standard error of 2, the power is . The study would catch an effect that size about 85% of the time, and miss it 15% of the time. Underpowered studies are the usual reason a real and useful effect comes back "not significant".

zc <- qnorm(0.975)   # 1.96, the two-sided 5% critical value
power_fn <- function(delta, se) pnorm(delta/se - zc) + pnorm(-delta/se - zc)

power_fn(6, 2)   # power for a true 6 mmHg difference, se = 2

power_fn(6, 2)   # about 0.851

d <- seq(0, 10, by = 0.1)
plot(d, power_fn(d, 2), type = "l", lwd = 2,
     xlab = "True difference (mmHg)", ylab = "Power",
     main = "Power of a two-sided 5% test (se = 2)")
abline(h = 0.8, lty = 2)

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
zc = norm.ppf(0.975)   # 1.96

def power_fn(delta, se):
    return norm.cdf(delta/se - zc) + norm.cdf(-delta/se - zc)
print(power_fn(6, 2))   # power for a true 6 mmHg difference

def power_fn(delta, se):
    return norm.cdf(delta/se - zc) + norm.cdf(-delta/se - zc)
print(power_fn(6, 2))   # about 0.851

d = np.arange(0, 10, 0.1)
plt.plot(d, power_fn(d, 2), lw=2)
plt.axhline(0.8, ls="--", color="grey")
plt.xlabel("True difference (mmHg)")
plt.ylabel("Power")
plt.title("Power of a two-sided 5% test (se = 2)")
plt.show()

The link between a confidence interval and a hypothesis test

The two outputs are tied together by arithmetic, because both are built from the estimate and its standard error. A two-sided test at the 5% level rejects the null exactly when the 95% confidence interval excludes the null value. They cannot disagree.

• If the 95% CI excludes the null value (zero for a difference), then .
• If the 95% CI includes the null value, then .
• The same pairing holds at other levels: a 99% CI that excludes the null matches .

In example 1 the 95% interval (2.08, 9.92) sits entirely above zero, and sure enough P = 0.0027 is below 0.05. The agreement is guaranteed, not a coincidence. This is why you do not need both a P-value and a "does the interval cross zero" check to decide significance. They are the same decision.

Statistical significance is not clinical importance

This is the point that separates a careful analyst from a P-value-counter. "Statistically significant" means the data are unlikely under the null. "Clinically important" means the effect is large enough to change a patient's care. The two come apart in both directions, and the standard error is the reason. Because the standard error shrinks as the sample grows, a big enough study turns even a trivial difference into a small P-value, while a small study can miss a large and useful one.

Compare two hypertension studies of the same drug. Both report the extra SBP drop versus standard care. Assume a 5 mmHg difference is clinically worthwhile and anything under about 2 mmHg is not.

Study A points at a worthwhile 5 mmHg effect but is "not significant" (P = 0.21), because its interval is so wide it runs from a small harm to a large benefit. Calling this a negative result and dropping the drug would be a mistake; the data are simply inconclusive. Study B is highly "significant" (P = 0.0027), yet its interval, 0.5 to 2.5 mmHg, rules out anything a patient would notice. The large sample detected a real but trivial effect. The P-values give the opposite ranking to the clinical message. Only by reading the intervals do you see that A might matter and B does not.

est <- c(5.0, 1.5)    # A (small), B (large)
se  <- c(4.0, 0.5)

z   <- est / se
p   <- 2 * (1 - pnorm(abs(z)))
low  <- est - 1.96 * se
high <- est + 1.96 * se
cbind(est, se, z, p, low, high)

z    <- est / se
p    <- 2 * (1 - pnorm(abs(z)))
low  <- est - 1.96 * se
high <- est + 1.96 * se
print(cbind(est, se, z, p, low, high))

plot(est, 1:2, xlim = c(-4, 13), ylim = c(0.5, 2.5),
     pch = 19, yaxt = "n", ylab = "",
     xlab = "Extra SBP drop vs standard care (mmHg)",
     main = "Significant is not the same as important")
segments(low, 1:2, high, 1:2, lwd = 2)
abline(v = 0, lty = 2)
axis(2, at = 1:2, labels = c("A (small)", "B (large)"))

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
est = np.array([5.0, 1.5])   # A (small), B (large)
se  = np.array([4.0, 0.5])

z    = est / se
p    = 2 * (1 - norm.cdf(np.abs(z)))
low  = est - 1.96 * se
high = est + 1.96 * se
print(np.c_[est, se, z, p, low, high])

z    = est / se
p    = 2 * (1 - norm.cdf(np.abs(z)))
low  = est - 1.96 * se
high = est + 1.96 * se
print(np.c_[est, se, z, p, low, high])

y = [1, 2]
plt.errorbar(est, y, xerr=1.96*se, fmt="o", capsize=4)
plt.axvline(0, ls="--", color="grey")
plt.yticks(y, ["A (small)", "B (large)"])
plt.xlabel("Extra SBP drop vs standard care (mmHg)")
plt.title("Significant is not the same as important")
plt.show()

Common mistakes

• Reading P as the probability the null is true. "P = 0.03 means a 3% chance the drug does nothing" is wrong. The P-value assumes the null is true, then measures how unusual the data are under it.
• Treating "not significant" as "no effect". A large P-value in a small study usually means low power, not a proven null. Absence of evidence is not evidence of absence; check the interval.
• Treating "significant" as "important". In a large study a trivial difference earns a small P-value. Always report the effect size and its interval next to the P-value.
• Switching to a one-sided test to clear 0.05. Halving a P-value by choosing the tail after seeing the data is not a valid test.
• Worshipping the 0.05 line. P = 0.049 and P = 0.051 carry the same weight of evidence. The threshold is an arbitrary convention, not a law of nature.
• Judging a confidence interval only by whether it crosses zero. That just repeats the P-value. The useful information is the range of effect sizes it rules in and out.

Tips

• Report the exact P-value with the 95% CI, and interpret both. The pair "1.5 mmHg (95% CI 0.5 to 2.5, P = 0.003)" tells the whole story; "P < 0.05" tells almost none of it.
• Decide your clinically important difference before the analysis. Then you can say at a glance whether an interval that excludes zero also excludes the values that would actually matter.
• Let the interval do the talking in small studies. When a result is "not significant", quote the upper limit: "consistent with a benefit as large as 12.8 mmHg" is honest where "no effect" is not.
• Plan power before you recruit. Knowing the effect size you care about and the standard error you expect tells you the sample size you need. It is far cheaper to fix an underpowered design on paper than after the data are in.
• Default to two-sided tests. Use one-sided only with a protocol-stated justification, and expect reviewers to ask why.