Measurement error and its consequences

A nurse cuffs your arm once at a busy klinik kesihatan and writes 148/96 on the chart. That single number is not your blood pressure. Your true pressure is the long-run average across many days and many readings, and the value on the chart sits some distance from it, nudged up by traffic, a rushed morning, or the wrong cuff size. Every variable you measure carries this gap between what you recorded and what is true. This lesson is about where the gap comes from, what it does to your estimates, and why no analysis can recover more than the measurement put in.

You have spent this whole course turning data into estimates and intervals. All of that assumed the numbers were what they claimed to be. Here we drop that assumption and trace what happens.

Two kinds of error: random and systematic

Measurement error splits into two parts that behave in different ways. Random error scatters readings around the truth with no fixed direction. Measure one patient ten times and the values land above and below the real value, averaging out to roughly nothing. Systematic error, also called bias, pushes every reading the same way. A cuff that always reads 8 mmHg high adds 8 to everyone, and repeating the measurement never washes it out.

These map onto two words used in every lab. Precision is the absence of random error: tight, repeatable readings. Validity, also called accuracy, is the absence of systematic error: readings centred on the true value. A weighing scale can be precise but invalid (always 2 kg heavy, but consistent) or valid but imprecise (right on average, but scattered widely). The two problems need different fixes. You beat random error by averaging more measurements. You beat systematic error by recalibrating the instrument, because averaging a biased reading just gives you a precise wrong answer.

Reliability and the intraclass correlation

To put a number on reliability, take repeat measurements on the same people and split the spread of the readings into two sources. Part of the variation is real differences between people. Part is measurement error, the wobble you would see remeasuring one unchanged person. Reliability is the share that is real:

Here is the between-subject variance (the true signal) and is the measurement-error variance (the noise). This ratio is the intraclass correlation coefficient (ICC). It runs from 0 to 1. An ICC of 1 means no measurement error, so every observed difference is a real difference. An ICC of 0 means all you are seeing is noise. Hold on to this number, because it controls almost everything that follows.

Random error in a continuous exposure: regression dilution

Suppose true blood pressure does raise some outcome, and the true relationship has slope 0.50. You cannot measure true pressure, so you record a single noisy reading instead. What happens to the slope you estimate?

It shrinks toward zero. Random error in an exposure variable flattens the regression line, so the association looks weaker than it is. This is regression dilution, and the bias is always toward the null of no effect. For linear regression the rule is exact:

The intuition is direct. Noise spreads the exposure values out horizontally without changing the outcome, so the cloud of points gets wider but no taller. A wider, equally tall cloud has a gentler slope. The worse the measurement (the lower the ICC), the flatter the line. Watch it happen by adding noise to a clean simulated exposure and refitting.

set.seed(2024)
n <- 5000
x_true <- rnorm(n, 130, 10)               # true systolic BP
y      <- 0.50 * x_true + rnorm(n, 0, 8)  # outcome, true slope 0.50

x_small <- # true BP plus modest noise (sd 5)
x_big   <- # true BP plus heavy noise (sd 15)

x_small <- x_true + rnorm(n, 0, 5)    # ICC ~ 100/125 = 0.80
x_big   <- x_true + rnorm(n, 0, 15)   # ICC ~ 100/325 = 0.31
b_true  <- coef(lm(y ~ x_true))[2]
b_small <- coef(lm(y ~ x_small))[2]
b_big   <- coef(lm(y ~ x_big))[2]
round(c(true = b_true, small_noise = b_small, big_noise = b_big), 3)
#       true small_noise   big_noise
#      ~0.50      ~0.40       ~0.15

import numpy as np
import statsmodels.api as sm
rng = np.random.default_rng(2024)
n = 5000
x_true = rng.normal(130, 10, n)               # true systolic BP
y      = 0.50 * x_true + rng.normal(0, 8, n)  # outcome, true slope 0.50

x_small = # true BP plus modest noise (sd 5)
x_big   = # true BP plus heavy noise (sd 15)

x_small = x_true + rng.normal(0, 5, n)    # ICC ~ 0.80
x_big   = x_true + rng.normal(0, 15, n)   # ICC ~ 0.31
def slope(x, yy):
    return sm.OLS(yy, sm.add_constant(x)).fit().params[1]
print(round(slope(x_true, y), 3),
      round(slope(x_small, y), 3),
      round(slope(x_big, y), 3))
# 0.491 0.393 0.16

The slope falls from about 0.49 to 0.39 to 0.16 as the noise grows, exactly as predicts. Nothing about the true relationship changed. Only the quality of the exposure measurement did, and it dragged the estimate toward zero. A real cohort study that measured pressure once would under-state the true effect for the same reason.

Regression to the mean: an effect that is not there

Random error plays a second trick when you measure the same thing twice. Take 50 people, record diastolic pressure, then record it again six months later with no treatment in between. Plot the change against the starting value and you will often see a clean downward slope: high starters fell, low starters rose. It looks like the body self-corrects, or that high readers respond to something. Usually it is an artefact called regression to the mean.

The reason is the random error you already met. Someone with an unusually high first reading probably had a true value that was high plus a positive measurement error that day. The error does not repeat, so their second reading drifts back down toward their own average. High first readings tend to fall, low first readings tend to rise, with no real change at all. When the true change is unrelated to the starting value, the apparent slope of change against baseline is , a negative number whenever there is any measurement error.

Binary classification: sensitivity, specificity, and misclassification

Now switch from a continuous reading to a yes-or-no label, the kind a screening test produces. A test sorts people into positive and negative, and a gold standard says who is truly positive. Two numbers describe how well it sorts. Sensitivity is the proportion of true positives the test catches. Specificity is the proportion of true negatives it correctly clears.

Sensitivity and specificity are properties of the test, not of the population, so they hold whether you screen a high-risk clinic or a healthy crowd. When a test is imperfect, it labels some people wrong, and that misclassification distorts any association you compute from the labels. The key result: when the error is non-differential (the misclassification rate is the same in cases and non-cases, so the error does not depend on the outcome), the odds ratio is pulled toward 1. As with regression dilution, sloppy classification hides real effects rather than inventing them.

TP <- 96; FN <- 24      # 120 true diabetics
FP <- 44; TN <- 836     # 880 true non-diabetics

sens <- # TP / (TP + FN)
spec <- # TN / (TN + FP)

sens <- TP / (TP + FN)   # 0.80
spec <- TN / (TN + FP)   # 0.95
round(c(sensitivity = sens, specificity = spec), 3)

# now: misclassification dilutes an odds ratio toward 1
set.seed(7)
N <- 20000
expo_true <- rbinom(N, 1, 0.3)
p         <- 1 / (1 + exp(-(-2 + log(3) * expo_true)))   # true OR = 3
disease   <- rbinom(N, 1, p)
or_fun <- function(e, d) {
  a  <- sum(e == 1 & d == 1); b  <- sum(e == 1 & d == 0)
  cc <- sum(e == 0 & d == 1); dd <- sum(e == 0 & d == 0)
  (a * dd) / (b * cc)
}
expo_meas <- expo_true
expo_meas[expo_true == 1 & runif(N) > 0.80] <- 0   # miss some exposed
expo_meas[expo_true == 0 & runif(N) > 0.80] <- 1   # mislabel some unexposed
round(c(true_OR = or_fun(expo_true, disease),
        misclassified_OR = or_fun(expo_meas, disease)), 3)
# sensitivity specificity
#        0.80        0.95
#    true_OR  misclassified_OR
#      ~2.8        ~1.8

import numpy as np
TP, FN = 96, 24      # 120 true diabetics
FP, TN = 44, 836     # 880 true non-diabetics

sens = # TP / (TP + FN)
spec = # TN / (TN + FP)

sens = TP / (TP + FN)   # 0.80
spec = TN / (TN + FP)   # 0.95
print(round(sens, 3), round(spec, 3))

# now: misclassification dilutes an odds ratio toward 1
rng = np.random.default_rng(7)
N = 20000
expo_true = rng.binomial(1, 0.3, N)
p = 1 / (1 + np.exp(-(-2 + np.log(3) * expo_true)))   # true OR = 3
disease = rng.binomial(1, p)
def odds_ratio(e, d):
    a = np.sum((e == 1) & (d == 1)); b  = np.sum((e == 1) & (d == 0))
    c = np.sum((e == 0) & (d == 1)); dd = np.sum((e == 0) & (d == 0))
    return (a * dd) / (b * c)
em = expo_true.copy()
em[(expo_true == 1) & (rng.random(N) > 0.80)] = 0   # miss some exposed
em[(expo_true == 0) & (rng.random(N) > 0.80)] = 1   # mislabel some unexposed
print(round(odds_ratio(expo_true, disease), 3),
      round(odds_ratio(em, disease), 3))
# 0.8 0.95
# 2.78 1.829

The test reads sensitivity 0.80 and specificity 0.95: it clears non-diabetics well but misses one diabetic in five. In the second block the true odds ratio of about 2.8 falls to about 1.8 once the exposure is misclassified at the same rate in cases and non-cases. The signal is real, but a blurry label hides much of it. If you reported 1.8, you would call a strong association a moderate one.

Assessing agreement

Before trusting a measurement you often want to quantify how well two methods, or two occasions, agree. The method depends on the variable type.

1. Continuous measurements: the Bland-Altman idea. Do not judge agreement by the correlation between two methods. A correlation of 0.98 can hide a method that reads twice as high as the other. Instead plot the difference between each pair against the mean of the pair. The average difference shows systematic bias (one method runs high or low), and the spread of differences, summarised as the mean difference plus or minus twice their standard deviation, gives the 95% limits of agreement. You then decide whether that spread is small enough for clinical use.
2. Categorical measurements: Cohen's kappa. Two raters will agree on some labels by luck alone. Cohen's kappa is the agreement beyond chance, scaled so that 1 is perfect and 0 is no better than guessing. Values above about 0.75 are usually read as strong agreement, and below 0.40 as poor.

The practical message: measurement sets the ceiling

Every result here points one way. Random error in an exposure dilutes its slope. Misclassification dilutes its odds ratio. Regression to the mean fabricates a change that is not there. None of these is a bug in your analysis. They are baked into the data the moment it is recorded, and a fancier model cannot remove what the instrument never captured. Measurement quality is the ceiling on what any analysis can recover. A single rushed blood-pressure reading, a one-off food recall, a screening test with 80% sensitivity: each sets a hard limit on the strength of association you can ever detect, no matter how large the sample. The cheapest way to improve a study is often a better measurement, or a second one, before any statistics begin.

Common mistakes

• Treating a weak association as proof of no effect. A flat slope from a noisy single measurement is the expected result of regression dilution, not evidence the exposure is harmless. Estimate the ICC from repeat measurements and correct before you conclude.
• Reading a before-and-after drop as a treatment effect. In a group selected for high baseline values, regression to the mean lowers the average on its own. Without a control arm you cannot separate the drug from the artefact.
• Assuming misclassification must be hiding a true null. Non-differential misclassification pulls toward 1, so a diluted odds ratio of 1.3 may sit on a true value of 2 or more. The null is not safe just because the measurement was rough.
• Judging agreement by correlation. High correlation between two methods says they move together, not that they agree. Two methods can correlate near 1 while one reads systematically higher. Use a Bland-Altman plot or kappa instead.
• Plotting change against the baseline reading. Differencing against the initial value builds in regression to the mean and manufactures a negative slope. Plot the difference against the mean of the pair if you must look at it at all.

Tips

• Build repeat measurements into the design. Even a subsample measured twice lets you estimate the ICC and correct the dilution afterward.
• Spend on the exposure, not the sample size alone. Phillips and Davey Smith showed that more measurements per person can buy more than more people, because it lifts the ICC and shrinks the bias.
• Match the test to the job. Choose high sensitivity when missing a case is costly (early screening), high specificity when a false positive is costly (confirming a diagnosis before harsh treatment).
• Report the limits of agreement, rather than a correlation, whenever you swap a cheap method for an accurate one. The decision is whether the spread is clinically tolerable.
• When an effect looks too good in a single-arm before-and-after, suspect regression to the mean first and look for a control comparison.