Systematic reviews and meta-analysis

One trial rarely settles a clinical question. Suppose five randomized trials across Southeast Asia each tested whether adding a low-cost generic drug to standard care reduces the odds that a patient with severe dengue needs intensive care. Some trials saw a clear benefit, one saw none, and they ran on different sample sizes. Reading them one at a time leaves you guessing. A meta-analysis combines all five into a single pooled effect with one confidence interval, so you can act on the whole body of evidence instead of the loudest study. This lesson shows how that pooling works, and how to do it by hand in R and Python.

You already know odds ratios, the log scale, confidence intervals, and that effects vary from study to study. Here you put those together. We work on the log odds ratio scale throughout, because that scale is roughly symmetric and its variance is easy to handle, then we exponentiate the pooled result back to an odds ratio at the end.

Systematic review, not a narrative review

A meta-analysis lives inside a systematic review. A narrative review is one expert's summary of the papers they happened to read, in the order they chose to tell it. A systematic review sets the question, the inclusion criteria, and the search strategy in advance, then finds and appraises every eligible study by that fixed protocol. The point is to limit the reviewer's bias before any number is pooled. If the review is sloppy, the meta-analysis just gives a precise summary of biased studies. The statistics here assume the review feeding them was done well.

Pooling effects into one summary

The pooled estimate is a weighted average of the individual log odds ratios. The only real question is what weight each study gets. Giving every study an equal vote would let a 40-patient trial outvote a 4,000-patient one, which throws away information. A large, precise study should count for more.

The standard answer is inverse-variance weighting: weight each study by one over the variance of its effect estimate. A precise study has a small variance, so one over that variance is large, and it pulls the pooled estimate toward itself. The weight and the pooled fixed-effect estimate are:

Here is the log odds ratio reported by study and is its variance, which you read off each study's standard error or confidence interval. Exponentiate and its interval to get a pooled odds ratio you can report.

Fixed-effect vs random-effects

There are two models, and they answer different questions. A fixed-effect meta-analysis assumes one common true effect sits behind all the studies, and that the only reason the estimates differ is sampling error. Under that assumption the pooled value estimates that single shared effect.

A random-effects meta-analysis assumes the true effect itself varies from study to study, drawn from a distribution with its own spread. Maybe the drug works better in one setting than another. The pooled value now estimates the mean of those varying true effects, and the model folds the between-study variation into the standard error. When studies disagree, random-effects intervals are wider, which is the honest price of admitting the effect is not one fixed number.

Heterogeneity: Q, I-squared, and tau-squared

Heterogeneity is real variation in the true effect across studies, on top of sampling noise. Three quantities describe it, and they are not interchangeable.

Cochran's Q is a weighted sum of how far each study's estimate sits from the pooled value. Large studies that miss the pool count more. Compare Q to a chi-squared distribution with degrees of freedom, where is the number of studies.

Q has a weakness: with few studies it misses real heterogeneity, and with many studies it flags trivial amounts. So we rescale it. I-squared is the percentage of the total variation in estimates that comes from real heterogeneity rather than chance. Rough guide: 25% is low, 50% moderate, 75% high.

Tau-squared is the actual estimated variance of the true effects between studies, on the log odds ratio scale. The DerSimonian-Laird formula reads it off Q, the weights, and the number of studies:

The difference matters. I-squared is a proportion with no units, so it does not shrink just because effects are small. Tau-squared is a variance in the units of the effect, and it is what the random-effects model actually plugs in. The random-effects weights add tau-squared to each variance, which shrinks the big studies' advantage and spreads the weights more evenly:

Reading a forest plot

A forest plot is the standard picture of a meta-analysis. Read it like this:

1. Each study is one row. The square marks its point estimate, and the horizontal line through it is the study's 95% confidence interval.
2. The square's area is the study's weight. Big squares are the studies that move the pool.
3. The x-axis is the odds ratio on a log scale, with a vertical line at 1 for no effect. A confidence line crossing 1 means that study alone is not statistically significant.
4. The diamond at the bottom is the pooled estimate. Its centre is the pooled odds ratio and its width is the pooled confidence interval. If the diamond clears the line at 1, the combined evidence shows an effect.

Worked example: fixed-effect pooling

Here are our five dengue trials. The outcome is ICU admission, so an odds ratio below 1 favours the drug. Each study reports a log odds ratio and its variance.

The Kuala Lumpur trial has the smallest variance, so it carries the most weight (about 36% of the total). The Manila trial is small, with the most extreme odds ratio and the least weight (about 9%). That pattern, where the smallest studies sit furthest from the middle, is the everyday face of sampling error.

est <- c(-0.50, -0.20, 0.10, -0.90, -0.35)   # log OR per study
v   <- c( 0.02,  0.03, 0.05,  0.08,  0.04)   # variance of each log OR
study <- c("KL", "Jakarta", "Bangkok", "Manila", "Hanoi")

w <- # inverse-variance weights
logOR_F <- # weighted mean of est

w       <- 1 / v
logOR_F <- sum(w * est) / sum(w)
se_F    <- sqrt(1 / sum(w))
ci_F    <- logOR_F + c(-1, 1) * 1.96 * se_F
round(c(OR = exp(logOR_F), lower = exp(ci_F[1]), upper = exp(ci_F[2])), 3)
#    OR lower upper
# 0.703 0.596 0.829

# optional forest-style plot on the odds-ratio scale
lo <- est - 1.96 * sqrt(v)
hi <- est + 1.96 * sqrt(v)
plot(exp(est), 5:1, xlim = c(0.2, 2), log = "x", pch = 15,
     xlab = "Odds ratio", ylab = "", yaxt = "n")
segments(exp(lo), 5:1, exp(hi), 5:1)
abline(v = 1, lty = 2); abline(v = exp(logOR_F), lty = 3)
axis(2, at = 5:1, labels = study, las = 1)

import numpy as np
est = np.array([-0.50, -0.20, 0.10, -0.90, -0.35])   # log OR per study
v   = np.array([ 0.02,  0.03, 0.05,  0.08,  0.04])   # variance of each log OR

w = # inverse-variance weights
logOR_F = # weighted mean of est

w        = 1 / v
logOR_F  = np.sum(w * est) / np.sum(w)
se_F     = np.sqrt(1 / np.sum(w))
ci_F     = logOR_F + np.array([-1, 1]) * 1.96 * se_F
print(np.round([np.exp(logOR_F), np.exp(ci_F[0]), np.exp(ci_F[1])], 3))
# [0.703 0.596 0.829]
print("weights %:", np.round(w / np.sum(w) * 100, 1))
# weights %: [35.5 23.7 14.2  8.9 17.8]

Worked example: DerSimonian-Laird random effects

The five trials do not all agree. Bangkok even points the other way. Before trusting the tight fixed-effect interval, test for heterogeneity, then refit with random effects if there is any. We compute Q, I-squared, and tau-squared, then the random-effects pool.

1. Compute the fixed-effect pool and its weights, as above.
2. Compute Cochran's Q as the weighted sum of squared deviations from the fixed-effect pool.
3. Turn Q into I-squared and the DerSimonian-Laird tau-squared.
4. Build the random-effects weights and pool again.

est <- c(-0.50, -0.20, 0.10, -0.90, -0.35)
v   <- c( 0.02,  0.03, 0.05,  0.08,  0.04)
k   <- length(est)

Q    <- # weighted sum of squared deviations
tau2 <- # DerSimonian-Laird between-study variance

w       <- 1 / v
logOR_F <- sum(w * est) / sum(w)
Q       <- sum(w * (est - logOR_F)^2)
df      <- k - 1
I2      <- max(0, (Q - df) / Q) * 100
C       <- sum(w) - sum(w^2) / sum(w)
tau2    <- max(0, (Q - df) / C)
round(c(Q = Q, df = df, I2 = I2, tau2 = tau2), 3)
#      Q     df     I2   tau2
#  9.705  4.000 58.785  0.053

wr      <- 1 / (v + tau2)
logOR_R <- sum(wr * est) / sum(wr)
se_R    <- sqrt(1 / sum(wr))
ci_R    <- logOR_R + c(-1, 1) * 1.96 * se_R
round(c(OR = exp(logOR_R), lower = exp(ci_R[1]), upper = exp(ci_R[2])), 3)
#    OR lower upper
# 0.704 0.539 0.921

import numpy as np
from scipy.stats import chi2
est = np.array([-0.50, -0.20, 0.10, -0.90, -0.35])
v   = np.array([ 0.02,  0.03, 0.05,  0.08,  0.04])
k   = len(est)

Q = # weighted sum of squared deviations
tau2 = # DerSimonian-Laird between-study variance

w       = 1 / v
logOR_F = np.sum(w * est) / np.sum(w)
Q       = np.sum(w * (est - logOR_F)**2)
df      = k - 1
p_Q     = chi2.sf(Q, df)
I2      = max(0, (Q - df) / Q) * 100
C       = np.sum(w) - np.sum(w**2) / np.sum(w)
tau2    = max(0, (Q - df) / C)
print("Q=%.3f p=%.3f I2=%.1f tau2=%.3f" % (Q, p_Q, I2, tau2))
# Q=9.705 p=0.046 I2=58.8 tau2=0.053

wr      = 1 / (v + tau2)
logOR_R = np.sum(wr * est) / np.sum(wr)
se_R    = np.sqrt(1 / np.sum(wr))
ci_R    = logOR_R + np.array([-1, 1]) * 1.96 * se_R
print(np.round([np.exp(logOR_R), np.exp(ci_R[0]), np.exp(ci_R[1])], 3))
# [0.704 0.539 0.921]

Publication bias and the funnel plot

Meta-analysis can only pool the studies it can find, and small studies with disappointing results are the ones most likely to go unpublished. That is publication bias, and it makes the pooled effect look stronger than the truth. The usual screen is a funnel plot: plot each study's effect on the x-axis against its precision on the y-axis. Large precise studies cluster near the top around the true effect; small imprecise studies scatter wide at the bottom. With no bias the cloud is a symmetric inverted funnel. A gap in one bottom corner, where small null or unfavourable trials should be, suggests missing studies. Asymmetry can also come from real heterogeneity or chance, so treat a funnel plot as a prompt to investigate, not as proof.

Common mistakes

• Reading I-squared as the amount of heterogeneity. I-squared is a percentage of total variation, not a variance. Two analyses with the same tau-squared can show sharply different I-squared values just because their studies differ in size. For the actual between-study spread, report tau-squared.
• Trusting a non-significant Q as proof of homogeneity. With only a handful of studies, the Q test has low power, so a p-value above 0.05 does not mean the effects agree. Look at I-squared and tau-squared as well, and do not switch to fixed effects just because Q was not significant.
• Pooling on the raw OR scale. Always weight and average on the log odds ratio scale, then exponentiate at the end. Averaging raw odds ratios is biased because the OR is not symmetric, and an OR of 2 and one of 0.5 should cancel, not average to 1.25.
• Treating random effects as more trustworthy by default. It is more conservative, but it leans harder on small studies, which carry the most publication bias. Wider is not the same as more correct.
• Pooling clinically incompatible studies. Combining trials with different drugs, doses, or outcomes gives a precise average of things that should not be averaged. Heterogeneity statistics flag this, but only judgement decides whether pooling makes sense.

Tips

• Get each study's variance from its reported confidence interval: on the log scale, , then square it.
• Always report both models. If the fixed-effect and random-effects pooled estimates agree, heterogeneity is not driving your conclusion. If they diverge, that gap is itself a finding worth explaining.
• When tau-squared comes out at zero, the random-effects and fixed-effect results are identical by construction. That is expected, not a bug.
• Quote I-squared with its rough bands (low, moderate, high) and tau-squared together. One gives the proportion, the other the size; readers need both.
• Treat a funnel plot or its asymmetry test as a screen for use with at least 10 studies. With fewer, it has too little power to be worth much.