Transformations

Many measurements in medicine pile up near small values and trail off into a few large ones. Serum C-reactive protein, parasite counts, antibody titres, hospital length of stay, and incubation periods all behave this way. The mean sits to the right of most of the data, and a t test or a confidence interval built on the raw numbers can mislead. A transformation rescales the measurement so that the methods you already know start to fit.

There are two reasons to reach for one. The first is skewness: the distribution is lopsided, with a long tail on the high side, so it is far from the symmetric bell that t tests and least-squares regression assume. The second is non-constant variance: the spread of the data grows as the mean grows, so groups with higher averages also scatter more. When the standard deviation rises roughly in step with the mean, the two-sample t test loses its footing, because it assumes both groups share one spread. A good transformation can fix both problems at once.

The log transformation

The workhorse is the natural logarithm. You replace each value with , run the whole analysis on the logged values, then convert the answer back at the end. The natural log uses base e, where e = 2.71828. Software almost always means base e when it writes log, and you will sometimes see it written ln. Base 10 logs differ only by a constant factor, so the choice of base does not change any test.

Picture what the log does to the number line. It stretches out the crowded low values and compresses the stretched-out high values. On a log scale the gap from 1 to 10 is the same size as the gap from 10 to 100 and from 100 to 1000, because each is a ten-fold step. That is the surgery a right-skewed variable needs. The long upper tail gets pulled in, and a lognormal variable (one whose logs follow a normal distribution) becomes symmetric.

Worked example 1: a geometric mean for CRP

Seven patients at a Klang Valley clinic have these C-reactive protein readings, in mg/L: 2, 4, 6, 10, 18, 40, 120. The arithmetic mean is 200/7 = 28.6 mg/L, but six of the seven patients sit below that. One high reading has dragged the average away from the typical patient. Plot the data before and after logging to see the shape change.

crp <- c(2, 4, 6, 10, 18, 40, 120)

hist(crp)   # right-skewed

par(mfrow = c(1, 2))
hist(crp, main = "CRP (mg/L)", xlab = "CRP")
hist(log(crp), main = "log CRP", xlab = "log CRP")
c(mean = mean(crp), gm = exp(mean(log(crp))))
# mean 28.571, gm 12.253

import numpy as np
import matplotlib.pyplot as plt
crp = np.array([2, 4, 6, 10, 18, 40, 120])

plt.hist(crp)   # right-skewed
plt.show()

fig, ax = plt.subplots(1, 2, figsize=(8, 3))
ax[0].hist(crp); ax[0].set_title("CRP (mg/L)")
ax[1].hist(np.log(crp)); ax[1].set_title("log CRP")
plt.tight_layout(); plt.show()
print("mean", round(crp.mean(), 3),
      "GM", round(np.exp(np.log(crp).mean()), 3))
# mean 28.571 GM 12.253

Now work on the log scale. The mean of the seven logged values is 2.506. Exponentiate it and you get back a number in the original mg/L units.

The confidence interval is built the same way. Compute the mean and standard deviation of the logs, form the usual t interval on the log scale, then exponentiate both limits.

crp <- c(2, 4, 6, 10, 18, 40, 120)
u <- log(crp)

ubar <- mean(u)
exp(ubar)   # geometric mean

n <- length(u)
ubar <- mean(u); s <- sd(u)
tcrit <- qt(0.975, df = n - 1)
ci_log <- c(ubar - tcrit * s / sqrt(n),
            ubar + tcrit * s / sqrt(n))
c(GM = exp(ubar), lower = exp(ci_log[1]),
  upper = exp(ci_log[2]))
# GM 12.25, lower 3.34, upper 44.94

import numpy as np
from scipy import stats
crp = np.array([2, 4, 6, 10, 18, 40, 120])
u = np.log(crp)

ubar = u.mean()
np.exp(ubar)   # geometric mean

n = u.size
ubar = u.mean(); s = u.std(ddof=1)
t = stats.t.ppf(0.975, n - 1)
ci = np.exp([ubar - t * s / np.sqrt(n),
             ubar + t * s / np.sqrt(n)])
print("GM", round(np.exp(ubar), 2), "CI", ci.round(2))
# GM 12.25 CI [ 3.34 44.94]

For these data the 95% interval on the log scale is 1.206 to 3.805. Exponentiating gives a geometric mean of 12.3 mg/L with a 95% confidence interval of 3.3 to 44.9 mg/L. Notice the interval is not symmetric around 12.3. On the original scale a step of one standard deviation is a multiplier, not an addition, so the upper limit sits further away in absolute terms. The ratio is what stays constant: 44.9 / 12.3 = 3.7 and 12.3 / 3.3 = 3.7.

Interpreting results on the log scale: ratios

The log turns differences into ratios, and that is the key to reading the output. Subtraction on the log scale is division on the original scale, because log A minus log B equals log(A/B). So when you compare two groups by the difference in their mean logs, back-transforming that difference gives the ratio of their geometric means, not a difference in raw units.

Worked example 2: comparing two treatment arms

A small trial compares two antimalarials by parasite density, in parasites per microlitre, measured on day 2. Five patients on drug A read 800, 1200, 1600, 2400, 3200. Five on drug B read 3000, 4500, 6000, 9000, 12000. The densities are right-skewed and the higher-density arm also scatters more, so the comparison runs on the log scale.

The geometric mean density is 1640 per microlitre on drug A and 6140 per microlitre on drug B. The difference in mean logs is 7.401 - 8.723 = -1.322. Back-transform it and the ratio of geometric means is exp(-1.322) = 0.27. Drug A leaves about a quarter of the parasite burden that drug B does, a 73% reduction. A two-sample t test on the logs (equal variances) gives t = -3.81 on 8 degrees of freedom, p ≈ 0.005, and the 95% interval for the ratio is 0.12 to 0.59. That interval excludes 1, so the difference is unlikely to be chance.

A <- c(800, 1200, 1600, 2400, 3200)
B <- c(3000, 4500, 6000, 9000, 12000)

tt <- t.test(log(A), log(B), var.equal = TRUE)
tt$statistic

tt <- t.test(log(A), log(B), var.equal = TRUE)
gmA <- exp(mean(log(A)))
gmB <- exp(mean(log(B)))
ratio <- exp(mean(log(A)) - mean(log(B)))
ci_ratio <- exp(tt$conf.int)   # ratio scale
c(gmA = gmA, gmB = gmB, ratio = ratio,
  lo = ci_ratio[1], hi = ci_ratio[2],
  t = tt$statistic)
# gmA 1638, gmB 6144, ratio 0.267, lo 0.12, hi 0.59, t -3.81

import numpy as np
from scipy import stats
A = np.array([800, 1200, 1600, 2400, 3200])
B = np.array([3000, 4500, 6000, 9000, 12000])

res = stats.ttest_ind(np.log(A), np.log(B))
res.statistic

res = stats.ttest_ind(np.log(A), np.log(B))   # equal_var=True
gmA = np.exp(np.log(A).mean())
gmB = np.exp(np.log(B).mean())
ratio = gmA / gmB
print("GM_A", round(gmA), "GM_B", round(gmB),
      "ratio", round(ratio, 3),
      "t", round(res.statistic, 3))
# GM_A 1638 GM_B 6144 ratio 0.267 t -3.807

Back-transformation to the original scale

Always run the analysis on the transformed values, then convert the final summaries back so a clinician can read them. For a log transform you back-transform by exponentiating, also called taking the antilog. The antilog of a mean is a geometric mean, and the antilog of a difference is a ratio. Do not back-transform a standard deviation. On the original scale it acts as a multiplier and has no clean interpretation, so report the geometric mean with its confidence interval instead.

Other transformations and how to choose

The log is the most common transform, but it is one of a family. Pick by how skewed the data are, or by how fast the spread grows with the mean.

• Square root (u = √x) is weaker than the log. Use it for mild positive skew, and for count data such as weekly dengue notifications, where the variance of a count tends to equal its mean.
• Logarithm (u = log x) suits a lognormal shape, and the case where the standard deviation rises in proportion to the mean.
• Reciprocal (u = 1/x) is stronger than the log. Use it when the data are more skewed than lognormal, or when the standard deviation grows with the square of the mean. It is natural for rates, since 1 over a time is a speed.

Negative skew, with a long tail on the low side, is the mirror image. Square or cube the values to pull the low tail in. The table below pairs each situation with its transform.

To choose in practice, try a few transforms and read which one brings the skewness nearest to zero.

counts <- c(1, 2, 3, 5, 8, 13, 21, 55, 144)

hist(counts)

skew <- function(x) {
  n <- length(x); m <- mean(x)
  (sum((x - m)^3) / n) / (sqrt(sum((x - m)^2) / n))^3
}
round(c(raw   = skew(counts),
        sqrt  = skew(sqrt(counts)),
        log   = skew(log(counts)),
        recip = skew(1 / counts)), 3)
# read which value is closest to 0

import numpy as np
from scipy import stats
counts = np.array([1, 2, 3, 5, 8, 13, 21, 55, 144])

stats.skew(counts)

for name, x in [("raw", counts), ("sqrt", np.sqrt(counts)),
                ("log", np.log(counts)), ("recip", 1 / counts)]:
    print(name, round(stats.skew(x), 3))
# read which value is closest to 0

Two practical rules close the decision. Choose the transform that makes the histogram most symmetric and the group spreads most equal, not the one that gives the smallest p value. And use the same transform for every group in a comparison, otherwise the groups are no longer on the same scale.

Common mistakes

• Calling the back-transformed mean an arithmetic mean. exp of the mean log is the geometric mean. The two differ, and labelling one as the other misreports the average.
• Back-transforming a standard deviation. On the log scale the s.d. is multiplicative, so its antilog is not a plus-or-minus range in the original units. Report the geometric mean with its confidence interval instead.
• Logging data that contain zeros. log(0) is undefined, so those records are silently dropped or turn into errors. Add a constant first, and subtract it after back-transforming.
• Using different transforms for the two groups you compare. Once the groups are on different scales the difference means nothing. Apply one transform everywhere.
• Reporting the geometric mean as a single plus-or-minus margin. A log-based interval is symmetric only on the log scale. On the original scale the limits are a multiplicative pair, so quote both limits.

Tips

• Keep the raw data and apply the transform inside the analysis script, so the original units are never lost and the work is reproducible.
• State the transform you used in the methods. A reviewer who sees "geometric mean" should be able to tell you logged the data.
• For a fast skewness check, compare the mean and the median. A mean well above the median signals a long right tail that a log may fix.
• For counts with many zeros, consider a Poisson or negative-binomial model instead of a transform. It handles zeros directly and needs no added constant.
• When you report a ratio of geometric means, give its confidence interval and check whether it crosses 1, the no-difference value on the ratio scale.