Rates and the Poisson distribution

Follow people over time and the same number of events can mean different things, depending on how long you watched. A cohort that tracks 500 patients for one year is not comparable to one that tracks 500 patients for five years: the second has five times the opportunity to see events. To compare fairly you count not just how many events happened, but how much time each person spent at risk. That total time is the person-time, and the event count divided by it is a rate. This lesson defines person-time and the rate, contrasts the rate with the risk you met in Part C, and introduces the Poisson distribution that models a count of events accumulated over person-time.

You already know risk, proportions, confidence intervals, and the binomial. A rate is the natural outcome measure once follow-up times differ between people, which is the rule rather than the exception in longitudinal studies.

Person-time: counting time at risk

In most follow-up studies people are watched for unequal lengths of time. Someone recruited late, someone who emigrates, someone who dies of an unrelated cause, and someone followed to the closing date all contribute different amounts of observation. A method that counts only people throws this information away.

Each person contributes time from the moment they enter until the moment they leave. They leave when they have the event, are lost to follow-up, or reach the end of the study, whichever comes first. That span is their period at risk: the time during which an event, if it happened, would be recorded. Add these spans across everyone and you have the total person-time, written . Measured in years it is called person-years at risk.

• Staggered entry: people are recruited across months but followed to one end date.
• Withdrawal or emigration out of the study area before the end.
• Death from a cause other than the event of interest.
• Age-defined cohorts, where people enter and leave the eligible age band as they age.

The sum equals the average number of people under observation multiplied by the length of the study. The same 3,200 person-years can come from 3,200 people for one year, 1,600 people for two years, or 640 people for five years. The rate is the same in each case.

The rate, and its units

The rate is the number of new events per unit of person-time. We write it with the Greek letter (lambda). You estimate it by dividing the observed events by the total person-time .

A rate carries units, and you must state them. An incidence of 0.015 per person-year is the same number as 15 per 1,000 person-years; the second is easier to read for a fairly rare event, so rates are often scaled by 1,000 or 100,000. The rate is also called the incidence rate, or the mortality rate when the event is death. Unlike a proportion it is not bounded by 1: a recurrent event such as asthma attacks can give more than one event per person-year.

Rate or risk: which one fits

Risk, the cumulative incidence from Chapter 15, divides events by the number of people at risk at the start. It is a probability, and it carries time only implicitly: the longer you watch, the higher the risk, because there is more time for events to occur. Quote a risk and you must quote the period it refers to, such as a 5-year risk.

A rate puts time in the denominator explicitly, so it handles varying follow-up without trouble. Use a rate when follow-up times differ, when events can recur, or when you want a measure that does not depend on choosing a fixed window. Use a risk when you want the plain probability that an individual will experience the event over a stated period, which is often what a patient wants to hear. When the rate is small, risk over a short time is close to rate times time.

The Poisson distribution

A rate is a count over person-time, so to attach uncertainty to it you need the sampling distribution of that count. For means we used the normal distribution and for proportions the binomial. For a count of events over time or space the model is the Poisson distribution. It describes how many occurrences you see in a fixed amount of person-time, and it depends on a single parameter: the mean count expected in that amount of time.

Here is the constant 2.71828, and equals 1, so the probability of zero events is . For a small mean the distribution is skewed with a real chance of zero; for a mean of 10 or more it is close to symmetric and the normal approximation works.

Mean equals variance

The Poisson distribution has one property that does all the later work: its mean and its variance are both equal to .

Two consequences follow. First, the spread of a count grows with its mean, but only as the square root, so larger counts are relatively more precise. Second, you can read the uncertainty of a count straight off the count itself: the standard error of an observed events is estimated by . If you ever see a count whose spread is much wider than its mean, the Poisson model is the wrong one, a situation called overdispersion.

What a Poisson count assumes

The Poisson distribution holds when two conditions are met. Events occur independently of each other, so one event does not change the chance of the next. And the underlying rate is constant, so events fall at random over the person-time rather than bunching. Congenital malformations in a district each year fit this, provided there is no epidemic. Counts that cluster do not: an outbreak of an infectious disease breaks independence, and repeated episodes in the same person break it too. When events cluster, the real spread exceeds and the plain Poisson standard error is too small.

Standard error of a rate and of the log rate

Since the rate is the count divided by , and the standard error of the count is , the standard error of the rate follows by dividing through by .

For confidence intervals it is better to work with the log rate, because the log scale keeps the interval above zero where a rate has to live. The standard error of the log rate has a clean form that depends only on the number of events, not on the follow-up time.

Confidence interval for a single rate

Build the interval on the log scale, then antilog back. Because the standard error there is , the two steps collapse into a single error factor.

Multiply and divide the rate by the error factor. For an exact cross-check that does not lean on the log approximation, R has poisson.test, which inverts the Poisson distribution directly. The two agree closely when is moderate and drift apart when events are few.

Worked example 1: incidence and its 95% CI

A Malaysian type 2 diabetes cohort recorded 48 first cardiovascular events over 3,200 person-years of follow-up. Report the incidence rate per 1,000 person-years with a 95% confidence interval.

1. Rate: per person-year, which is per 1,000 person-years.
2. Standard error of the log rate: .
3. Error factor: .
4. Interval: per 1,000 person-years.

So the incidence is 15 per 1,000 person-years, with a 95% interval of about 11 to 20. The exact Poisson interval from poisson.test lands at almost the same place.

d <- 48
T <- 3200

rate <- # events per 1000 person-years
EF   <- # error factor from 1/sqrt(d)

rate <- d / T * 1000
EF   <- exp(1.96 / sqrt(d))
ci   <- c(rate / EF, rate * EF)
round(c(rate = rate, lower = ci[1], upper = ci[2]), 2)
#  rate  lower  upper
# 15.00  11.30  19.90

# exact cross-check, scaled to per 1000 person-years
pt <- poisson.test(d, T)
round(pt$conf.int * 1000, 2)
# 11.06 19.89

import numpy as np
from scipy.stats import chi2
d, T = 48, 3200

rate = # events per 1000 person-years
EF   = # error factor from 1/sqrt(d)

rate = d / T * 1000
EF   = np.exp(1.96 / np.sqrt(d))
ci   = np.array([rate / EF, rate * EF])
print(round(rate, 2), np.round(ci, 2))
# 15.0 [11.3 19.9]

# exact (Poisson) interval, per 1000 person-years
lower = chi2.ppf(0.025, 2 * d)       / 2 / T * 1000
upper = chi2.ppf(0.975, 2 * (d + 1)) / 2 / T * 1000
print(round(lower, 2), round(upper, 2))
# 11.06 19.89

Worked example 2: probabilities from the Poisson distribution

A state surveillance unit records on average 3.5 cases of a notifiable infection per week, and the counts behave like a Poisson process. What is the probability of seeing exactly 6 cases in a given week, and the probability of 8 or more, which would trigger a review?

Use the Poisson distribution with . The probability of exactly 6 is . For 8 or more, add the probabilities of 0 to 7 and subtract from 1, which gives 0.0267. So a week with 6 cases is unremarkable at about 8%, while a review-triggering week of 8 or more happens only about 2.7% of the time under the assumed mean.

mu <- 3.5

p6     <- # P(exactly 6) with dpois
p8plus <- # P(8 or more) with ppois

p6     <- dpois(6, mu)                       # exactly 6 cases
p8plus <- ppois(7, mu, lower.tail = FALSE)   # P(X >= 8)
round(c(p6 = p6, p8plus = p8plus), 4)
#     p6  p8plus
# 0.0771  0.0267

from scipy.stats import poisson
mu = 3.5

p6     = # P(exactly 6) with poisson.pmf
p8plus = # P(8 or more) with poisson.sf

p6     = poisson.pmf(6, mu)   # exactly 6 cases
p8plus = poisson.sf(7, mu)    # P(X >= 8)
print(round(p6, 4), round(p8plus, 4))
# 0.0771 0.0267

Common mistakes

• Dividing by people when follow-up varies. If people are watched for different lengths of time, events over the head count is not a rate. Use total person-time in the denominator, or you mix a one-year and a five-year experience into one misleading number.
• Treating a rate like a probability. A rate is not bounded by 1. For a recurrent event it can exceed one event per person-year, and a rate of 1,500 per 1,000 person-years is real, not an error. Do not clamp it to the 0-to-1 range you use for risks.
• Building the interval on the natural scale. Using directly can push the lower bound below zero when events are few. Work on the log scale with the error factor, which keeps both bounds positive.
• Forgetting the count drives the precision. The standard error of the log rate is . More person-time with the same handful of events barely narrows the interval; you need more events.
• Using Poisson on clustered counts. Outbreaks and repeated episodes in the same person break independence. The true spread then exceeds the mean, and the Poisson standard error is too small.

Tips

• Always report a rate with its units. "15" means nothing; "15 per 1,000 person-years" is a result.
• Precision is bought with events, not time. Because the log-rate standard error is , halving the interval width needs roughly four times the events.
• For a small number of events, cross-check the log-based interval against poisson.test in R or the exact chi-square bounds in Python. They diverge when is small.
• When the expected count reaches about 10, the Poisson is close to normal, so a z-based interval and the exact interval nearly coincide.
• Before trusting any Poisson interval, ask whether events could cluster. If one person can contribute many events, or an outbreak is possible, the independence assumption is the thing to check first.