Linear regression and correlation

You measured two numbers on each patient, and you want to know how one moves with the other. Does birth weight rise with gestational age? Does blood pressure climb with age? Linear regression draws the single straight line that best summarises that relationship, and the correlation coefficient puts a number on how tightly the points hug it. This lesson fits both by hand on small datasets, reads off what each number means, and tests whether the trend is real or just sampling noise.

The running example is a small antenatal audit at a Klang district clinic. For six babies you recorded gestational age at delivery (in completed weeks) and birth weight (in kilograms). The outcome you want to explain is birth weight, written y. The thing you explain it with is gestational age, written x. By convention the outcome goes on the vertical axis and the explanatory variable on the horizontal.

The regression line and least squares

The model says each birth weight is a straight-line function of gestational age plus an error. Write it like this.

Here is the intercept, is the slope, and is the error: the vertical gap between a real point and the line. The error has mean zero, so the line passes through the middle of the cloud. We never see the true and . We estimate them from the sample.

The least-squares slope and intercept have closed forms. Let and be the two means.

Work the three sums first. The means are weeks and kg. The cross-product sum is , and the spread of x is . So the slope is and the intercept is . The fitted line is birth weight = -4.38 + 0.194 × age. As a check, drop in and you get back . A least-squares line always passes through the point of means , so this is a quick sanity check on your arithmetic.

Reading the slope and intercept

The slope is the part you almost always care about. It is the change in the outcome for a one-unit rise in the explanatory variable, carried in the units of both. Here kg per week. Each extra week in the womb is associated with about 0.194 kg more birth weight, roughly 194 grams. The sign tells you the direction: positive means y rises with x. A slope of zero would be a flat line, which means no linear association.

The intercept is the predicted outcome when . Read literally, ours says a baby of zero weeks gestation weighs -4.38 kg. That is impossible, and it is not a flaw in the maths. Gestational age of zero is far outside the 36 to 41 week range you actually observed, and a negative weight is what a straight line gives when you push it somewhere it was never meant to go.

Fit the same line in code. R uses base lm; Python uses statsmodels, which prints the slope, intercept, and their confidence intervals in one summary. Both draw the scatter with the fitted line on top so you can eyeball the fit before trusting any number.

age <- c(36, 37, 38, 39, 40, 41)
bw  <- c(2.5, 3.0, 2.9, 3.3, 3.3, 3.6)

m <- lm(bw ~ age)
# print the coefficients, then add a CI and a plot

m <- lm(bw ~ age)
print(coef(m))          # (Intercept) -4.38 , age 0.194
print(confint(m))       # 95% CI for each coefficient

plot(age, bw, pch = 19, xlab = "Gestational age (weeks)",
     ylab = "Birth weight (kg)", main = "Klang antenatal audit")
abline(m, col = "#B22222", lwd = 2)   # the least-squares line

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
df = pd.DataFrame({"age": [36, 37, 38, 39, 40, 41],
                   "bw":  [2.5, 3.0, 2.9, 3.3, 3.3, 3.6]})

m = smf.ols("bw ~ age", data=df).fit()
# print params, CI, then plot points and line

m = smf.ols("bw ~ age", data=df).fit()
print(m.params)              # Intercept -4.38 , age 0.194
print(m.conf_int())          # 95% CI for each coefficient

plt.scatter(df.age, df.bw, color="black")
xs = np.array([36, 41])
plt.plot(xs, m.params["Intercept"] + m.params["age"] * xs, color="#B22222")
plt.xlabel("Gestational age (weeks)")
plt.ylabel("Birth weight (kg)")
plt.title("Klang antenatal audit")
plt.show()

The correlation coefficient r

The slope tells you the rate of change. It does not tell you how tightly the points sit around the line. Two datasets can share the same slope while one is a clean trend and the other is a scattered mess. The correlation coefficient measures that tightness on a fixed scale.

You already have the top and part of the bottom. The new piece is the spread of y, . So . Birth weight and gestational age are strongly, positively correlated in this audit.

Read by its sign and size. It always lands between -1 and +1. A value of +1 means the points sit exactly on an upward line, -1 means exactly on a downward line, and 0 means no linear association. Positive means x and y rise together; negative means one falls as the other rises. The closer to either end, the tighter the cloud. It carries no units, so the same 0.945 describes the relationship whether you measure weight in kilograms or grams.

Correlation versus the regression slope

People mix these two up constantly, so pin down the difference. The slope and the correlation always share the same sign, because they share the same numerator. When one is zero the other is zero. But they answer different questions and behave differently.

• The slope has units (kg per week) and a direction. Regress weight on age and you get 0.194 kg per week; regress age on weight instead and you get a different number with different units. Swapping which variable explains which changes the slope.
• The correlation is unitless and symmetric. The correlation of age with weight equals the correlation of weight with age. Rescaling either variable leaves it untouched.

They are linked by the two standard deviations. Let and be the sample standard deviations of x and y.

Check it: , , so , matching exactly. One consequence is worth remembering: if you standardise both variables (divide each by its own standard deviation so both have standard deviation 1), the regression slope of the standardised outcome on the standardised predictor is exactly . That is why software can report as a "standardised coefficient".

age <- c(36, 37, 38, 39, 40, 41)
bw  <- c(2.5, 3.0, 2.9, 3.3, 3.3, 3.6)

cor(age, bw)        # the point estimate r
# now get the CI and p-value

cor(age, bw)                 # 0.945
ct <- cor.test(age, bw)      # Pearson, with Fisher CI
print(ct$estimate)           # r = 0.945
print(ct$conf.int)           # about 0.57 to 0.99
print(ct$p.value)            # about 0.0045

import numpy as np
from scipy import stats
age = np.array([36, 37, 38, 39, 40, 41])
bw  = np.array([2.5, 3.0, 2.9, 3.3, 3.3, 3.6])

r, p = stats.pearsonr(age, bw)
print(r, p)
# build the Fisher CI by hand

r, p = stats.pearsonr(age, bw)
print(round(r, 3), round(p, 4))     # 0.945 , 0.0045
n = len(age)
z = np.arctanh(r)                    # Fisher transform
se = 1 / np.sqrt(n - 3)
lo, hi = np.tanh(z - 1.96*se), np.tanh(z + 1.96*se)
print(round(lo, 2), round(hi, 2))    # about 0.57 to 0.99

Testing the slope and its confidence interval

Your slope of 0.194 is an estimate from six babies, so it carries sampling error. The question that matters is whether the true slope could be zero, which would mean no real association. To answer it you need the standard error of the slope, and for that you need , the standard deviation of the points about the line.

The divisor is , the sample size minus the two coefficients you estimated. These are the degrees of freedom. Plugging in: the residual sum of squares is , so kg. The standard error of the slope is .

Now test the null hypothesis that the true slope is zero. The test statistic is the slope over its standard error, compared to a t distribution with degrees of freedom.

A t of 5.77 on 4 degrees of freedom gives a two-sided p-value of about 0.004. There is good evidence that birth weight really does rise with gestational age. The confidence interval uses the same pieces. With the 97.5th percentile of the t distribution on 4 degrees of freedom, which is 2.776, the interval is the estimate plus or minus standard errors.

So you are 95 percent confident each extra week adds between 0.101 and 0.288 kg of birth weight. The interval excludes zero, which agrees with the significant test. In a large sample you would swap the t percentile for the normal value 1.96, but with only four degrees of freedom the t value 2.776 is much wider, and using 1.96 would understate the uncertainty. There is a tidy bonus here: the t statistic for the slope (5.77) is exactly the t statistic for testing that . Testing the slope and testing the correlation are the same test.

R-squared and what it means

The last number ties correlation back to variance. Start with the total variation in the outcome, the sum of squared distances of each y from its mean, . Least squares splits this into two parts: the part the line explains and the part it leaves over as residual scatter.

R-squared is the share the line explains: the regression sum of squares over the total. For the birth-weight audit that is . Gestational age accounts for about 89 percent of the variation in birth weight here, and the remaining 11 percent is other things plus noise. R-squared is exactly the square of the correlation: . That equality is why the letter is squared to write it.

Take a second example to see the same machinery on a different dataset, and to meet the analysis-of-variance view. A hypertension screening at a Penang clinic recorded age and systolic blood pressure for five adults.

The means are and . The sums work out to , , and . So the slope is mmHg per year and the intercept is . The fitted line is systolic BP = 98 + 0.64 × age. Each extra year of age is associated with about 0.64 mmHg more systolic pressure.

The total sum of squares is 440. The regression sum of squares is , leaving a residual of . So : age explains about 93 percent of the spread in these five pressures. The analysis-of-variance table puts those two sums side by side, each divided by its degrees of freedom (1 for the regression, for the residual) to get mean squares, then forms an F statistic as the ratio.

That F of 40.4 on (1, 3) degrees of freedom gives the same p-value as the t test for the slope, about 0.008. This is not a coincidence: for a single predictor, F is exactly the square of the slope t statistic. Here the slope t is 6.36, and . The F test, the t test, and the test that are three views of one question.

age <- c(30, 40, 50, 60, 70)
sbp <- c(118, 124, 130, 132, 146)

m <- lm(sbp ~ age)
# show R-squared and the ANOVA table

m <- lm(sbp ~ age)
print(summary(m)$r.squared)   # 0.931
print(anova(m))               # F = 40.4 on (1, 3) df, p = 0.0079

tval <- coef(summary(m))["age", "t value"]
print(tval^2)                 # 40.4 , equals the F statistic

import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm
df = pd.DataFrame({"age": [30, 40, 50, 60, 70],
                   "sbp": [118, 124, 130, 132, 146]})

m = smf.ols("sbp ~ age", data=df).fit()
print(m.rsquared)
# print the ANOVA table

m = smf.ols("sbp ~ age", data=df).fit()
print(round(m.rsquared, 3))           # 0.931
print(sm.stats.anova_lm(m))           # F = 40.4 , p = 0.0079
print(round(m.tvalues["age"]**2, 1))  # 40.4 , equals the F statistic

Common mistakes

• Reading the intercept literally. Our intercept of -4.38 kg is the line at zero weeks, far outside the data. It is a number the model needs, not a clinical prediction. Centre the predictor if you want a meaningful intercept.
• Extrapolating past the data. The line was fitted on 36 to 41 weeks. Predicting birth weight at 28 weeks assumes the straight-line shape holds where you never checked it. It usually does not.
• Confusing correlation with the slope. A correlation of 0.95 says the points are tight, not that y rises fast with x. A steep slope can have a low correlation if the scatter is wide. Report both.
• Treating a high R-squared as proof of cause. R-squared measures how much variation the line explains, nothing about causation. Age does not cause blood pressure on its own; both move with other factors.
• Using 1.96 in a small sample. With 4 degrees of freedom the right multiplier is 2.776, not 1.96. Using the normal value makes the confidence interval too narrow and overstates your certainty.
• Fitting a line to a curve. If the scatter plot bends, the slope and r summarise the wrong shape. Look at the plot before you fit, every time.

Tips

• Plot first, fit second. The scatter plot tells you in one glance whether a straight line is even the right tool. No summary statistic replaces looking.
• Quote the slope with its confidence interval, not just the p-value. "0.194 kg per week, 95% CI 0.101 to 0.288" tells a clinician the size and the uncertainty. A bare "p = 0.004" tells them neither.
• Say which variable is the outcome. Regressing weight on age and age on weight give different slopes. State the direction so the slope is readable.
• Use R-squared to compare, not to bless. A high R-squared on six points is easy to get and easy to over-read. Pair it with the residual plot and the sample size.
• Match units to the audience. "194 grams per week" lands better than "0.194 kg per week" in an antenatal ward. The model is identical; the units are a communication choice.