Last modified: January 27, 2025

This article is written in: 🇺🇸

Multiple linear regression is a statistical technique used to model the relationship between a single dependent variable and two or more independent variables. It extends the concept of simple linear regression by incorporating multiple predictors to explain the variability in the dependent variable. This method is widely used in fields such as economics, engineering, social sciences, and natural sciences to predict outcomes and understand the impact of various factors.

The Multiple Linear Regression Model

The general form of the multiple linear regression model is:

y=β0+β1x1+β2x2+⋯+βpxp+ε

Where:

• y is the dependent variable (response variable).
• x1,x2,…,xp are the independent variables (predictor variables).
• β0 is the intercept term (the expected value of y when all xj=0).
• β1,β2,…,βp are the coefficients representing the change in y for a one-unit change in xj, holding all other variables constant.
• ε is the random error term, accounting for the variability in y not explained by the linear relationship.

Matrix Representation

In matrix notation, the model can be expressed as:

y=Xβ+ε

Where:

• y is an n×1 vector of observations of the dependent variable.
• X is an n×(p+1) matrix of independent variables, including a column of ones for the intercept.
• β is a (p+1)×1 vector of coefficients.
• ε is an n×1 vector of error terms.

Assumptions of the Model

For the multiple linear regression model to provide valid results, several key assumptions must be met:

1. Linearity means that the relationship between the dependent variable and each independent variable is linear.
2. Independence assumes that the observations are independent of one another.
3. Homoscedasticity ensures that the variance of the error terms remains constant across all levels of the independent variables.
4. Normality requires that the error terms are normally distributed with a mean of zero.
5. Finally, no multicollinearity ensures that the independent variables are not perfectly correlated with each other.

Estimation of Coefficients

Least Squares Method

The coefficients β are estimated using the Ordinary Least Squares (OLS) method, which minimizes the sum of squared residuals (the differences between observed and predicted values of y).

The objective is to find ˆβ such that:

ˆβ=argminβ(y−Xβ)⊤(y−Xβ)

Solution Using Matrix Algebra

By taking the derivative of the sum of squared residuals with respect to β and setting it to zero, we obtain the normal equations:

X⊤Xˆβ=X⊤y

Solving for ˆβ:

ˆβ=(X⊤X)−1X⊤y

Conditions for Invertibility:

• The matrix X⊤X must be invertible.
• This requires that the columns of X are linearly independent (no perfect multicollinearity).

Interpretation of Coefficients

• The intercept (ˆβ0) represents the expected value of y when all independent variables are zero.
• The slope coefficients (ˆβj) indicate the expected change in y for a one-unit increase in xj, while holding all other variables constant.

Assessing Model Fit

Coefficient of Determination (R2)

R2=1−SSRSST

• SSR (Sum of Squared Residuals) measures the unexplained variation in the model, representing the error.
• SST (Total Sum of Squares) captures the total variation in y, showing the overall variability in the dependent variable.

An R2 value close to 1 indicates a good fit.

Adjusted R2

Adjusts R2 for the number of predictors in the model:

Adjusted R2=1−(SSR/(n−p−1)SST/(n−1))

Hypothesis Testing

Testing Individual Coefficients

• Null Hypothesis (H0): βj=0
• Alternative Hypothesis (Ha): βj≠0
• Test Statistic:

tj=ˆβjSE(ˆβj)

• Degrees of Freedom: n−p−1

Testing Overall Model Significance

• Null Hypothesis (H0): All βj=0
• Alternative Hypothesis (Ha): At least one βj≠0
• F-Statistic:

F=(SST−SSR)/pSSR/(n−p−1)

• Degrees of Freedom: p and n−p−1

Diagnosing Multicollinearity

Variance Inflation Factor (VIF)

Measures how much the variance of an estimated coefficient increases due to multicollinearity:

VIFj=11−R2j

• R2j is the R2 from regressing xj on the other predictors.
• A VIF value greater than 5 or 10 indicates high multicollinearity.

Remedies

• Remove variables by excluding one of the correlated variables from the model.
• Combine variables to create composite or aggregated variables that reduce redundancy.
• Regularization techniques, such as ridge regression, can be applied to handle multicollinearity effectively.

Assumption Diagnostics

Residual Analysis

• Plot residuals vs. fitted values to check for any patterns that may indicate non-linearity or heteroscedasticity.
• Normal Q-Q plot to assess whether the residuals follow a normal distribution.

Durbin-Watson Test

Checks for autocorrelation in residuals:

D=∑ni=2(ei−ei−1)2∑ni=1e2i

Values close to 2 indicate no autocorrelation.

Extensions

Interaction Terms

Include products of independent variables to model interactions:

y=β0+β1x1+β2x2+β3(x1x2)+ε

Polynomial Regression

Model non-linear relationships by including polynomial terms:

y=β0+β1x+β2x2+⋯+βkxk+ε

Regularization Techniques

Ridge Regression: Adds penalty for large coefficients.

ˆβridge=(X⊤X+λI)−1X⊤y

Lasso Regression: Encourages sparsity in coefficients.

$$ \text{Minimize } \sum_{i=1}^n (y_i - \hat{y}i)^2 + \lambda \sum{j=1}^p |\beta_j| $$

Example: Multicollinearity Between Variables

Suppose we have the following data on the number of hours studied (x1), the number of practice exams taken (x2), and the test scores (y):

Observations

Before proceeding, it's important to notice that x2 is directly proportional to x1:

x2=12x1

This means that there is perfect multicollinearity between x1 and x2. In multiple linear regression, perfect multicollinearity causes the design matrix to be singular, making it impossible to uniquely estimate the regression coefficients for x1 and x2.

Implications of Multicollinearity

When independent variables are perfectly correlated, the matrix X⊤X (where X is the design matrix) becomes singular (non-invertible). This prevents us from calculating the coefficients using the normal equation:

ˆβ=(X⊤X)−1X⊤y

Adjusted Approach

Given the perfect linear relationship between x1 and x2, we can simplify the model by combining x1 and x2 into a single variable or by using only one of them in the regression.

Simplifying the Model

Since x2=12x1, we can express y solely in terms of x1:

y=β0+β1x1+β2(12x1)=β0+(β1+β22)x1

Let γ=β1+β22. The model becomes:

y=β0+γx1

Now, we can proceed with a simple linear regression of y on x1.

Step-by-Step Calculation

1. Calculate the Means

Compute the mean of x1 and y:

ˉx1=2+4+6+84=204=5

ˉy=50+60+70+804=2604=65

2. Calculate the Sum of Squares

Compute the sum of squares for x1 and the cross-product of x1 and y:

SSx1x1=n∑i=1(x1i−ˉx1)2=(2−5)2+(4−5)2+(6−5)2+(8−5)2=20

SSx1y=n∑i=1(x1i−ˉx1)(yi−ˉy)=(2−5)(50−65)+(4−5)(60−65)+(6−5)(70−65)+(8−5)(80−65)=100

3. Calculate the Regression Coefficients

Compute the slope (ˆγ) and intercept (ˆβ0):

ˆγ=SSx1ySSx1x1=10020=5

ˆβ0=ˉy−ˆγˉx1=65−(5)(5)=40

4. Write the Regression Equation

The best-fitting line is:

ˆy=40+5x1

5. Verify the Model with the Data

Compute the predicted y values using the regression equation:

For x1=2:

ˆy=40+5(2)=50

For x1=4:

ˆy=40+5(4)=60

For x1=6:

ˆy=40+5(6)=70

For x1=8:

ˆy=40+5(8)=80

The predicted values match the actual test scores perfectly.

Plot:

Example: No Perfect Multicollinearity

Suppose we have the following data on the number of hours studied (x1), attendance rate (x2) as a percentage, and test scores (y):

Objective

We aim to fit a multiple linear regression model of the form:

y=β0+β1x1+β2x2+ε

where:

• y is the dependent variable (test score),
• x1 is the number of hours studied,
• x2 is the attendance rate,
• ε is the error term.

Step-by-Step Calculation

1. Organize the Data

First, compute the necessary sums and products:

2. Compute the Means

$$ \bar{x}1 = \frac{\sum x{1i}}{n} = \frac{26}{5} = 5.2 $$

$$ \bar{x}2 = \frac{\sum x{2i}}{n} = \frac{395}{5} = 79 $$

ˉy=∑yin=3905=78

3. Compute Sum of Squares and Cross Products

Sum of Squares for x1:

SSx1x1=∑x21i−nˉx21=168−5(5.2)2=168−135.2=32.8

Sum of Squares for x2:

SSx2x2=∑x22i−nˉx22=32,025−5(79)2=32,025−31,205=820

Sum of Cross Products between x1 and x2:

SSx1x2=∑x1ix2i−nˉx1ˉx2=2,165−5(5.2)(79)=2,165−2,054=111

Sum of Cross Products between x1 and y:

SSx1y=∑x1iyi−nˉx1ˉy=2,165−5(5.2)(78)=2,165−2,028=137

Sum of Cross Products between x2 and y:

SSx2y=∑x2iyi−nˉx2ˉy=31,325−5(79)(78)=31,325−30,810=515

4. Compute the Regression Coefficients

We use the formulas for multiple linear regression coefficients:

Denominator (Determinant):

D=SSx1x1SSx2x2−(SSx1x2)2=(32.8)(820)−(111)2=26,896−12,321=14,575

Coefficient ˆβ1:

$$ \hat{\beta}1 = \frac{SS{x_1 y} SS_{x_2 x_2} - SS_{x_1 x_2} SS_{x_2 y}}{D} $$

ˆβ1=(137)(820)−(111)(515)14,575=112,340−57,16514,575=55,17514,575≈3.785

Coefficient ˆβ2:

$$ \hat{\beta}2 = \frac{SS{x_2 y} SS_{x_1 x_1} - SS_{x_1 x_2} SS_{x_1 y}}{D} $$

ˆβ2=(515)(32.8)−(111)(137)14,575=16,892−15,20714,575=1,68514,575≈0.116

Intercept ˆβ0:

ˆβ0=ˉy−ˆβ1ˉx1−ˆβ2ˉx2

ˆβ0=78−(3.785)(5.2)−(0.116)(79)=78−19.642−9.164≈49.194

5. Write the Regression Equation

The estimated multiple linear regression model is:

ˆy=49.194+3.785x1+0.116x2

6. Interpret the Coefficients

• The intercept (β0=49.194) represents the estimated test score when both hours studied and attendance rate are zero. While this value may not have practical significance, it is necessary for constructing the mathematical model.
• The coefficient of x1 (β1=3.785) suggests that for each additional hour studied, the test score increases by approximately 3.785 points, assuming the attendance rate remains constant.
• The coefficient of x2 (β2=0.116) indicates that for each 1% increase in attendance rate, the test score rises by approximately 0.116 points, holding hours studied constant.

7. Verify the Model with the Data

Compute the predicted test scores (ˆyi) and residuals (ei=yi−ˆyi).

For Student 1:

ˆy1=49.194+3.785(2)+0.116(70)=49.194+7.570+8.120=64.884

e1=y1−ˆy1=65−64.884=0.116

For Student 2:

ˆy2=49.194+3.785(3)+0.116(80)=49.194+11.355+9.280=69.829

e2=70−69.829=0.171

For Student 3:

ˆy3=49.194+3.785(5)+0.116(60)=49.194+18.925+6.960=75.079

e3=75−75.079=−0.079

For Student 4:

ˆy4=49.194+3.785(7)+0.116(90)=49.194+26.495+10.440=86.129

e4=85−86.129=−1.129

For Student 5:

ˆy5=49.194+3.785(9)+0.116(95)=49.194+34.065+11.020=94.279

e5=95−94.279=0.721

The residuals are small, indicating a good fit of the model to the data.

Plot:

Checking for Multicollinearity

Compute the correlation coefficient between x1 and x2:

rx1x2=SSx1x2√SSx1x1×SSx2x2

rx1x2=111√32.8×820=111√26,896=111164≈0.677

A correlation coefficient of approximately 0.677 indicates a moderate correlation, not perfect multicollinearity.