Formatting ordinary least squares equations into matrix form can be a very efficient way of solving for a least squares regression line. This is because, instead of taking every value individually, the values are taken as a whole column. Try to apply what you’ve learned from our other guides to the problem below. If you struggle to come up with a solution or are learning about this concept for the first time, read through this guide.

 

Problem 8

A shop owner is interested in understanding the demand of certain goods in her store based off of the price. In order to help the store owner, you’re tasked with conducting a simple regression analysis. Given the following data set, your first task is to explain how to format these observations into matrices along with the benefits of using matrices in statistics.

 

Price Demand
120 110
125 100
130 90
135 45
140 20

 

The best Maths tutors available
1st lesson free!
Intasar
4.9
4.9 (23 reviews)
Intasar
£42
/h
1st lesson free!
Matthew
5
5 (17 reviews)
Matthew
£25
/h
1st lesson free!
Dr. Kritaphat
4.9
4.9 (6 reviews)
Dr. Kritaphat
£49
/h
1st lesson free!
Paolo
4.9
4.9 (11 reviews)
Paolo
£25
/h
1st lesson free!
Ayush
5
5 (28 reviews)
Ayush
£60
/h
1st lesson free!
Petar
4.9
4.9 (9 reviews)
Petar
£27
/h
1st lesson free!
Rajan
4.9
4.9 (11 reviews)
Rajan
£15
/h
1st lesson free!
Farooq
5
5 (13 reviews)
Farooq
£35
/h
1st lesson free!
Intasar
4.9
4.9 (23 reviews)
Intasar
£42
/h
1st lesson free!
Matthew
5
5 (17 reviews)
Matthew
£25
/h
1st lesson free!
Dr. Kritaphat
4.9
4.9 (6 reviews)
Dr. Kritaphat
£49
/h
1st lesson free!
Paolo
4.9
4.9 (11 reviews)
Paolo
£25
/h
1st lesson free!
Ayush
5
5 (28 reviews)
Ayush
£60
/h
1st lesson free!
Petar
4.9
4.9 (9 reviews)
Petar
£27
/h
1st lesson free!
Rajan
4.9
4.9 (11 reviews)
Rajan
£15
/h
1st lesson free!
Farooq
5
5 (13 reviews)
Farooq
£35
/h
First Lesson Free>

OLS Linear Equation

Recall that ordinary least squares, or OLS, is a type of regression analysis that strives to reduce the sum of squared residuals. OLS can take on two forms, summarized in the table below.

 

Simple Linear Regression Multiple Linear Regression
SLR MLR
\hat{y} = b_{0} + b_{1}x_{1} + u_{i} \hat{y} = b_{0} + b_{1}x_{1} + … + b_{n}x_{n} + u_{i}
One dependent variable One dependent variable
One independent variable Two or more independent variables

 

mlr_equation

 

slr_equation

 

To understand OLS regression, you have to understand the formulas involved. There are two major formulas, which are the least squares linear equation and the residual formula.

 

\hat{y} = b_{0} + b_{1}x_{1} + … + b_{n}x_{n} + u_{i} The least squares linear equation
b_{0} The y-intercept, the value of y when all x’s are zero
b_{1}, b_{2}...b_{n} The regression coefficients, which tell us how one independent variable relates to y
u_{i} The estimation of the error term, otherwise known as the residual

 

To understand the reason why OLS tries to reduce the sum of the squared error, look at the image below.

 

residual_estimation

 

The residual is the difference between the actual observed value y_{i} given by an x_{i} and the predicted value \hat{y}, also known as y-hat, given by the same x_{i}. The best line, naturally, would be the regression line with the least distance between the actual observed y values and the predicted y values. However, how can you compare residuals when they have different signs?

 

Example Sign Interpretation
y_{i} - \hat{y} \rightarrow 45 - 30 = 15 positive Overestimated
y_{i} - \hat{y} \rightarrow 20 - 35 = -15 negative Underestimated

 

As you can see, while the magnitudes are both 15, if we didn’t square the residuals the negative sign would deflate the total sum. This is why residuals are squared.

 

OLS Matrix Equation

If we were to solve this equation using OLS, we would have to utilize the steps and equations summarized below.

 

Step Description Formula
1 Write down the OLS equation \hat{y} = b_{0} + b_{1}x_{1} + … + b_{n}x_{n} + u_{i}
2 Find the slope \bar{y} - b_{1}x_{1} - b_{2}x_{2}
3 Find all \sum x_{i}^2 \sum x_{i}^2 - \frac{(\sum x_{i})^2}{N}
4 Find all \sum x_{i}y \sum x_{i}y - \frac{(\sum x_{i})(\sum y)}{N}
5 Find all parameters b_{n} \frac{ (\sum x_{2}^2) (\sum x_{1}y) - (\sum x_{1}x_{2})(\sum x_{2}y) }{ (\sum x_{1}^2)(\sum x_{2}^2) - (\sum x_{1} x_{2})^2 }
\frac{ (\sum x_{1}^2) (\sum x_{2}y) - (\sum x_{1}x_{2})(\sum x_{1}y) }{ (\sum x_{1}^2)(\sum x_{2}^2) - (\sum x_{1} x_{2})^2 }
...

 

This can be time consuming, especially if we had more than just two independent variables. This is where matrices come into play. Think about the points (x_{i}, y_{i}) in your data as a system of equations.

 

Points Equation
(x_{1}, y_{1}) y_{1} = (b_{0} + b_{1}x_{1}) + u_{1}
(x_{2}, y_{2}) y_{2} = (b_{0} + b_{1}x_{2}) + u_{2}
... ...
(x_{n}, y_{n}) y_{n} = (b_{0} + b_{1}x_{n}) + u_{n}

 

Taking this system of equations, you can reformulate this data into matrix form.

 

matrix_regression

 

In this form, we only have two steps, summarized below.

 

Step Description Formula
1 Write down the OLS equation Y = XA + U
2 Solve for the matrix A A

 

Solving OLS Matrix Equation

By multiplying the matrices out, you can see the resulting system of equations from earlier. Take the first two entries, as an example.

 

matrix_regression_example

 

This would multiply out to be the following.

 

Y = XA + U \rightarrow y_{1} = (1*b_{0}) + (x_{1}*b_{1}) + u_{1} y_{1} = (b_{0} + b_{1}x_{1}) + u_{1}
Y = XA + U \rightarrow y_{2} = (1*b_{0}) + (x_{2}*b_{1}) + u_{1} y_{2} = (b_{0} + b_{1}x_{2}) + u_{2}

 

Here, we would like to solve for A as it contains the y-intercept and slope information. Because matrices can’t just be moved around on either side of the equals sign, we have to perform special operations known as taking the transpose and inverse of a matrix, resulting in the following formula.

 

matrix_regression_solve

 

The transpose of a matrix is simply flipping a matrix from, for example, being 2x3 to 3x2, like below.

 

transpose_matrix

 

The inverse of a matrix, on the other hand, involves elementary row operations to get from AI ot IA^{-1}.

 

inverse_3x3_matrix

 

OLS Matrix Equation Example

In order to get used to matrix notation, let’s go through an example. You have data on height and heart beats per minute, which are illustrated below.

 

Height Heart Beat
160 55
169 50
174 70
180 80

 

You know this could be written in the following equations:

 

(160, 55) 55 = b_{0} + b_{1}160
(169, 50) 50 = b_{0} + b_{1}169
(174, 70) 70 = b_{0} + b_{1}174
(180, 80) 80 = b_{0} + b_{1}180

 

Instead, we can now write them in matrix notation.

 

regression_matrix_formula

OLS Matrix Equation Problem

In this problem, you were asked to format the data set into matrix multiplication. First, let’s start by separating the data into a system of equations. Doing this will help us visualize which numbers we have to place in each matrix.

 

Y X Formula
120 110 120 = b_{0} + b_{1}110
125 100 125 = b_{0} + b_{1}100
130 90 130 = b_{0} + b_{1}90
135 45 130 = b_{0} + b_{1}45
140 20 140 = b_{0} + b_{1}20

 

Now that we have the data formatted this way, it’s easy to see what to put in which matrix. Now, we just follow the formula from above, where each matrix contains the different variables and/or parameters necessary. Below, you'll see what this should look like.

 

regression_matrix_formula

 

The benefit to this is that it will be a lot easier to solve by hand, and computationally, than without matrices. As you can see, instead of having to find the solutions to the formulas of about 5 or more metrics, we can now just solve for A through less complicated matrix multiplication.

Need a Maths teacher?

Did you like the article?

1 Star2 Stars3 Stars4 Stars5 Stars 5.00/5 - 1 vote(s)
Loading...

Danica

Located in Prague and studying to become a Statistician, I enjoy reading, writing, and exploring new places.