**Home** / **find the least squares solution calculator** / Least Squares - UMD

Least Squares

Justin Wyss-Gallifent

July 21, 2021

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

3.2 Reminder - Solutions and Column Space . . . . . . . . . . . . . . 2

3.3 The Intuition and Theory . . . . . . . . . . . . . . . . . . . . . . 3

3.4 Theory: Least-Squares Solution . . . . . . . . . . . . . . . . . . . 4

3.5 Practical: Least-Squares Solution . . . . . . . . . . . . . . . . . . 5

3.6 Picture of a Simple Case . . . . . . . . . . . . . . . . . . . . . . . 8

3.7 Matlab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1

3.1 Introduction

Let's go back to the matrix equation

Ax? = ?b

We know that a unique solution exists if A is invertible and if A is not invertible

then there are either no solutions or infinitely many solutions.

Specifically the question that we'd like to address here is what can we do if there

are no solutions at all? One answer might be to just stop, however maybe we

could ask the question - what's the nearest solution we could find?

In other words if we can't find x? so that Ax? = ?b, can we find some x? so that Ax?

is as close as possible to ?b?

More rigorously can we find some x^ such that:

For all x? we have ||Ax^ - ?b|| ||Ax? - ?b||

3.2 Reminder - Solutions and Column Space

First let's recall:

Definition 3.2.0.1. Given an n ? m matrix A the column space of A is the

subspace of Rn given by:

Col(A) = span {Columns of A}

Fact 3.2.0.1. The column space of A is exactly the vectors ?b such that Ax? = ?b

has at least one solution.

To reinforce this, a simple example will do:

Example 3.1. The equation

12 4

0 3 x? = 5

-1 0 6

2

has a solution if and only if there are x1, x2 with

12 4

0 3 x? = 5

-1 0 6

4

1 2 x1 = 5

0 3

-1 0 x2 6

1x1 + 2x2 4

0x1 + 3x2 = 5

-1x1 + 0x2 6

1 24

x1 0 + x2 3 = 5

-1 0 6

4

1 2

5 col 0 , 3

6 -1 0

3.3 The Intuition and Theory

So the situation we're in is that ?b is not in Col(A) and we wish to find x^ so that

Ax^, which is in Col(A), is as close as possible to ?b.

Here's a picture of the situation:

?b

Close As Possible!

o? Col(A)

This picture suggests that we can obtain a solution by projecting ?b onto Col(A)

to get PrCol(A)?b and then finding x^ to solve the equation:

Ax^ = PrCol(A)?b

Assuming this is correct, the problem with this approach, practically, is that

3

How to calculate least squares line?Method of Least Squares. In Correlation we study the linear correlation between two random variables x and y. We now look at the line in the xy plane that best fits the data (x1, y1), …, (xn, yn). Recall that the equation for a straight line is y = bx + a, where. b = the slope of the line.

Title:

Subject:

Keywords:

Author:

Creator: LaTeX with hyperref

Producer: pdfTeX-1.40.21

CreationDate: Wed Jul 21 07:59:01 2021

ModDate: Wed Jul 21 07:59:01 2021

Tagged: no

Form: none

Pages: 12

Encrypted: no

Page size: 612 x 792 pts (letter) (rotated 0 degrees)

File size: 181431 bytes

Optimized: no

PDF version: 1.5