Extreme Optimization > Mathematics Library for .NET > User's Guide > Current Page > Solving Least Squares Problems

Extreme Optimization Mathematics Library for .NET

Solving Least Squares Problems

The Extreme Optimization Mathematics Library for .NET supports a simple mechanism for solving linear least squares problems.

The LeastSquaresSolver Class

The LeastSquaresSolver class implements the calculation of least squares solutions. The constructor takes two arguments: the matrix of observations, and the vector of outcomes:

Matrix a = new GeneralMatrix(6, 4, new double[]
    {
        1, 1, 1, 1, 1, 1,
        1, 2, 3, 4, 5, 6,
        1, 4, 9, 16, 25, 36,
        1, 2, 1, 2, 1, 2
    });
Vector b = new GeneralVector(1, 3, 6, 11, 15, 21);
LeastSquaresSolver solver = new LeastSquaresSolver(a, b);

Dim a As Matrix = New GeneralMatrix(6, 4, New Double() _
    { _
        1, 1, 1, 1, 1, 1, _
        1, 2, 3, 4, 5, 6, _
        1, 4, 9, 16, 25, 36, _
        1, 2, 1, 2, 1, 2 _
    })
Dim b As Vector = New GeneralVector(1, 3, 6, 11, 15, 21)
Dim solver As LeastSquaresSolver = New LeastSquaresSolver(a, b)

Least squares problems can be solved using a variety of techniques. The method to use is specified by the SolutionMethod property. This property is of type LeastSquaresSolutionMethod and can take on the following values:

The Solve method performs the actual calculation. This method returns a Vector containing the least squares solution. This vector is also available through the Solution property.

solver.SolutionMethod = LeastSquaresSolutionMethod.QRDecomposition;
// The Solve method calculates the solution:
Vector x = solver.Solve();
Console.WriteLine("x = {0}", x.ToString("F4"));
// The Solution property also returns the solution:
Console.WriteLine("x = {0}", solver.Solution.ToString("F4"));

solver.SolutionMethod = LeastSquaresSolutionMethod.QRDecomposition
' The Solve method calculates the solution:
Dim x As Vector = solver.Solve()
Console.WriteLine("x = {0}", x.ToString("F4"))
' The Solution property also returns the solution:
Console.WriteLine("x = {0}", solver.Solution.ToString("F4"))

Various other methods let you retrieve more information about the least squares solution. The GetPredictions method returns the vector of the outcomes predicted by the solution. The GetResidues method returns the residues: the difference between the predicted and actual outcome.

By default, the least squares solver uses an algorithm based on the QR decomposition of the matrix of observations. Depending on the numerical properties of the observation matrix, it may be acceptable to calculate the solution using the normal equations. This is faster, especially for problems with many more observations than variables, but the numerical accuracy tends to be questionable for larger numbers of variables.

Methods using a complete orthogonal decomposition and the singular value decomposition will be supported in the future.

Directly Solving the Normal Equations

In some instances, it is convenient and cheap to accumulate the normal equations directly. If the condition of the observation matrix is good enough, a least squares solution may be obtained by solving the accumulated equations. The following example illustrates this procedure:

SymmetricMatrix aTa = SymmetricMatrix.FromOuterProduct(a);
Vector aTb = b * a;
Vector x = aTa.Solve(aTb);
Console.WriteLine("x = {0}", x.ToString("F4"));

Dim aTa As SymmetricMatrix = SymmetricMatrix.FromOuterProduct(a);
Dim aTb As Vector = b * a
Dim x As Vector = aTa.Solve(aTb)
Console.WriteLine("x = {0}", x.ToString("F4"))

References

G. H. Golub, C. F. Van Loan, Matrix Computations (3rd Ed), Johns Hopkins University Press, 1996.

Copyright 2004-2008, Extreme Optimization. All rights reserved.
Extreme Optimization, Complexity made simple, M#, and M Sharp  are trademarks of ExoAnalytics Inc.
Microsoft, Visual C#, Visual Basic, Visual Studio, Visual Studio.NET, and the Visual Studio Logo are registered trademarks of Microsoft Corporation