Linear Curve Fitting

Most curve fitting is done using linear combinations of a set of basis functions. A linear combination of a set of functions, f₁, f₂..., f_n is a function whose value is given by

f(x) = a₁f₁(x) + a₂f₂(x) + ... + a_nf_n(x)

where a₁, a₂..., a_n are the coefficients. The advantage of using this type of approximation is that the math is relatively simple. The solution can be found by solving a standard linear algebra problem. In many cases, this can be done very efficiently.

The basis functions f₁, f₂..., f_n that are combined to create the approximation together form a function basis. A linear combination is therefore defined by the function basis and the coefficients of the basis functions in the combination.

Perhaps the most well-known example of a linear combination of basis functions is provided by polynomials. A polynomial is a function that is a combination of the argument raised to different integer powers:

f(x) = a₀ + a₁x + ... + a_nxⁿ

The integer powers are the basis functions. Chebyshev polynomials form an alternate basis for the polynomials. They can be defined mathematically by the identity

T_n ( cos x) = cos nx

Their mathematical properties make them particularly suited for approximating functions.

In the Extreme Optimization Numerical Libraries for .NET, function bases are implemented by the abstract FunctionBasis class and its derived classes. Linear combinations are implemented using the LinearCombination class and its derived classes, including Polynomial and ChebyshevSeries. The table below summarizes the classes that implement function bases and the corresponding linear combinations.

Many classes expose static methods that allow you to obtain a least squares fit. The main drawback of the methods discussed in this section is that they don't offer much control over the calculation of the least squares fit. There is also no easy way to obtain information about the quality of the fit. However, they may be useful because they offer a very quick way to get a result.

The LeastSquaresFit method of the Polynomial class constructs a Polynomial that is the least squares fit of a specified degree through a given set of points.

Vector<double> xValues = Vector.Create(7, i =>  Math.Cos(i * Constants.Pi / 6));
Vector<double> yValues = Vector.Create(7, i => -Math.Sin(i * Constants.Pi / 6));
Polynomial p = Polynomial.LeastSquaresFit(xValues, yValues, 4);

Dim xValues As Vector(Of Double) =
    Vector.Create(7, Function(i) Math.Cos(i * Constants.Pi / 6))
Dim yValues As Vector(Of Double) =
    Vector.Create(7, Function(i) -Math.Sin(i * Constants.Pi / 6))
Dim p As Polynomial = Polynomial.LeastSquaresFit(xValues, yValues, 4)

let xValues = Vector.Create(7, fun i ->  cos((float i) * Constants.Pi / 6.0))
let yValues = Vector.Create(7, fun i -> -sin((float i) * Constants.Pi / 6.0))
let p = Polynomial.LeastSquaresFit(xValues, yValues, 4)

To fit a line, set the degree equal to 1.

You can also create a function basis, and obtain the least squares fit using the functions in the function basis for a set of data points using the LeastSquaresFit method. It returns a LinearCombination curve. This allows you to fit data to an arbitrary set of functions. You can also specify weights for each of the data points. The weights specify how much weight each data point should have in the sum of squares that is to be minimized.

The following example fits a set of data points to a combination of the sine, cosine, and exponential functions.

Vector<double> xValues = Vector.Create(0.0, 1.0, 2.0, 3.0, 4.0);
Vector<double> yValues = Vector.Create(4.0, 1.0, 4.0, 10.0, 8.0);
GeneralFunctionBasis basis = new GeneralFunctionBasis(Math.Sin, Math.Cos, Math.Exp);
LinearCombination f = basis.LeastSquaresFit(xValues, yValues);

Dim xValues = Vector.Create(0.0, 1.0, 2.0, 3.0, 4.0)
Dim yValues = Vector.Create(4.0, 1.0, 4.0, 10.0, 8.0)
Dim basis As GeneralFunctionBasis =
    New GeneralFunctionBasis(AddressOf Math.Sin,
                                AddressOf Math.Cos, AddressOf Math.Exp)
Dim f As LinearCombination = basis.LeastSquaresFit(xValues, yValues)

let xValues = Vector.Create(0.0, 1.0, 2.0, 3.0, 4.0)
let yValues = Vector.Create(4.0, 1.0, 4.0, 10.0, 8.0)
let basis = new GeneralFunctionBasis(
                Func<_,_>(Math.Sin), 
                Func<_,_>(Math.Cos), 
                Func<_,_>(Math.Exp))
let f = basis.LeastSquaresFit(xValues, yValues)

The LinearCurveFitter class performs a linear least squares fit. It offers greater control over the procedure, and gives more extensive results.

To perform the fit, a LinearCurveFitter needs data points, and a curve to fit. You must set the Curve property to an instance of a LinearCombination object. A LinearCombination object can represent any combination of functions. Other objects, for example of type Polynomial or ChebyshevSeries are allowed, as they inherit from LinearCombination.

The data is supplied as VectorT objects. The XValues and YValues properties specify the X and Y values of the data points, respectively.

By default, the fit is unweighted. All weights are set equal to 1. Weights can be specified in one of two ways. The first way is to set the WeightVector property to a vector with as many elements as there are data points. Each component of the vector specifies the weight of the corresponding data point.

Alternatively, the WeightFunction can be used to calculate the weights based on the data points. The WeightFunctions class contains a number of predefined weight functions listed in the table below. In the descriptions, x and y stand for the x and y values of each data point.

The Fit method performs the actual least squares calculation, and adjusts the parameters of the Curve to correspond to the least squares fit. The BestFitParameters property returns a VectorT containing the parameters of the curve. The GetStandardDeviations returns a VectorT containing the standard deviations of each parameter as estimated in the least squares calculation.

In the first example, we fit some data to a quadratic polynomial.

Vector<double> loadData = Vector.Create<double>(/*...*/);
Vector<double> deflectionData = Vector.Create<double>(/*...*/);
Polynomial poly = new Polynomial(2);
LinearCurveFitter fitter = new LinearCurveFitter();
fitter.Curve = poly;
fitter.XValues = loadData;
fitter.YValues = deflectionData;
fitter.Fit();

Dim loadData = Vector.Create(Of Double)()
Dim deflectionData = Vector.Create(Of Double)()
Dim poly As Polynomial = New Polynomial(2)
Dim fitter As LinearCurveFitter = New LinearCurveFitter()
fitter.Curve = poly
fitter.XValues = loadData
fitter.YValues = deflectionData
fitter.Fit()

let loadData = Vector.Create<double>((*...*))
let deflectionData = Vector.Create<double>((*...*))
let poly = new Polynomial(2)
let fitter = new LinearCurveFitter()
fitter.Curve <- poly
fitter.XValues <- loadData
fitter.YValues <- deflectionData
fitter.Fit()

First, the data is loaded into the loadData and deflectionData vectors. We then create the polynomial, poly. That gives us what we need to create the LinearCurveFitter objects. We set its properties, and perform the fit.

In the next example, we fit measurements of the conductance of copper to a theoretical model:

We have measurements of k(T) for a series of temperatures. To estimate the parameters c₁ and c₂, we need to take the following steps:

1. Transform the dependent variable k(T) into y = 1/k(T). The expression for y is then linear in the basis functions 1/T and T².
2. Create a function basis with these two basis functions.
3. Create a LinearCombination curve using this function basis.
4. Create the LinearCurveFitter object and set its properties.
5. Perform the fit and report the results.

The code below performs these steps. It assumes we have created two vector variables, temperature and conductance, that contain the data of the observations.

Vector<double> y = Vector.Reciprocal(conductance);
Func<double,double>[] basisFunctions = new Func<double,double>[]
    { x => 1/x, x => x * x };
GeneralFunctionBasis basis = new GeneralFunctionBasis(basisFunctions);
LinearCombination curve = new LinearCombination(basis);
LinearCurveFitter fitter = new LinearCurveFitter();
fitter.Curve = curve;
fitter.XValues = temperature;
fitter.YValues = y;
fitter.Fit();
Vector<double> solution = fitter.BestFitParameters;
Vector<double> s = fitter.GetStandardDeviations();
Console.WriteLine("c1: {0,20:E10} {1,20:E10}", solution[0], s[0]);
Console.WriteLine("c2: {0,20:E10} {1,20:E10}", solution[1], s[1]);

Dim y = Vector.Reciprocal(conductance)
Dim basisFunctions As Func(Of Double, Double)() =
     {Function(x) 1 / x, Function(x) x * x}
Dim basis As GeneralFunctionBasis = New GeneralFunctionBasis(basisFunctions)
Dim curve As LinearCombination = New LinearCombination(basis)
Dim fitter As LinearCurveFitter = New LinearCurveFitter()
fitter.Curve = curve
fitter.XValues = temperature
fitter.YValues = y
fitter.Fit()
Dim solution = fitter.BestFitParameters
Dim s = fitter.GetStandardDeviations()
Console.WriteLine("c1: 0,20:E10 1,20:E10", solution(0), s(0))
Console.WriteLine("c2: 0,20:E10 1,20:E10", solution(1), s(1))

let y = Vector.Reciprocal(conductance)
let basisFunctions = [| 
    Func<_,_>(fun x -> 1.0 / x);
    Func<_,_>(fun x -> x * x) |]
let basis = new GeneralFunctionBasis(basisFunctions)
let curve = new LinearCombination(basis)
let fitter = new LinearCurveFitter()
fitter.Curve <- curve
fitter.XValues <- temperature
fitter.YValues <- y
fitter.Fit() |> ignore
let solution = fitter.BestFitParameters
let s = fitter.GetStandardDeviations()
Console.WriteLine("c1: 0,20:E10 1,20:E10", solution.[0], s.[0])
Console.WriteLine("c2: 0,20:E10 1,20:E10", solution.[1], s.[1])