Download Linear Regression: Fitting a Straight Line and Measuring Dependence between Parameters and more Study notes Environmental Science in PDF only on Docsity!
Linear Regression - Fitting a Straight Line
In the past two lectures we have focused on the statistics of repeated measurements of a single property. However, in the geoscience we generally measuring multiple variables - even if we think we are only measuring a single variable (e.g., temperature, grain size), it will generally be referenced to time and/or spatial coordinate(s). A very common problem in data analysis is looking for relationships between different parameters and fitting lines or surfaces to data. The simplest example is fitting a straight line and we will discuss that here.
Most of the material for this lecture can be found in Chapters 1 and 4 of Paul Wessel’s notes.
Covariance and correlation
If we measure properties xi and yi for i = 1, 2, …, n , we can write the sample variances for x and y as
sx^2 = n^1! 1 i = 1 ( xi! x )^2
n
"^ (1)
and
sy^2 = n^1! 1 i = 1 ( yi! y )^2
n
"^ (2)
We define the covariance between x and y , sxy as
sx^2 y^ = n^1! 1 i = 1 ( xi! x ) ( yi! y )
n
"^ (3)
The covariance tells us how x and y values depend on each other. The correlation coefficient, r is a normalized version of the covariance and is given by
r = sxy
2 sxsy^ (4) The correlation coefficient is constrained for fall in the range ±1. A value of +1 tells us that the points ( xi , yi ) define a straight line with a positive slope (Figure 1). A value of - 1 tells us that the points ( xi , yi ) define a straight line with a negative slope. A value of 0 shows that there is no dependence of y on x or vice versa (i.e., no correlation) (Figure 1).
The quantity r^2 is the coefficient of determination and it is a measure of the fraction of the variance of y that can be attributed to a relationship with x.
It is important to note that r measures the strength of the linear relationship between x and y but a high value of | r | does not necessarily imply a cause and effect relationship or that the two variables are linearly related. It is easy to devise non-linear relationships that give a high correlation coefficient. It is important to look at the data and use common sense.
Least-squares straight-line fitting
The process of fitting a straight line is one of the simplest examples of an inverse problem. If we fit our y and x data with a relationship
y = a + bx (5) we can define can use the χ^2 statistic to measure the misfit
! 2 ( a , b ) = yi^ "^ # a^ "^ bxi
i
%^ &
(^ )
2 i = 1
n
*^ (6)
where σ i are the uncertainties (or estimates of the uncertainties) in the y values. Our goal is to find the values of a and b that mimimize χ 2. To do this we find take the partial derivates of χ 2 with respect to a and b and solve for the values of a and b at which they are both zero ! " 2 ! a =^ #^2
yi # a # bxi $ (^) i^2
&^ '
i = 1 )^ *
n
+ =^0
! b =^ #^2 xi
yi # a # bxi $ (^) i^2
&^ '
i = 1 )^ *
n
+ =^0
If we define S = (^)!^1 i = (^1) i^2
n
" ,^ Sx =^ i = 1! x ii 2
n
" ,^ Sy =^! y ii 2 ,^ Sxx =^ xi
2 i = 1!^ i^2
n
" ,^ Sxy =^ i = 1 x! i^ y i 2 i
n
i = 1 "
n
"^ (8)
then equation (7) reduces to
aS + bSx = Sy aSx + bSxx = Sxy^ (9) The solution is a = Sxx^ Sy^ "!^ Sx^ Sxy b = SSxy^! "^ Sx^ Sy
with ! = SSxx " Sx^2 (11) We can also estimate the uncertainties in a and b. To do this we sum the variance in a and b resulting from the variance in each of the y values. This can be written mathematically
sa^2 =! (^) i^2 "" ya i
$^ %
i = 1 '^ (
n
2
sb^2 =! (^) i^2 "" yb i
$^ %
i = 1 '^ (
n
After substituting derivatives obtained from equation (10) and a fair amount of manipulation we get sa^2 = S! xx sb^2 = (^)! S
Robust Line Fitting
In the least squares line fitting process in which we shall assume all the data have the same uncertainty we seek to minimize Minimize a , b
E = i = 1 ( yi! a! bxi )^2
n
" =^ i = 1 ri^2
n
"^ (20)
This process is sensitive to outliers, particularly so when the outliers lie near the lower or upper limits of the range of xi (Figure 2). The breakdown point for the least squares line fit (L 2 regression) is 1/n. We can overcome this problem to a small extent by minimizing the sum of the absolute misfits (L 1 regression) Minimize a , b
E = (^) i = 1 ri
n
!^ (21)
but the L 1 norm also has a breakdown point of 1/ n.
A robust approach with a breakdown point of ½ is to minimize the median misfit. Minimize a , b
median ri = median yi! a! bxi (22)
This is equivalent to finding the narrowest strip that encloses half the points. The only way to do this, is by a systematic search through different values of b. For each value of b we calculate yi - bxi , and then find the value of a that minimizes the median of | yi - bxi - a |. One then choses the a and b value that gives the minimum median among all the values of b analyzed.
We can this robust statistical method to find and eliminate outliers yi! a! bxi Median yi! a! bxi^ >^ zcut^ (23) where a value of zcut =4.45 is equivalent to 3 standard deviations for a normal distribution. Once the outliers are eliminated, we can apply the least squares line fitting approach.
