Linear Regression: Properties and Simple Linear Regression Example, Study notes of Statistics

Download Linear Regression: Properties and Simple Linear Regression Example and more Study notes Statistics in PDF only on Docsity!

ioc.pdf

REGRESSION MODELLING

David M. Lane. et al. Introduction to Statistics : pp. 462

ioc.pdf

Contents

1 What is Linear Regression?

2 Properties of a Regression Line

3 Simple Linear Regression Example

4 Regression models in R

ioc.pdf

Introduction

Recall that In a cause and eect relationship, the independent variable is the cause, and the dependent variable is the eect.

Linear Regression is used predict or estimate the value of a dependent variable by modelling it against one or more independent variables. The variables must be pairwise, continuous and are assumed to have a linear relationship between them. This technique is widely popular in predictive analysis.

Here, we focus on the case where there is only one independent variable. This is called simple regression (as opposed to multiple regression, which handles two or more independent variables). Least squares linear regression is a method for predicting the value of a dependent variable Y, based on the value of an independent variable X.

ioc.pdf

An Introductory Example

Example data:

A scatter plot of the example data. The black line consists of

the predictions, the points are the actual data, and the vertical lines between the points and the black line represent errors of prediction (called residuals).

ioc.pdf

Prerequisites for Regression

Simple linear regression is appropriate when the following conditions are satised. (^1) The dependent variable Y has a linear relationship to the independent variable X. To check this, make sure that the XY scatterplot is linear and that the residual plot shows a random pattern (= normal distributed). (^2) For each value of X , the probability distribution of Y has the same standard deviation σ. When this condition is satised, the variability of the residuals will be relatively constant across all values of X , which is easily checked in a residual plot. (^3) For any given value of X , I (^) The Y values are independent, as indicated by a random pattern on the residual plot. I (^) The Y values are roughly normally distributed (i.e., symmetric and unimodal). A little skewness is ok if the sample size is large. A histogram or a dotplot will show the shape of the distribution.

ioc.pdf

The Least Squares Regression Line

Linear regression nds the straight line, called the least squares regression line or LSRL, that best represents observations in a bivariate data set. Suppose Y is a dependent variable, and X is an independent variable. The population regression line is: Y = B 0 + B 1 X

where B 0 is a constant, B 1 is the regression coecient, X is the value of the independent variable, and Y is the value of the dependent variable. Given a random sample of observations, the population regression line is estimated by:

̂ y = b 0 + b 1 x

where b 0 is a constant, b 1 is the regression coecient, x is the value of the independent variable, and ŷ is the predicted value of the dependent variable.

ioc.pdf

Properties of the Regression Line

When the regression parameters (b 0 and b 1 ) are dened as described above, the regression line has the following properties. The line minimizes the sum of squared dierences between observed values (the y values) and predicted values (the ̂y values computed from the regression equation).

The regression line passes through the mean of the X values (c) and through the mean of the Y values (y ).

The regression constant (b 0 ) is equal to the y intercept of the regression line.

The regression coecient (b 1 ) is the average change in the dependent variable (Y ) for a 1-unit change in the independent variable (X ). It is the slope of the regression line.

The least squares regression line is the only straight line that has all of these properties.

ioc.pdf

The Coecient of Determination

The coecient of determination (denoted by R^2 ) is a key output of

regression analysis. It is interpreted as the proportion of the variance in the

dependent variable that is predictable from the independent variable.

The coecient of determination ranges from 0 to 1. An R^2 of 0 means that the dependent variable cannot be predicted from the independent variable. An R^2 of 1 means the dependent variable can be predicted without error from the independent variable. An R^2 between 0 and 1 indicates the extent to which the dependent variable is predictable. An R^2 of 0.10 means that 10 percent of the variance in Y is predictable from X ; an R^2 of 0.20 means that 20 percent is predictable; and so on.

ioc.pdf

Next section

1 What is Linear Regression?

2 Properties of a Regression Line

3 Simple Linear Regression Example

4 Regression models in R

ioc.pdf

Problem Statement

Last year, ve randomly selected students took a math aptitude test before they began their statistics course. The Statistics Department has three questions. What linear regression equation best predicts statistics performance, based on math aptitude scores? If a student made an 80 on the aptitude test, what grade would we expect her to make in statistics? How well does the regression equation t the data?

ioc.pdf

Usage of the Regression Equation

Once you have the regression equation, using it is a snap. Choose a value for the independent variable (x), perform the computation, and you have an estimated value ( ŷ ) for the dependent variable. In our example, the independent variable is the student's score on the aptitude test. The dependent variable is the student's statistics grade. If a student made an 80 on the aptitude test, the estimated statistics grade would be:

ŷ = 26. 768 + 0. 644 x = 26. 768 + 0. 644 · 80 = 26. 768 + 51. 52 = 78. 288

Warning: When you use a regression equation, do not use values for the independent variable that are outside the range of values used to create the equation. Such an extrapolation can produce unreasonable estimates.

ioc.pdf

Finding the Coecient of Determination

To assess how well the regression equation ts the data, coecient of determination can be checked. For our example:

σx =

∑ni= 1 (xi −^ x)^2 N

σx =

∑ni= 1 (yi − y )^2 N

R^2 =

∑(xi − x)(yi − y ) Nσx σy

= ( 0. 693 )^2 = 0. 480

A coecient of determination equal to 0.48 indicates that about 48% of the variation in statistics grades (the dependent variable) can be explained by the relationship to math aptitude scores (the independent variable). This would be considered a good t to the data, in the sense that it would substantially improve an educator's ability to predict student performance in statistics class.

ioc.pdf

How To Fit Linear Regression Models?

Use the lm() function to t linear models

Example

x <- c(95, 85, 80, 70, 60) y <- c(85, 95, 70, 65, 70) lmMod <- lm (y ~ x) lmMod

Call: lm(formula = y ~ x)

Coefficients: (Intercept) x 26.7808 0.

ioc.pdf

How To Fit Linear Regression Models?

Use the lm() function to t linear models

Example: complete information

lmMod <- lm (y ~ x) summary(lmMod)

Call: lm(formula = y ~ x)

Residuals: 1 2 3 4 5 -2.945 13.493 -8.288 -6.849 4.

Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 26.7808 30.5182 0.878 0. x 0.6438 0.3866 1.665 0.

Residual standard error: 10.45 on 3 degrees of freedom Multiple R-squared: 0.4803, Adjusted R-squared: 0. F-statistic: 2.773 on 1 and 3 DF, p-value: 0.