libro..................., Schemi e mappe concettuali di Statistica

Scarica libro................... e più Schemi e mappe concettuali in PDF di Statistica solo su Docsity!

Gareth James •^ Daniela Witten •

Trevor Hastie •^ Robert Tibshirani

An Introduction to Statistical

Learning

with Applications in R

Second Edition

This is page ii Printer: Opaque

To our parents:

Alison and Michael James

Chiara Nappi and Edward Witten

Valerie and Patrick Hastie

Vera and Sami Tibshirani

and to our families:

Michael, Daniel, and Catherine

Tessa, Theo, Otto, and Ari

Samantha, Timothy, and Lynda

Charlie, Ryan, Julie, and Cheryl

viii The first edition of ISL covered a number of important topics, including sparse methods for classification and regression, decision trees, boosting, support vector machines, and clustering. Since it was published in 2013, it has become a mainstay of undergraduate and graduate classrooms across the United States and worldwide, as well as a key reference book for data scientists. In this second edition of ISL, we have greatly expanded the set of topics covered. In particular, the second edition includes new chapters on deep learning (Chapter 10 ), survival analysis (Chapter 11 ), and multiple testing (Chapter 13 ). We have also substantially expanded some chapters that were part of the first edition: among other updates, we now include treatments of naive Bayes and generalized linear models in Chapter 4 , Bayesian addi- tive regression trees in Chapter 8 , and matrix completion in Chapter 12. Furthermore, we have updated the R code throughout the labs to ensure that the results that they produce agree with recent R releases. We are grateful to these readers for providing valuable comments on the first edition of this book: Pallavi Basu, Alexandra Chouldechova, Patrick Danaher, Will Fithian, Luella Fu, Sam Gross, Max Grazier G’Sell, Court- ney Paulson, Xinghao Qiao, Elisa Sheng, Noah Simon, Kean Ming Tan, Xin Lu Tan. We thank these readers for helpful input on the second edi- tion of this book: Alan Agresti, Iain Carmichael, Yiqun Chen, Erin Craig, Daisy Ding, Lucy Gao, Ismael Lemhadri, Bryan Martin, Anna Neufeld, Ge- off Tims, Carsten Voelkmann, Steve Yadlowsky, and James Zou. We also thank Anna Neufeld for her assistance in reformatting the R code through- out this book. We are immensely grateful to Balasubramanian “Naras” Narasimhan for his assistance on both editions of this textbook. It has been an honor and a privilege for us to see the considerable impact that the first edition of ISL has had on the way in which statistical learning is practiced, both in and out of the academic setting. We hope that this new edition will continue to give today’s and tomorrow’s applied statisticians and data scientists the tools they need for success in a data-driven world. It’s tough to make predictions, especially about the future. -Yogi Berra

This is page ix

Printer: Opaque Contents

• 1 Introduction Preface vii
• 2 Statistical Learning
  • 2.1 What Is Statistical Learning?
    • 2.1.1 Why Estimate f?
    • 2.1.2 How Do We Estimate f?
      • and Model Interpretability 2.1.3 The Trade-Off Between Prediction Accuracy
    • 2.1.4 Supervised Versus Unsupervised Learning
    • 2.1.5 Regression Versus Classification Problems
  • 2.2 Assessing Model Accuracy
    • 2.2.1 Measuring the Quality of Fit
    • 2.2.2 The Bias-Variance Trade-Off
    • 2.2.3 The Classification Setting
  • 2.3 Lab: Introduction to R
    • 2.3.1 Basic Commands
    • 2.3.2 Graphics
    • 2.3.3 Indexing Data
    • 2.3.4 Loading Data
    • 2.3.5 Additional Graphical and Numerical Summaries
  • 2.4 Exercises
• 3 Linear Regression
  • 3.1 Simple Linear Regression
    • 3.1.1 Estimating the Coefficients
      • Estimates 3.1.2 Assessing the Accuracy of the Coefficient
    • 3.1.3 Assessing the Accuracy of the Model
  • 3.2 Multiple Linear Regression
    • 3.2.1 Estimating the Regression Coefficients
    • 3.2.2 Some Important Questions x Contents
  • 3.3 Other Considerations in the Regression Model
    • 3.3.1 Qualitative Predictors
    • 3.3.2 Extensions of the Linear Model
    • 3.3.3 Potential Problems
  • 3.4 The Marketing Plan
    • Neighbors 3.5 Comparison of Linear Regression with K-Nearest
  • 3.6 Lab: Linear Regression
    • 3.6.1 Libraries
    • 3.6.2 Simple Linear Regression
    • 3.6.3 Multiple Linear Regression
    • 3.6.4 Interaction Terms
    • 3.6.5 Non-linear Transformations of the Predictors
    • 3.6.6 Qualitative Predictors
    • 3.6.7 Writing Functions
  • 3.7 Exercises
• 4 Classification
  • 4.1 An Overview of Classification
  • 4.2 Why Not Linear Regression?
  • 4.3 Logistic Regression
    • 4.3.1 The Logistic Model
    • 4.3.2 Estimating the Regression Coefficients
    • 4.3.3 Making Predictions
    • 4.3.4 Multiple Logistic Regression
    • 4.3.5 Multinomial Logistic Regression
  • 4.4 Generative Models for Classification
    • 4.4.1 Linear Discriminant Analysis for p =
    • 4.4.2 Linear Discriminant Analysis for p >
    • 4.4.3 Quadratic Discriminant Analysis
    • 4.4.4 Naive Bayes
  • 4.5 A Comparison of Classification Methods
    • 4.5.1 An Analytical Comparison
    • 4.5.2 An Empirical Comparison
  • 4.6 Generalized Linear Models
    • 4.6.1 Linear Regression on the Bikeshare Data
    • 4.6.2 Poisson Regression on the Bikeshare Data
    • 4.6.3 Generalized Linear Models in Greater Generality
  • 4.7 Lab: Classification Methods
    • 4.7.1 The Stock Market Data
    • 4.7.2 Logistic Regression
    • 4.7.3 Linear Discriminant Analysis
    • 4.7.4 Quadratic Discriminant Analysis
    • 4.7.5 Naive Bayes
    • 4.7.6 K-Nearest Neighbors Contents xi
    • 4.7.7 Poisson Regression
  • 4.8 Exercises
• 5 Resampling Methods
  • 5.1 Cross-Validation
    • 5.1.1 The Validation Set Approach
    • 5.1.2 Leave-One-Out Cross-Validation
    • 5.1.3 k-Fold Cross-Validation
      • Cross-Validation 5.1.4 Bias-Variance Trade-Off for k-Fold
    • 5.1.5 Cross-Validation on Classification Problems
  • 5.2 The Bootstrap
  • 5.3 Lab: Cross-Validation and the Bootstrap
    • 5.3.1 The Validation Set Approach
    • 5.3.2 Leave-One-Out Cross-Validation
    • 5.3.3 k-Fold Cross-Validation
    • 5.3.4 The Bootstrap
  • 5.4 Exercises
• 6 Linear Model Selection and Regularization
  • 6.1 Subset Selection
    • 6.1.1 Best Subset Selection
    • 6.1.2 Stepwise Selection
    • 6.1.3 Choosing the Optimal Model
  • 6.2 Shrinkage Methods
    • 6.2.1 Ridge Regression
    • 6.2.2 The Lasso
    • 6.2.3 Selecting the Tuning Parameter
  • 6.3 Dimension Reduction Methods
    • 6.3.1 Principal Components Regression
    • 6.3.2 Partial Least Squares
  • 6.4 Considerations in High Dimensions
    • 6.4.1 High-Dimensional Data
    • 6.4.2 What Goes Wrong in High Dimensions?
    • 6.4.3 Regression in High Dimensions
    • 6.4.4 Interpreting Results in High Dimensions
  • 6.5 Lab: Linear Models and Regularization Methods
    • 6.5.1 Subset Selection Methods
    • 6.5.2 Ridge Regression and the Lasso
    • 6.5.3 PCR and PLS Regression
  • 6.6 Exercises
• 7 Moving Beyond Linearity
  • 7.1 Polynomial Regression
  • 7.2 Step Functions xii Contents
  • 7.3 Basis Functions
  • 7.4 Regression Splines
    • 7.4.1 Piecewise Polynomials
    • 7.4.2 Constraints and Splines
    • 7.4.3 The Spline Basis Representation
      • of the Knots 7.4.4 Choosing the Number and Locations
    • 7.4.5 Comparison to Polynomial Regression
  • 7.5 Smoothing Splines
    • 7.5.1 An Overview of Smoothing Splines
    • 7.5.2 Choosing the Smoothing Parameter λ
  • 7.6 Local Regression
  • 7.7 Generalized Additive Models
    • 7.7.1 GAMs for Regression Problems
    • 7.7.2 GAMs for Classification Problems
  • 7.8 Lab: Non-linear Modeling
    • 7.8.1 Polynomial Regression and Step Functions
    • 7.8.2 Splines
    • 7.8.3 GAMs
  • 7.9 Exercises
• 8 Tree-Based Methods
  • 8.1 The Basics of Decision Trees
    • 8.1.1 Regression Trees
    • 8.1.2 Classification Trees
    • 8.1.3 Trees Versus Linear Models
    • 8.1.4 Advantages and Disadvantages of Trees
    • Regression Trees 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive
    • 8.2.1 Bagging
    • 8.2.2 Random Forests
    • 8.2.3 Boosting
    • 8.2.4 Bayesian Additive Regression Trees
    • 8.2.5 Summary of Tree Ensemble Methods
  • 8.3 Lab: Decision Trees
    • 8.3.1 Fitting Classification Trees
    • 8.3.2 Fitting Regression Trees
    • 8.3.3 Bagging and Random Forests
    • 8.3.4 Boosting
    • 8.3.5 Bayesian Additive Regression Trees
  • 8.4 Exercises
• 9 Support Vector Machines
  • 9.1 Maximal Margin Classifier
    • 9.1.1 What Is a Hyperplane? Contents xiii
    • 9.1.2 Classification Using a Separating Hyperplane
    • 9.1.3 The Maximal Margin Classifier
    • 9.1.4 Construction of the Maximal Margin Classifier
    • 9.1.5 The Non-separable Case
  • 9.2 Support Vector Classifiers
    • 9.2.1 Overview of the Support Vector Classifier
    • 9.2.2 Details of the Support Vector Classifier
  • 9.3 Support Vector Machines - Boundaries 9.3.1 Classification with Non-Linear Decision
    • 9.3.2 The Support Vector Machine
    • 9.3.3 An Application to the Heart Disease Data
  • 9.4 SVMs with More than Two Classes
    • 9.4.1 One-Versus-One Classification
    • 9.4.2 One-Versus-All Classification
  • 9.5 Relationship to Logistic Regression
  • 9.6 Lab: Support Vector Machines
    • 9.6.1 Support Vector Classifier
    • 9.6.2 Support Vector Machine
    • 9.6.3 ROC Curves
    • 9.6.4 SVM with Multiple Classes
    • 9.6.5 Application to Gene Expression Data
  • 9.7 Exercises
• 10 Deep Learning
  • 10.1 Single Layer Neural Networks
  • 10.2 Multilayer Neural Networks
  • 10.3 Convolutional Neural Networks
    • 10.3.1 Convolution Layers
    • 10.3.2 Pooling Layers
    • 10.3.3 Architecture of a Convolutional Neural Network
    • 10.3.4 Data Augmentation
    • 10.3.5 Results Using a Pretrained Classifier
  • 10.4 Document Classification
  • 10.5 Recurrent Neural Networks
    • 10.5.1 Sequential Models for Document Classification
    • 10.5.2 Time Series Forecasting
    • 10.5.3 Summary of RNNs
  • 10.6 When to Use Deep Learning
  • 10.7 Fitting a Neural Network
    • 10.7.1 Backpropagation
    • 10.7.2 Regularization and Stochastic Gradient Descent
    • 10.7.3 Dropout Learning
    • 10.7.4 Network Tuning
  • 10.8 Interpolation and Double Descent xiv Contents
  • 10.9 Lab: Deep Learning
    • 10.9.1 A Single Layer Network on the Hitters Data
    • 10.9.2 A Multilayer Network on the MNIST Digit Data
    • 10.9.3 Convolutional Neural Networks
    • 10.9.4 Using Pretrained CNN Models
    • 10.9.5 IMDb Document Classification
    • 10.9.6 Recurrent Neural Networks
  • 10.10 Exercises
• 11 Survival Analysis and Censored Data
  • 11.1 Survival and Censoring Times
  • 11.2 A Closer Look at Censoring
  • 11.3 The Kaplan–Meier Survival Curve
  • 11.4 The Log-Rank Test
  • 11.5 Regression Models With a Survival Response
    • 11.5.1 The Hazard Function
    • 11.5.2 Proportional Hazards
    • 11.5.3 Example: Brain Cancer Data
    • 11.5.4 Example: Publication Data
  • 11.6 Shrinkage for the Cox Model
  • 11.7 Additional Topics
    • 11.7.1 Area Under the Curve for Survival Analysis
    • 11.7.2 Choice of Time Scale
    • 11.7.3 Time-Dependent Covariates
    • 11.7.4 Checking the Proportional Hazards Assumption
    • 11.7.5 Survival Trees
  • 11.8 Lab: Survival Analysis
    • 11.8.1 Brain Cancer Data
    • 11.8.2 Publication Data
    • 11.8.3 Call Center Data
  • 11.9 Exercises
• 12 Unsupervised Learning
  • 12.1 The Challenge of Unsupervised Learning
  • 12.2 Principal Components Analysis
    • 12.2.1 What Are Principal Components?
    • 12.2.2 Another Interpretation of Principal Components
    • 12.2.3 The Proportion of Variance Explained
    • 12.2.4 More on PCA
    • 12.2.5 Other Uses for Principal Components
  • 12.3 Missing Values and Matrix Completion
  • 12.4 Clustering Methods
    • 12.4.1 K-Means Clustering
    • 12.4.2 Hierarchical Clustering
    • 12.4.3 Practical Issues in Clustering Contents xv
  • 12.5 Lab: Unsupervised Learning
    • 12.5.1 Principal Components Analysis
    • 12.5.2 Matrix Completion
    • 12.5.3 Clustering
    • 12.5.4 NCI60 Data Example
  • 12.6 Exercises
• 13 Multiple Testing
  • 13.1 A Quick Review of Hypothesis Testing
    • 13.1.1 Testing a Hypothesis
    • 13.1.2 Type I and Type II Errors
  • 13.2 The Challenge of Multiple Testing
  • 13.3 The Family-Wise Error Rate
    • 13.3.1 What is the Family-Wise Error Rate?
    • 13.3.2 Approaches to Control the Family-Wise Error Rate
    • 13.3.3 Trade-Off Between the FWER and Power
  • 13.4 The False Discovery Rate
    • 13.4.1 Intuition for the False Discovery Rate
    • 13.4.2 The Benjamini–Hochberg Procedure
    • Rates 13.5 A Re-Sampling Approach to p-Values and False Discovery
    • 13.5.1 A Re-Sampling Approach to the p-Value
    • 13.5.2 A Re-Sampling Approach to the False Discovery Rate
    • 13.5.3 When Are Re-Sampling Approaches Useful?
  • 13.6 Lab: Multiple Testing
    • 13.6.1 Review of Hypothesis Tests
    • 13.6.2 The Family-Wise Error Rate
    • 13.6.3 The False Discovery Rate
    • 13.6.4 A Re-Sampling Approach
  • 13.7 Exercises
  • Index

2 1. Introduction 20 40 60 80 50 100 200 300 Age Wage 2003 2006 2009 50 100 200 300 Year Wage 1 2 3 4 5 50 100 200 300 Education Level Wage FIGURE 1.1. Wage data, which contains income survey information for men from the central Atlantic region of the United States. Left: wage as a function of age_. On average,_ wage increases with age until about 60 years of age, at which point it begins to decline. Center: wage as a function of year_. There is a slow but steady increase of approximately_ $10, 000 in the average wage between 2003 and 2009_._ Right: Boxplots displaying wage as a function of education , with 1 indicating the lowest level (no high school diploma) and 5 the highest level (an advanced graduate degree). On average, wage increases with the level of education. Given an employee’s age, we can use this curve to predict his wage. However, it is also clear from Figure 1.1 that there is a significant amount of vari- ability associated with this average value, and so age alone is unlikely to provide an accurate prediction of a particular man’s wage. We also have information regarding each employee’s education level and the year in which the wage was earned. The center and right-hand panels of Figure 1.1, which display wage as a function of both year and education, indicate that both of these factors are associated with wage. Wages increase by approximately $10, 000 , in a roughly linear (or straight-line) fashion, between 2003 and 2009 , though this rise is very slight relative to the vari- ability in the data. Wages are also typically greater for individuals with higher education levels: men with the lowest education level (1) tend to have substantially lower wages than those with the highest education level (5). Clearly, the most accurate prediction of a given man’s wage will be obtained by combining his age, his education, and the year. In Chapter 3 , we discuss linear regression, which can be used to predict wage from this data set. Ideally, we should predict wage in a way that accounts for the non-linear relationship between wage and age. In Chapter 7 , we discuss a class of approaches for addressing this problem.

1. Introduction 3 Down Up −^4 −^2 0 2 4 6 Yesterday Today’s Direction Percentage change in S&P Down Up −^4 −^2 0 2 4 6 Two Days Previous Today’s Direction Percentage change in S&P Down Up −^4 −^2 0 2 4 6 Three Days Previous Today’s Direction Percentage change in S&P FIGURE 1.2. Left: Boxplots of the previous day’s percentage change in the S&P index for the days for which the market increased or decreased, obtained from the Smarket data. Center and Right: Same as left panel, but the percentage changes for 2 and 3 days previous are shown.

Stock Market Data

The Wage data involves predicting a continuous or quantitative output value. This is often referred to as a regression problem. However, in certain cases we may instead wish to predict a non-numerical value—that is, a categorical or qualitative output. For example, in Chapter 4 we examine a stock market data set that contains the daily movements in the Standard & Poor’s 500 (S&P) stock index over a 5-year period between 2001 and 2005. We refer to this as the Smarket data. The goal is to predict whether the index will increase or decrease on a given day, using the past 5 days’ percentage changes in the index. Here the statistical learning problem does not involve predicting a numerical value. Instead it involves predicting whether a given day’s stock market performance will fall into the Up bucket or the Down bucket. This is known as a classification problem. A model that could accurately predict the direction in which the market will move would be very useful! The left-hand panel of Figure 1.2 displays two boxplots of the previous day’s percentage changes in the stock index: one for the 648 days for which the market increased on the subsequent day, and one for the 602 days for which the market decreased. The two plots look almost identical, suggest- ing that there is no simple strategy for using yesterday’s movement in the S&P to predict today’s returns. The remaining panels, which display box- plots for the percentage changes 2 and 3 days previous to today, similarly indicate little association between past and present returns. Of course, this lack of pattern is to be expected: in the presence of strong correlations be- tween successive days’ returns, one could adopt a simple trading strategy

1. Introduction 5 − 40 − 20 0 20 40 60 −^60 −^40 −^20 0 20 − 40 − 20 0 20 40 60 −^60 −^40 −^20 0 20 Z 1 Z 1 Z^2 Z^2 FIGURE 1.4. Left: Representation of the NCI60 gene expression data set in a two-dimensional space, Z 1 and Z 2_. Each point corresponds to one of the_ 64 cell lines. There appear to be four groups of cell lines, which we have represented using different colors. Right: Same as left panel except that we have represented each of the 14 different types of cancer using a different colored symbol. Cell lines corresponding to the same cancer type tend to be nearby in the two-dimensional space. The left-hand panel of Figure 1.4 addresses this problem by represent- ing each of the 64 cell lines using just two numbers, Z 1 and Z 2. These are the first two principal components of the data, which summarize the 6 , 830 expression measurements for each cell line down to two numbers or dimensions. While it is likely that this dimension reduction has resulted in some loss of information, it is now possible to visually examine the data for evidence of clustering. Deciding on the number of clusters is often a difficult problem. But the left-hand panel of Figure 1.4 suggests at least four groups of cell lines, which we have represented using separate colors. In this particular data set, it turns out that the cell lines correspond to 14 different types of cancer. (However, this information was not used to create the left-hand panel of Figure 1.4.) The right-hand panel of Fig- ure 1.4 is identical to the left-hand panel, except that the 14 cancer types are shown using distinct colored symbols. There is clear evidence that cell lines with the same cancer type tend to be located near each other in this two-dimensional representation. In addition, even though the cancer infor- mation was not used to produce the left-hand panel, the clustering obtained does bear some resemblance to some of the actual cancer types observed in the right-hand panel. This provides some independent verification of the accuracy of our clustering analysis.

6 1. Introduction

A Brief History of Statistical Learning

Though the term statistical learning is fairly new, many of the concepts that underlie the field were developed long ago. At the beginning of the nine- teenth century, the method of least squares was developed, implementing the earliest form of what is now known as linear regression. The approach was first successfully applied to problems in astronomy. Linear regression is used for predicting quantitative values, such as an individual’s salary. In order to predict qualitative values, such as whether a patient survives or dies, or whether the stock market increases or decreases, linear discrim- inant analysis was proposed in 1936. In the 1940s, various authors put forth an alternative approach, logistic regression. In the early 1970s, the term generalized linear model was developed to describe an entire class of statistical learning methods that include both linear and logistic regression as special cases. By the end of the 1970s, many more techniques for learning from data were available. However, they were almost exclusively linear methods be- cause fitting non-linear relationships was computationally difficult at the time. By the 1980s, computing technology had finally improved sufficiently that non-linear methods were no longer computationally prohibitive. In the mid 1980s, classification and regression trees were developed, followed shortly by generalized additive models. Neural networks gained popularity in the 1980s, and support vector machines arose in the 1990s. Since that time, statistical learning has emerged as a new subfield in statistics, focused on supervised and unsupervised modeling and prediction. In recent years, progress in statistical learning has been marked by the increasing availability of powerful and relatively user-friendly software, such as the popular and freely available R system. This has the potential to continue the transformation of the field from a set of techniques used and developed by statisticians and computer scientists to an essential toolkit for a much broader community.

This Book

The Elements of Statistical Learning (ESL) by Hastie, Tibshirani, and Friedman was first published in 2001. Since that time, it has become an important reference on the fundamentals of statistical machine learning. Its success derives from its comprehensive and detailed treatment of many important topics in statistical learning, as well as the fact that (relative to many upper-level statistics textbooks) it is accessible to a wide audience. However, the greatest factor behind the success of ESL has been its topical nature. At the time of its publication, interest in the field of statistical

8 1. Introduction in order to contribute to their chosen fields through the use of statistical learning tools. ISL is based on the following four premises.

1. Many statistical learning methods are relevant and useful in a wide range of academic and non-academic disciplines, beyond just the sta- tistical sciences. We believe that many contemporary statistical learn- ing procedures should, and will, become as widely available and used as is currently the case for classical methods such as linear regres- sion. As a result, rather than attempting to consider every possible approach (an impossible task), we have concentrated on presenting the methods that we believe are most widely applicable.
2. Statistical learning should not be viewed as a series of black boxes. No single approach will perform well in all possible applications. With- out understanding all of the cogs inside the box, or the interaction between those cogs, it is impossible to select the best box. Hence, we have attempted to carefully describe the model, intuition, assump- tions, and trade-offs behind each of the methods that we consider.
3. While it is important to know what job is performed by each cog, it is not necessary to have the skills to construct the machine inside the box! Thus, we have minimized discussion of technical details related to fitting procedures and theoretical properties. We assume that the reader is comfortable with basic mathematical concepts, but we do not assume a graduate degree in the mathematical sciences. For in- stance, we have almost completely avoided the use of matrix algebra, and it is possible to understand the entire book without a detailed knowledge of matrices and vectors.
4. We presume that the reader is interested in applying statistical learn- ing methods to real-world problems. In order to facilitate this, as well as to motivate the techniques discussed, we have devoted a section within each chapter to computer labs. In each lab, we walk the reader through a realistic application of the methods considered in that chap- ter. When we have taught this material in our courses, we have al- located roughly one-third of classroom time to working through the labs, and we have found them to be extremely useful. Many of the less computationally-oriented students who were initially intimidated by the labs got the hang of things over the course of the quarter or semester. We have used R because it is freely available and is powerful enough to implement all of the methods discussed in the book. It also has optional packages that can be downloaded to implement literally thousands of additional methods. Most importantly, R is the language of choice for academic statisticians, and new approaches often become available in R years before they are implemented in commercial pack- ages. However, the labs in ISL are self-contained, and can be skipped

1. Introduction 9 if the reader wishes to use a different software package or does not wish to apply the methods discussed to real-world problems.

Who Should Read This Book?

This book is intended for anyone who is interested in using modern statis- tical methods for modeling and prediction from data. This group includes scientists, engineers, data analysts, data scientists, and quants, but also less technical individuals with degrees in non-quantitative fields such as the social sciences or business. We expect that the reader will have had at least one elementary course in statistics. Background in linear regression is also useful, though not required, since we review the key concepts behind linear regression in Chapter 3. The mathematical level of this book is mod- est, and a detailed knowledge of matrix operations is not required. This book provides an introduction to the statistical programming language R. Previous exposure to a programming language, such as MATLAB or Python, is useful but not required. The first edition of this textbook has been used to teach master’s and PhD students in business, economics, computer science, biology, earth sci- ences, psychology, and many other areas of the physical and social sciences. It has also been used to teach advanced undergraduates who have already taken a course on linear regression. In the context of a more mathemat- ically rigorous course in which ESL serves as the primary textbook, ISL could be used as a supplementary text for teaching computational aspects of the various approaches.

Notation and Simple Matrix Algebra

Choosing notation for a textbook is always a difficult task. For the most part we adopt the same notational conventions as ESL. We will use n to represent the number of distinct data points, or observa- tions, in our sample. We will let p denote the number of variables that are available for use in making predictions. For example, the Wage data set con- sists of 11 variables for 3 , 000 people, so we have n = 3, 000 observations and p = 11 variables (such as year, age, race, and more). Note that throughout this book, we indicate variable names using colored font: Variable Name. In some examples, p might be quite large, such as on the order of thou- sands or even millions; this situation arises quite often, for example, in the analysis of modern biological data or web-based advertising data. In general, we will let xij represent the value of the jth variable for the ith observation, where i = 1, 2 ,... , n and j = 1, 2 ,... , p. Throughout this book, i will be used to index the samples or observations (from 1 to n) and