Statistical Learning and Modeling: Supervised Learning Techniques, Exams of Nursing

Download Statistical Learning and Modeling: Supervised Learning Techniques and more Exams Nursing in PDF only on Docsity!

Artificial Intelligence

Statistical Learning and Modeling: Supervised Learning

Fei Wu

College of Computer Science Zhejiang University

http://person.zju.edu.cn/wufei/

Outlines

 Linear model for classification

 Ada Boosting

Learning the parameters of Linear Discriminant Functions

• Three approaches:
  • Least-squares approach:
    • making the model predictions as close as possible to a set of target values
  • Fisher‟s linear discriminant:
    • maximum class separation in the output space
  • The perceptron algorithm of Rosenblatt:
    • generalized linear model

Linear Basis Function Models

Parameter optimization via Maximum likelihood

• Assume:
• Thus:
• For data set (^) X = { x 1 ,... , x N} and target vector t = (t 1 ,... , tN)T, the likelihoodfunction:

SSE: sum-of-squares

error function

Parameter optimization via Maximum likelihood

• Solving w by Maximum likelihood:

N ×M design matrix

Moore-Penrose pseudo-inverse Φ† = (Φ𝑇Φ)− 1 Φ𝑇

• Problem:

Parameter optimization via Least Square

- : - group together: -

• Learning with training data set:
• minimizing a sum-of-squares error function :
• Discriminant function:

Maximum likelihood and least squares for linear regressionclassification

 Maximum likelihood estimation method (MLE)   The likelihood function indicates how likely the observed sample is as a function of possible parameter values. Therefore, maximizing the likelihood function determines the parameters that are most likely to produce the observed data. From a statistical point of view, MLE is usually recommended for large samples because it is versatile, applicable to most models and different types of data, and produces the most precise estimates.

 Least squares estimation method (LSE)   Least squares estimates are calculated by fitting a regression line to the points from a data set that has the minimal sum of the deviations squared (least square error). In reliability analysis, the line and the data are plotted on a probability plot.

 In a linear model, if the errors belong to a normal distribution the least squaresestimators are also the maximum likelihood estimators. 

Fisher‟s linear discriminant

• The Fisher’s criterion: maximize the separation between the projected class means as wellas the inverse of the total within-class variance.

Generalized Rayleigh quotient

Between-class covariance matrix Within-class covariance matrix

• Fisher’s linear discriminant:

Fisher‟s linear discriminant

𝑆−^1 𝑆 𝒘^ =^ λ𝐰 𝑤 𝐵

Fisher‟s discriminant for multiple classes

• Assume input space dimensionality (^) D > K (number of classes, (^) K>2 ):

covariance matrices defined in the original x-space

• Total covariance matrix:
• The generalization of the within-class covariance matrix:
• The generalization of the between-class covariance matrix:

Fisher‟s discriminant for multiple classes

• Assume input space dimensionality (^) D > K ( K is the number of classes, (^) K>2 ):

covariance matrices defined in the projected y-space

• The generalization of the within-class and between-class covariance matrix:
• The Fisher’s criterion:

The perceptron algorithm

• Stochastic gradient descent algorithm:
• Perceptron convergence theorem:
  • If there exists an exact solution (in other words, if the training data set is linearly separable), then the perceptron learning algorithm is guaranteed to find an exactsolution in a finite number of steps.

The perceptron algorithm