Docsity
Docsity
Prepare for your exams
Prepare for your exams

Study with the several resources on Docsity

Earn points to download
Earn points to download

Earn points by helping other students or get them with a premium plan

Guidelines and tips
Guidelines and tips
Sell on Docsity
Sell on Docsity
Docsity AI
Docsity AI
Log inSign up

Prepare for your exams

Study with the several resources on Docsity

Find documents

Prepare for your exams with the study notes shared by other students like you on Docsity

Search for your university

Find the specific documents for your university's exams

Docsity AINEW

Summarize your documents, ask them questions, convert them into quizzes and concept maps

Explore questions

Clear up your doubts by reading the answers to questions asked by your fellow students

Earn points to download

Earn points by helping other students or get them with a premium plan

Share documents
download point icon20 Points

For each uploaded document

Answer questions
download point icon5 Points

For each given answer (max 1 per day)

All the ways to get free points
Get points immediately

Choose a premium plan with all the points you need

Study Opportunities

Choose your next study program

Get in touch with the best universities in the world. Search through thousands of universities and official partners

Community

Ask the community

Ask the community for help and clear up your study doubts

Free resources

Our save-the-student-ebooks!

Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors

Learning Parameters with Logistic Regression and SVM: Maximum Likelihood Estimation, Study notes of Computer Science

Oregon State University (OSU)Computer Science

Maximum likelihood estimation (mle) and its applications to logistic regression and support vector machines. Mle is a method to estimate parameters given data, assuming the data is identically independently distributed. The likelihood function, mle estimator, and optimization techniques for logistic regression and support vector machines. It also compares these methods and discusses their differences.

Typology: Study notes

Pre 2010

Uploaded on 08/30/2009

koofers-user-daq
koofers-user-daq 🇺🇸

10 documents

1 / 9

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
1
Lecture 9
Oct - 12 - 2007
Maximum Likelihood Estimation
Goal: estimate the parameters given data
Assuming the data is i.i.d (identically independently
distributed)
For example, given the results of n coin tosses, we like
to estimate the probability of head p.
Likelihood function:
MLE estimator:
∑∏ ==
===
n
i
ii
n
i
ii yxPyxPDPL
11
)|,(log)|,(log)|(log)(
θθθθ
)(maxarg
θ
θ
θ
L
MLE
=
pf3
pf4
pf5
pf8
pf9

Partial preview of the text

Download Learning Parameters with Logistic Regression and SVM: Maximum Likelihood Estimation and more Study notes Computer Science in PDF only on Docsity!

Lecture 9

Oct - 12 - 2007

Maximum Likelihood Estimation

Goal : estimate the parameters given data Assuming the data is i.i.d (identically independently distributed) For example, given the results of n coin tosses, we like to estimate the probability of head p. Likelihood function:

MLE estimator:

= =

= = =

n i

i i n i

L PD Pxi^ yi Px y 1 1

(θ) log ( |θ) log ( , |θ) log ( , | θ)

θ argmax ( θ ) θ MLE =^ L

Example

  • Data
    • We observe n iid coin toss: D={0, 0, 1, 0,…1}
  • Binary random variable x (^) i={0,1}
  • Model:
  • Likelihood function?
  • MLE estimate?

P ( x )= θ x ( 1 − θ)^1 −^ x

MLE for logistic regression

( 0 11 ... )

e w wx wm^ xm

P y x − + + +

Batch Learning for Logistic Regression

d

d d error

error y y

e

y

i N

d

, y , i N

i

i i

·

i

i i

i

−

w w

x

For to do

Repeatuntilconvergence

Letw (0,0,0,...,0)

Given:trainingexamples x

w x )

y ˆ^ i ← sign ( w · x i )

Note: y takes 0/1 here, not 1/-

Logistic Regression Vs. Perceptron

  • Note the striking similarity between the two

algorithms

  • In fact LR learns a linear decision

boundary – how so?

  • What are the difference?
    • Different ways to train the weights
    • LR by statistician and Perceptron by CS
    • LR produces a probability estimation!

There are more!

  • If we assume Gaussian distribution for P(x|y) in

Naïve Bayes, P(y|x) will take the same functional form of Logistic Regression

  • What are the differences here?
    • Different ways of training
      • Naïve bayes estimates θi by maximizing P(X|y=vi , θi ), and while doing so assumes conditional independence among attributes
      • Logistic regression estimates w by maximizing P(y|x, w ) and make no conditional independence assumption.

Comparatively

  • Naïve Bayes - generative model: p(X, y), P(X|y)
    • makes strong conditional independence assumption about the data
    • When the assumptions are ok, naïve bayes can use small amount of training data and estimate a reasonable model
  • Logistic regression-discriminative model: p(y|X)
    • has fewer parameters to estimate, but they are tied together and make learning harder
    • Makes no strong assumptions
    • May need large number of training examples

Bottom line: if the naïve assumption holds, NB would be a good choice; otherwise, logistic regression works better

Intuition of Margin

  • Consider points A, B, and C
  • We are quite confident in our prediction for A because it is far from the decision boundary.
  • In contrast, we are not so confident in our prediction for C because a slight change in the decision boundary may flip the decision.

+ + +

+

+

+ +

+

− −

−

− − −

−

−

− −

−

A

+

B

C

Given a training set, we would like to make all of our predictions correct and confident! This leads to the concept of margin.

w · x + b = 0

Functional Margin

  • Given a linear classifier parameterized by ( w , b ), we define its functional margin w.r.t training example ( x i , yi^ ) is defined as:
  • If we rescale ( w , b ) by a factor α, functional margin gets multiplied by α - we can make it arbitrarily large without change anything meaningful

Basic facts about lines

w · x + b = 0

X^1?

1

w

w ⋅ x + b

Geometric Margin

  • The geometric margin of ( w , b ) w.r.t. x (i)^ is the distance from x (i)^ to the decision surface
  • This distance can be computed as

+ + +

+

+

+ +

+

− −

−

− − −

−

−

− −

−

A

+

B

C

γA

w

i y^ i ( w^^ ⋅ x i + b )

( ) 1

min i

i N

=L

  • Given training set S ={( x i, y i): i=1,…, N }, the geometric margin of the classifier w.r.t. S is

Points closest to the boundary are called Support vectors – only these points really matters, other examples are ignorable