Download Structured Prediction: Learning Complex Structures from Examples and more Study notes Computer Science in PDF only on Docsity!
Structured Prediction
Hal Daumé III CS5350: Machine Learning
02 Dec 2008 Hal Daumé III (U Utah)
Structured Prediction (1 / 23)
What are complex structures?^ The
man
ate^
a^
tasty
sandwich
DT^
NN^
VBD
DT^
JJ^
NN
NP
VP S
NPNP-NN
CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (2 / 23)
Problem formulation^ Informally
, given:
1.^
A bunch of inputs
2.^
Correct corresponding outputs Induce a function that maps novel inputs to corresponding outputs. Formally
, given:
1.^
An input space
X
2.^
An output space
Y
(that is “hairy”)
3.^
A distribution
D
over
X × Y
4.^
A loss function
ℓ^
:^ Y × Y →
R
Induce function
f^
:^ X → Y
with low expected loss (with respect to
D
Hal Daumé III (U Utah)
Structured Prediction (3 / 23)
Example 1: Sequence labeling
Input
The
man
ate
the
really
tasty
sandwich
Output
DET
NOUN
VERB
DET
ADV
ADJ
NOUN
Inputs are sequences; outputs are (equal-length) seqences of labelsdrawn from a label set.Three approaches:^ ◮
Independent prediction:^ Ignore the structure and predict each label independently ◮ Global prediction:^ Learn a score
f^
:^ X × Y →
R
+^ so that good
ys have high score
◮^
Productive prediction: Learn a function that generates the output sequentially CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (4 / 23)
Independent Prediction
Hal Daumé III (U Utah)
Structured Prediction (5 / 23)
Method 1: Independent prediction
t^ =
t^ =
t^ =
t^ =
t^ =
t^ =
t^ =
Input
The
man
ate
the
really
tasty
sandwich
Output
DET
NOUN
VERB
DET
ADV
ADJ
NOUN
Features may depend on any of the input, but must
decompose
entirely over the output
:^ f^ ( x,^ y) =
T∑ t=^1
⊤ w
x,^
y)t^
One may use any classifier, for instance:^ ◮
Logistic regression (aka maximum entropy) classifiers ◮ Support vector machines: ◮... Pros:^ ◮
Really efficient at training and test time ◮ Can use any off the shelf multiclass classifier CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (6 / 23)
Independent prediction: Enforcing Constraints^ For instance:
Input
George
Bush
spoke
to^
Congress
today
Output
B-P
ER
I-P
ER
O^
O^
B-O
RG
O
Must ensure that I-P
ER
always follows B-P
ER
or I-P
ER
◮^
At test time, constraints can be enforced by (eg) integerprogramming ◮^
Incurs additional test-time complexity ◮^
See
[Punyakanok and Roth; IJCAI 2005]
Hal Daumé III (U Utah)
Structured Prediction (7 / 23)
Global prediction^ Goal:
Learn a score
f^
:^ X × Y →
R
+^ so that good
ys have high score
Three issues:^ ◮
How to represent
f
◮^
How to learn
f^ from a finite sample
◮^
Given
f^ and a new input
x, how to find best
y
CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (8 / 23)
Global prediction: how to learn (II)^ Instead of forcing
all
other outputs to look bad, just look at the best!
Leads to
maximum margin Markov networks^ min
w
ensures large margin
︷^
|| w
subject to
f^ (
xn
,^ y
)n
f^ (
xn
,^ y
︸^
predicted scores
(y
,^ n
′y
︸^
cost of error
n,
y
See
[Taskar, Guestrin and Koller; NIPS 2002]
Looks like exponentially many constraints, but can be reduced (ifclever) to depend on the tree-width. (Technically, must introduce slack variables.) CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (13 / 23)
Global prediction: Summary^ Your task:
◮^
Represent output structure as a graph ◮^
Define features over cliques Prediction can be solved efficiently for many reasonable problems:^ ◮
chains, trees, bipartite matchings, graph cuts (sort of) CRF normalization efficient for many problems:^ ◮
chains, trees (^3) M N contraints polynomial for more problems: ◮^ chains, trees, bipartite matchings CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (14 / 23)
Can we trade structure for features?
Input
The
man
ate
the
really
tasty
sandwich
Output
DET
NOUN
VERB
DET
ADV
ADJ
NOUN
Idea:
instead of using V
ERB
-D
ET
as a feature, just add extra
input
features
and use independent predictions.
Problem:^ ◮
This introduces a ton new features, whose weights we need toestimate ◮ Probably don’t have enough data to do this reliably ◮ CRFs are
computationally complex
but
statistically simple
; IRL is
computationally simple
but
statistically complex
Solution:^ ◮
“Structure Compilation”
[Liang, Daumé and Klein, ICML 2008]
CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (15 / 23)
Structure Compilation^ Simple idea/algorithm:
◮^
Train a CRF on your labeled data ◮^
Run this CRF on a large amount of unlabeled data ◮^
Train an IRL on the data labeled by the CRF, using more features
2
4
8 16
32
64 128 200
m^ (thousands)
(^100989694) tag accuracy^92
CRF(
f)^1 ILR(
f)^1 ILR(
f)^2
2
4
8 16
32
64 128 200
m^
(thousands)
(^1009218476) Labeled F^68
CRF(
f)^1 ILR(
f)^1 ILR(
f)^2
(a) POS
(b) NER
CS 5350/6350^ Can bound the IRL classifier’s error by:
Hal Daumé III (U Utah)
Structured Prediction (16 / 23)
Productive prediction:
Striking a middle ground
Hal Daumé III (U Utah)
Structured Prediction (17 / 23)
Productive prediction: adding history to classifiers^ Idea:
Follow independent classifier methodology, but: ◮^
Predict variables in a prescribed order ◮^
Allow features to depend on
any
past decision
Input
The
man
ate
the
really
tasty
sandwich
Output
DET
NOUN
VERB
DET
ADV
ADJ
NOUN
◮^
At training time:^ 1.
Make a classification example for D
ET
2.^
Make an example for N
OUN
, knowing D
ET
3.^
Make an example for V
ERB
, knowing D
ET N OUN
4.^
Make an example for D
ET , knowing D
ET
... V
ERB
5.^
And so on...
◮^
At test time:^ 1.
Predict the first label
y^1
2.^
Predict the second
y^2
, knowing
y^1
3.^
Predict
y^3
, knowing
y^1
,^ y^2
4.^
And so on...
What could gowrong in thispicture?
CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (18 / 23)
Structured prediction via
search
Key Idea:
view prediction task as search
◮^
A^
path
corresponds to a full output
◮^
Each
decision
is over a small set of
options ◮^
Train a
classifier
to make search
predictions How can we train this classifier?^ ◮
Train on every node in the search space^ This is effectively what CRFs do ◮ Train only on the correct path^ This incurs label-bias problem^ See
[Daumé & Marcu; ICML 2005]
and
[Xu and Fern, ICML 2007]
◮^
Train on the subset of states that we are likely to reach! How do we know what states we are likely to reach?Chicken-and-egg problem!
Hal Daumé III (U Utah)
Structured Prediction (19 / 23)
S
EARN
: Search-based structured prediction
Idea:
Train on the subset of states that we are likely to reach ◮^
Iterative algorithm ◮^
First train on the correct path
( h 1 )
◮^
Then on an interpolation between thebest path and
( h 1 )^
( h 2 )
◮^
Then on an interpolation between thebest path and
( h 1 )^ and
( h 2 )^
( h 3 )
◮^
And so on... Guaranteed to converge in a polynomial number of iterations to amodel with regret:
R
(D
,^ h
last
)^ ≤
T^
ln
T
ℓavg
ln
T
T^
cmax
See
[Daumé III, Langford and Marcu; MLJ 2007]
CS 5350/
Hal Daumé III (U Utah)
Structured Prediction (20 / 23)
