Chapter 6 SNLP Vorlesung, Slides von Statistische Analysen

Nur auf Docsity: Lade Chapter 6 SNLP Vorlesung und mehr Slides als PDF für Statistische Analysen herunter!

See Chapter 16 in Manning&Schütze

Chapter 6:

Text Categorization

Text Categorization and

related Tasks

Spam/junk/bulk Emails

• The messages you spend your time with

just to delete them

• Spam: do not want to get unsolicited

messages

• Junk: irrelevant to the recipient, unwanted

• Bulk: mass mailing for business marketing

(or fill-up mailbox etc.)

Classification task: decide for each e-mail

whether it is spam/not-spam

Author identification

• They agreed that Mrs. X should only hear of the departure of the family, without being alarmed on the score of the gentleman's conduct; but even this partial communication gave her a great deal of concern, and she bewailed it as exceedingly unlucky that the ladies should happen to go away, just as they were all getting so intimate together.
• Gas looming through the fog in divers places in the streets, much as the sun may, from the spongey fields, be seen to loom by husbandman and ploughboy. Most of the shops lighted two hours before their time--as the gas seems to know, for it has a haggard and unwilling look. The raw afternoon is rawest, and the dense fog is densest, and the muddy streets are muddiest near that leaden-headed old obstruction, appropriate ornament for the threshold of a leaden-headed old corporation, Temple Bar.

Author identification

• Federalist papers

• 77 short essays written in 1787-1788 by Hamilton,

Jay and Madison to persuade NY to ratify the US

Constitution; published under a pseudonym

• The authorships of 12 papers was in dispute

( disputed papers )

• In 1964 Mosteller and Wallace*^ solved the problem

• They identified 70 function words as good

candidates for authorships analysis

• Using statistical inference they concluded the author

was Madison

Function words for Author

Identification

Text Categorization

Speech Recognition

Information Retrieval

Computer Linguistics

Everything else

bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla

Text Categorization

• Topic categorization: classify the

document into semantics topics

The U.S. swept into the Davis

Cup final on Saturday when twins

Bob and Mike Bryan defeated

Belarus's Max Mirnyi and Vladimir

Voltchkov to give the Americans

an unsurmountable 3-0 lead in the

best-of-five semi-final tie.

One of the strangest, most

relentless hurricane seasons on

record reached new bizarre heights

yesterday as the plodding approach

of Hurricane Jeanne prompted

evacuation orders for hundreds of

thousands of Floridians and high

wind warnings that stretched 350

miles from the swamp towns south

of Miami to the historic city of St.

Augustine.

Text categorization

• Reuters

• Collection of (21,578) newswire documents.

• For research purposes: a standard text

collection to compare systems and

algorithms

• 135 valid topics categories

Top topics in Reuters