LING 120: Computers and Language
Topic: Information Retrieval
Recommended Reading: Tzoukermann et al., 2003
●Information retrieval: The science of finding objects (text, picture, video, etc.) in any media
relevant to user query.
●Text retrieval: The science of finding text relevant to user query in a collection of documents
○Query may be formulated by user, or may be pre-formulated based on a user profile.
●Dates back to 1890 when Herman Hollerith invented a machine to tabulate US census data.
●Most uses for years were in scientific, legal, and medical fields. Widespread use with the advent
of the Worldwide Web.
●Main components of any IR system:
○Document processing (represent documents in a searchable form)
○Query processing (represent queries in a searchable form)
○Matching and retrieval (mechanism to measure document relevance to query)
●Document processing:
○Documents are usually represented as indexed keywords (“inverted index”).
build → 12 school → 85
The numbers show position of word in a document in characters. An inverted index is built
for each document.
○To save time/space, “stop words” (i.e., function words) are removed first.
○Also key words are sometimes “stemmed” (i.e., prefixes and suffixes are removed) before
indexing.
■There is still debate on stemming due to mixed results. Stemming typically increases
“recall”, but reduces “precision” (see notes on text categorization).
■Stemming could be done using
●a traditional stemmer (see Chapter 3 of textbook; exercise 3.3.4 number 3),
●or a full-fledged morphological analyzer.
●Query processing
○A number of models are generally used, e.g., boolean, vector space, & probabilistic
○Boolean systems: Queries are built based on the presence or absence of query terms in
documents.
■These systems are very efficient and are widely used in bibliographic and other database
searches.
■Decisions are binary (relevant, or not relevant). Therefore, may miss documents that are
somewhat relevant.
1
