Deep Dive into AI Programming for Web Crawling and Information Retrieval - Prof. Christoph, Study Guides, Projects, Research of Computer Science

Download Deep Dive into AI Programming for Web Crawling and Information Retrieval - Prof. Christoph and more Study Guides, Projects, Research Computer Science in PDF only on Docsity!

Artificial Intelligence Programming^ Comments on Project 1^ Chris BrooksDepartment of Computer ScienceUniversity of San Francisco

Department of Computer Science — University of San Francisco – p.1/

9-2: Web crawlers^ •^ A web crawler is a program that crawls the web.^ •^ Starts with a set of root pages.^ •^ Extracts links to other pages.^ •^ Fetch each of those pages.^ •^ Extract their links and repeat.

Department of Computer Science — University of San Francisco – p.2/

9-4: Focused crawlers^ •^ A focused crawler is a crawler that looks for pages that satisfy aparticular set of criteria.^ •^ In this project, we’ll build a focused crawler.^ •^ Two methods for users to indicate criteria:^ ◦^ Keywords^ ◦^ Example pages

Department of Computer Science — University of San Francisco – p.4/

9-5: The Crawler in a nutshell^ queue^ = [rootpage]while^ not^ done^ : dequeue^ best^ URL^ in

queue.if not visited previously^ : fetch contents of^ URLextract outward linksscore pageenqueue all outward^ links^ according^ to

the^ page’s^ score.^ Department of Computer Science — University of San Francisco – p.5/

9-7: How to do this?^ •^ You’ll want to build a helper class that can parse HTML.^ •^ I recommend subclassing the SGMLParser class.^ ◦^ Dive Into Python shows how to do this^ ◦^ You may have done this in homework 1.^ •^ This will let you extract anchors, headings, title and text.

Department of Computer Science — University of San Francisco – p.7/

9-8: Filtering content^ •^ You’ll then remove non-useful content^ ◦^ Any words with a non-alphabetic character^ ◦^ Any stop words.

Department of Computer Science — University of San Francisco – p.8/

9-10: Hints and Issues^ •^ You’ll need to deal gracefully with malformed HTML.^ •^ You’ll need to deal gracefully with remote server timeouts.^ •^ You may need to deal with authentication.^ •^ Remember Python’s motto: “Batteries included”

Department of Computer Science — University of San Francisco – p.10/

9-11: the FocusedCrawler class^ •^ The FocusedCrawler will manage the process of creatingDocuments according to their expected value.^ ◦^ Much like the ’search()’ function in homework 3.^ •^ Maintain a heapq of (score, URL) tuples.^ •^ Dequeue the best URL, check to see if we’ve visited it before,create a Document, score it, enqueue (score, URL) tuples forits outward Links.

Department of Computer Science — University of San Francisco – p.11/

9-13: Dealing with robots.txt^ •^ Your crawler should also repsect robots.txt^ •^ Before fetching a URL, use the robotparser module todetermine if that URL is allowed.

Department of Computer Science — University of San Francisco – p.13/

9-14: Scoring a Document^ •^ Now we know how to fetch Documents, process them, andcollect their outward links.^ •^ How to determine the ’goodness’ of a document?^ •^ Assumption: the estimated score of a document will bedetermined by the quality of documents that link to it.

Department of Computer Science — University of San Francisco – p.14/

9-16: Query-driven search^ •^ To score a document, count the number of ANDed words thatare contained within the document, along with the number ofORed clauses that have any word in the document.^ •^ To normalize this, we then divide by the length of the query,which is the number of ANDed terms plus the number of ORedclauses.^ •^ This gives us a number between 0 and 1.

Department of Computer Science — University of San Francisco – p.16/

9-17: Example^ •^ Let’s say our document contains ’cat cat dog dog bunny squirrelfish’^ •^ Our query is ’dog (cat snake) bunny lion’^ •^ Our score is 3/4 = 0.75.^ •^ Note that multiple occurrences of a query term count as onematch.

Department of Computer Science — University of San Francisco – p.17/

9-19: Searching by topic^ •^ Our user may want to say “find more documents like these!”^ •^ We’ll call a set of documents that are topically related a^ similarity set^ •^ Task: construct a model of these documents and use this tocompare to other documents.

Department of Computer Science — University of San Francisco – p.19/

9-20: Documents as vectors^ •^ To do this, we’ll use a

vector representation

of a document.

-^ Imagine an^ n-dimensional vector, where

n^ is all words of interest. • The value in the^ nth place is the number of occurrences of word n^ in that document.^ ◦^ Note: implement this using a dictionary, not a list. • This vector gives us a representation of a document that can becompared to other documents using linear algebra.

Department of Computer Science — University of San Francisco – p.20/