String Lexers - Compiler Design - Lecture Slides, Slides of Computer Science

Download String Lexers - Compiler Design - Lecture Slides and more Slides Computer Science in PDF only on Docsity!

Lecture #6, Jan. 31, 2007

• More about ML-LEX
  • String lexers
  • File lexers
  • Handling exceptions
  • Using named REs
• Library functions
• Anonymous functions
• The compile manager

Example lex file

type lexresult = unit; type pos = int; type svalue = int; exception EOF; fun eof () = (print "eof"; raise EOF); %% %% [\t\ ]+ => ( lex() ( ignore whitespace ) ) ; Anne|Bob|Spot => ( print (yytext^": is a proper noun\n")); a|the => ( print(yytext^": is an article\n") ); boy|girl|dog => ( print(yytext^": is a noun\n") ); walked|chased|ran|bit => ( print(yytext^": is a verb\n") ); [a-zA-Z]+ => ( print(yytext^": Might be a noun?\n") ); .|\n => ( print yytext ( Echo the string ) ); lexresult must be defined in every lex program The function eof must be defined in every lex program

Building the lexer

• Consider the function: Mlex.makeLexer Mlex.makeLexer : (int -> string) -> unit -> lexresult
• It takes a function as an argument. This function

feeds the lexical analyzer the input “n” characters

at a time.

val testString = ref "the boy chased the dog"; fun feed n = let val ss = !testString in if String.size ss < n then ( testString := ""; ss ) else ( testString := String.extract(ss,n,NONE); String.extract(ss,0,SOME n) ) end; val lex = Mlex.makeLexer feed;

Running the lexer

• val lex = Mlex.makeLexer feed; val lex = fn : unit -> Mlex.Internal.result
• lex(); the: is an article val it = () : Mlex.Internal.result
• lex(); boy: is a noun val it = () : Mlex.Internal.result
• lex(); chased: is a verb val it = () : Mlex.Internal.result
• lex(); the: is an article val it = () : Mlex.Internal.result
• lex(); dog: is a noun val it = () : Mlex.Internal.result
• lex(); unexpected end of file uncaught exception EOF raised at: english.lex.sml:9.53-9.

Syntax of “case” vs “handle”

• “case” is before the computation being analyzed

case (revonto (g x) 4) of

[] => true

| (x:xs) => false

• “handle” is after the computation that might fail

(compute (g y) (length zs))

handle Error s => g s

| BadLevel n => n+

| other => raise other

Catching Exceptions in the Lexer

val lex = let val f = Mlex.makeLexer feed fun lex () = (f ()) handle Mlex.UserDeclarations.EOF => print "\nReached end of file\n" | other => print "Lex Error" in lex end; Things defined in the first section of a lex file appear in the inner library

Testing a lexer interactively

• val f = makeStringLexer "the boy sang"; val f = fn : unit -> Mlex.Internal.result
• f (); the: is an article val it = () : Mlex.Internal.result
• f (); boy: is a noun val it = () : Mlex.Internal.result
• f (); sang: Might be a noun? val it = () : Mlex.Internal.result
• f (); unexpected end of file uncaught exception EOF raised at: english.lex.sml:9.53-9.

Exercise

• Can we make a string lexer that catches

exceptions?

• Try it in class …

Running lex

val lex = Mlex.makeLexer (inputc "test.english");

• lex(); the: is an article val it = () : unit
• lex(); 9val it = () : unit
• lex(); 9val it = () : unit
• lex(); chased: is a verb val it = () : unit the 99 chased the dog The file test.english

Named REs in SML-Lex

type lexresult = Token type pos = int; type svalue = int; fun eof () = EOF; %% digit=[0-9]; number={digit}+ ; %% [\t\ ]+ => ( lex() ) ; Anne|Bob|Spot| {number} => ( ProperNoun yytext ); a|the => ( Article yytext ); boy|girl|dog => ( Noun yytext ); walked|chased|ran|bit => ( Verb yytext ); We can name a RE Name = RE Semicolon Use a name by surrounding it in { }s english2.lex Note that all unrecognized input will raise the default exception. Not sure why we need pos and svalue

Test it out

• val f = makeStringLexer "Anne chased the dog"; val f = fn : unit -> Token
• f(); val it = ProperNoun "Anne" : Token
• f(); val it = Verb "chased" : Token
• f(); val it = Article "the" : Token
• f(); val it = Noun "dog" : Token
• f(); val it = EOF : Token

More SML

• In SML we use library functions all the time.
  • Int.toString
  • List.exists
• The list library functions are particularly useful.
  • These library functions often take a function as an argument
  • List.map : ('a -> 'b) -> 'a list -> 'b list
  • List.find : ('a -> bool) -> 'a list -> 'a option
  • List.filter : ('a -> bool) -> 'a list -> 'a list
  • List.exists : ('a -> bool) -> 'a list -> bool
  • List.all : ('a -> bool) -> 'a list -> bool
  • List.foldr : ('a * 'b -> 'b) -> 'b -> 'a list -> 'b
• It is worth studying these functions closely

Pattern

fun addone [] = [] | addone (x::xs) = (x + 1) :: addone xs fun stringy [] = [] | stringy (x::xs) = (Int.toString x) :: stringy xs fun negL [] = [] | negL (x::xs) = (not x) :: negL xs

fun map f [] = []

| map f (x::xs) = (f x) :: (map f

xs)

val ex1 = map (fn x => (x+1)) [2,3,4]; val ex1 = [3,4,5] : int list val ex2 = map Int.toString [2,5,7]; val ex2 = ["2","5","7"] : string list val ex3 = map not [true, 3 > 4]; val ex3 = [false,true] : bool list

Anonymous functions

• Study: (fn x => (x+1))
  • It is an anonymous function. A function without a name.
  • It has one parameter “x”
  • It adds one to its parameter, and returns the result. (fn x => (x+1)) 4; val it = 5 : int
• Any non-recursive function can be written anonymously.
  • (fn x => x = 5)
    • Tests if its parameter is equal to 5 map (fn x => x=5) [1,4,5,3,5]; val it = [false,false,true,false,true] : bool list
  • (fn x => fn y => (x,y))
    • Has two parameters
    • Returns a pair