Sohail Aslam Compiler Construction Notes
1
L
Le
ec
ct
tu
ur
re
e
6
6
How to Describe Tokens?
Regular Languages are the most popular for specifying tokens because
• These are based on simple and useful theory,
• Are easy to understand and
• Efficient implementations exist for generating lexical analysers based on such
languages.
Languages
Let Σ ?be a set of characters. Σ is called the alphabet. A language over Σ is set of strings
of characters drawn from Σ.? Here are some examples of languages:
• Alphabet = English characters
Language = English sentences
• Alphabet = ASCII
Language = C++, Java, C# programs
Languages are sets of strings (finite sequence of characters). We need some notation for
specifying which sets we want. For lexical analysis we care about regular languages.
Regular languages can be described using regular expressions. Each regular expression is
a notation for a regular language (a set of words). If A is a regular expression, we write
L(A) to refer to language denoted by A.
Regular Expression
A regular expression (RE) is defined inductively
a ordinary character from Σ
ε the empty string
R|S either R or S
RS R followed by S (concatenation)
R* concatenation of R zero or more times (R* = ε|R|RR|RRR...)
Regular expression extensions are used as convenient notation of complex RE:
R? ε | R (zero or one R)
R+ RR* (one or more R)
(R) R (grouping)
[abc] a|b|c (any of listed)
[a-z] a|b|....|z (range)
[^ab] c|d|... (anything but ‘a’‘b’)
