introduction to formal languages, Essays (university) of Programming Languages

Download introduction to formal languages and more Essays (university) Programming Languages in PDF only on Docsity!

An Introduction to

FORMAL LANGUAGES

and AUTOMATA

Fifth Edition

PETER LINZ University of California at Davis

JONES & BARTLETT LEARNING

To the Memory of my Parents

Contents

Preface

1 Introduction to the Theory of Computation

1.1 Mathematical Preliminaries and Notation Sets Functions and Relations Graphs and Trees Proof Techniques 1.2 Three Basic Concepts Languages Grammars Automata 1.3 Some Applications*

2 Finite Automata

2.1 Deterministic Finite Accepters Deterministic Accepters and Transition Graphs Languages and Dfa's Regular Languages 2.2 Nondeterministic Finite Accepters Definition of a Nondeterministic Accepter Why Nondeterminism? 2.3 Equivalence of Deterministic and Nondeterministic Finite Accepters 2.4 Reduction of the Number of States in Finite Automata*

3 Regular Languages and Regular Grammars

3.1 Regular Expressions Formal Definition of a Regular Expression Languages Associated with Regular Expressions 3.2 Connection Between Regular Expressions and Regular Languages Regular Expressions Denote Regular Languages Regular Expressions for Regular Languages Regular Expressions for Describing Simple Patterns

7.1 Nondeterministic Pushdown Automata Definition of a Pushdown Automaton The Language Accepted by a Pushdown Automaton 7.2 Pushdown Automata and Context-Free Languages Pushdown Automata for Context-Free Languages Context-Free Grammars for Pushdown Automata 7.3 Deterministic Pushdown Automata and Deterministic Context-Free Languages 7.4 Grammars for Deterministic Context-Free Languages*

8 Properties of Context-Free Languages

8.1 Two Pumping Lemmas A Pumping Lemma for Context-Free Languages A Pumping Lemma for Linear Languages 8.2 Closure Properties and Decision Algorithms for Context-Free Languages Closure of Context-Free Languages Some Decidable Properties of Context-Free Languages.

9 Turing Machines

9.1 The Standard Turing Machine Definition of a Turing Machine Turing Machines as Language Accepters Turing Machines as Transducers 9.2 Combining Turing Machines for Complicated Tasks 9.3 Turing's Thesis

10 Other Models of Turing Machines

10.1 Minor Variations on the Turing Machine Theme Equivalence of Classes of Automata Turing Machines with a Stay-Option Turing Machines with Semi-Infinite Tape The Off-Line Turing Machine 10.2 Turing Machines with More Complex Storage Multitape Turing Machines Multidimensional Turing Machines 10.3 Nondeterministic Turing Machines 10.4 A Universal Turing Machine 10.5 Linear Bounded Automata

11 A Hierarchy of Formal Languages and Automata

11.1 Recursive and Recursively Enumerable Languages Languages That Are Not Recursively Enumerable A Language That Is Not Recursively Enumerable A Language That Is Recursively Enumerable but Not Recursive 11.2 Unrestricted Grammars 11.3 Context-Sensitive Grammars and Languages Context-Sensitive Languages and Linear Bounded Automata Relation Between Recursive and Context-Sensitive Languages 11.4 The Chomsky Hierarchy

12 Limits of Algorithmic Computation

12.1 Some Problems That Cannot Be Solved by Turing Machines Computability and Decidability The Turing Machine Halting Problem Reducing One Undecidable Problem to Another 12.2 Undecidable Problems for Recursively Enumerable Languages 12.3 The Post Correspondence Problem 12.4 Undecidable Problems for Context-Free Languages 12.5 A Question of Efficiency

13 Other Models of Computation

13.1 Recursive Functions Primitive Recursive Functions Ackermann's Function μ Recursive Functions 13.2 Post Systems 13.3 Rewriting Systems Matrix Grammars Markov Algorithms L-Systems

14 An Overview of Computational Complexity

14.1 Efficiency of Computation 14.2 Turing Machine Models and Complexity 14.3 Language Families and Complexity Classes 14.4 The Complexity Classes P and NP

T

Preface

his book is designed for an introductory course on formal languages, automata, computability, and related matters. These topics form a major part of what is known as the theory of computation. A course on this subject matter is now standard in the computer science curriculum and is often taught fairly early in the program. Hence, the prospective audience for this bookconsists primarily of sophomores and juniors majoring in computer science or computer engineering.

Prerequisites for the material in this bookare a knowledge of some higher-level programming language (commonly C, C++, or Java™) and familiarity with the fundamentals of data structures and algorithms. A course in discrete mathematics that includes set theory, functions, relations, logic, and elements of mathematical reasoning is essential. Such a course is part of the standard introductory computer science curriculum.

The study of the theory of computation has several purposes, most importantly (1) to familiarize students with the foundations and principles of computer science, (2) to teach material that is useful in subsequent courses, and (3) to strengthen students’ ability to carry out formal and rigorous mathematical arguments. The presentation I have chosen for this text favors the first two purposes, although I would argue that it also serves the third. To present ideas clearly and to give students insight into the material, the text stresses intuitive motivation and illustration of ideas through examples. When there is a choice, I prefer arguments that are easily grasped to those that are concise and elegant but difficult in concept. I state definitions and theorems precisely and give the motivation for proofs, but often leave out the routine and tedious details. I believe that this is desirable for pedagogical reasons. Many proofs are unexciting applications of induction or contradiction with differences that are specific to particular problems. Presenting such arguments in full detail is not only unnecessary, but interferes with the flow of the story. Therefore, quite a few of the proofs are brief and someone who insists on completeness may consider them lacking in detail. I do not see this as a drawback. Mathematical skills are not the byproduct of reading someone else's arguments, but come from thinking about the essence of a problem, discovering ideas suitable to make the point, then carrying them out in precise detail. The latter skill certainly has to be learned, and I thinkthat the proof sketches in this text provide very appropriate starting points for such a practice.

Computer science students sometimes view a course in the theory of computation as unnecessarily abstract and of no practical consequence. To convince them otherwise, one needs to appeal to their specific interests and strengths, such as tenacity and inventiveness in dealing with hard-to-solve problems. Because of this, my approach emphasizes learning through problem solving.

By a problem-solving approach, I mean that students learn the material primarily through problem-type illustrative examples that show the motivation behind the concepts, as well as their connection to the theorems and definitions. At the same time, the examples may involve a nontrivial aspect, for which students must discover a solution. In such an approach, homeworkexercises contribute to a major part of the learning process. The exercises at the end of each section are designed to illuminate and illustrate the material and call on students’ problem-solving ability at various levels. Some of the exercises are fairly simple, picking up where the discussion in the text

leaves off and asking students to carry on for another step or two. Other exercises are very difficult, challenging even the best minds. The more difficult exercises are marked with a star. A good mix of such exercises can be a very effective teaching tool. Students need not be asked to solve all problems, but should be assigned those that support the goals of the course and the viewpoint of the instructor. Computer science curricula differ from institution to institution; while a few emphasize the theoretical side, others are almost entirely oriented toward practical application. I believe that this text can serve either of these extremes, provided that the exercises are selected carefully with the students’ background and interests in mind. At the same time, the instructor needs to inform the students about the level of abstraction that is expected of them. This is particularly true of the proof-oriented exercises. When I say “prove that” or “show that,” I have in mind that the student should think about how a proof can be constructed and then produce a clear argument. How formal such a proof should be needs to be determined by the instructor, and students should be given guidelines on this early in the course.

The content of the text is appropriate for a one-semester course. Most of the material can be covered, although some choice of emphasis will have to be made. In my classes, I generally gloss over proofs, giving just enough coverage to make the result plausible, and then ask students to read the rest on their own. Overall, though, little can be skipped entirely without potential difficulties later on. A few sections, which are marked with an asterisk, can be omitted without loss to later material. Most of the material, however, is essential and must be covered.

The fifth edition of this text introduces a substantial amount of new material. While the presentation in the fourth edition has been retained with only minor modifications, two appendices have been added. The first is an entire chapter on finite-state transducers, Appendix A. While transducers play no significant role in formal language theory, they are important in other areas of computer science, such as digital design. Students can benefit from an early exposure to this subject; if time permits it is worthwhile to do so. Due to the similarity with finite accepters, this involves few new concepts.

I also added an introduction to JFLAP, an interactive software tool that I feel is of great help in both learning the material and in teaching this course. JFLAP implements most of the ideas and constructions in this book. It not only helps students visualize abstract concepts, but it is also a great time-saver. Many of the exercises in this bookrequire creating structures that are complicated and that have to be thoroughly tested for correctness. JFLAP can reduce the time required for this by an order of magnitude. Appendix B gives a brief introduction to JFLAP and the CD that comes with the bookexpands on this. I very much recommend the use of JFLAP for both students and instructors.

Peter Linz

intellectually stimulating and fun. It provides many challenging, puzzle-like problems that can lead to some sleepless nights. This is problem solving in its pure essence.

In this book, we will look at models that represent features at the core of all computers and their applications. To model the hardware of a computer, we introduce the notion of an automaton (plural, automata ). An automaton is a construct that possesses all the indispensable features of a digital computer. It accepts input, produces output, may have some temporary storage, and can make decisions in transforming the input into the output. A formal language is an abstraction of the general characteristics of programming languages. A formal language consists of a set of symbols and some rules of formation by which these symbols can be combined into entities called sentences. A formal language is the set of all sentences permitted by the rules of formation. Although some of the formal languages we study here are simpler than programming languages, they have many of the same essential features. We can learn a great deal about programming languages from formal languages. Finally, we will formalize the concept of a mechanical computation by giving a precise definition of the term algorithm and study the kinds of problems that are (and are not) suitable for solution by such mechanical means. In the course of our study, we will show the close connection between these abstractions and investigate the conclusions we can derive from them.

In the first chapter, we look at these basic ideas in a very broad way to set the stage for later work. In Section 1.1, we review the main ideas from mathematics that will be required. While intuition will frequently be our guide in exploring ideas, the conclusions we draw will be based on rigorous arguments. This will involve some mathematical machinery, although the requirements are not extensive. The reader will need a reasonably good grasp of the terminology and of the elementary results of set theory, functions, and relations. Trees and graph structures will be used frequently, although little is needed beyond the definition of a labeled, directed graph. Perhaps the most stringent requirement is the ability to follow proofs and an understanding of what constitutes proper mathematical reasoning. This includes familiarity with the basic proof techniques of deduction, induction, and proof by contradiction. We will assume that the reader has this necessary background. Section 1.1 is included to review some of the main results that will be used and to establish a notational common ground for subsequent discussion.

In Section 1.2, we take a first look at the central concepts of languages, grammars, and automata. These concepts occur in many specific forms throughout the book. In Section 1.3, we give some simple applications of these general ideas to illustrate that these concepts have widespread uses in computer science. The discussion in these two sections will be intuitive rather than rigorous. Later, we will make all of this much more precise; but for the moment, the goal is to get a clear picture of the concepts with which we are dealing.

1.1 Mathematical Preliminaries and Notation

Sets

A set is a collection of elements, without any structure other than membership. To indicate that x is an element of the set S , we write x ∈ S. The statement that x is not in S is written x ∉ S. A set can be specified by enclosing some description of its elements in curly braces; for example, the set of

integers 0, 1, 2 is shown as

S = {0, 1, 2}.

Ellipses are used whenever the meaning is clear. Thus, { a, b,…, z } stands for all the lowercase letters of the English alphabet, while {2, 4, 6,…} denotes the set of all positive even integers. When the need arises, we use more explicit notation, in which we write

for the last example. We read this as “ S is the set of all i , such that i is greater than zero, and i is even,” implying, of course, that i is an integer.

The usual set operations are union (∪), intersection (∩), and difference ( − ) defined as

Another basic operation is complementation. The complement of a set S , denoted by consists of all elements not in S. To make this meaningful, we need to know what the universal set U of all possible elements is. If U is specified, then

The set with no elements, called the empty set or the null set , is denoted by ∅. From the definition of a set, it is obvious that

The following useful identities, known as DeMorgan's laws ,

are needed on several occasions.

A set S 1 is said to be a subset of S if every element of S 1 is also an element of S. We write this as

S 1 ⊆ S.

If S 1 ⊆ S , but S contains an element not in S 1 , we say that S 1 is a proper subset of S ; we write this as

1. The subsets S 1 , S 2 ,… Sn are mutually disjoint;

2. S 1 ∪ S 2 ∪…∪ Sn = S ;

3. none of the Si is empty.

Then S 1 , S 2 ,… Sn is called a partition of S.

Functions and Relations

A function is a rule that assigns to elements of one set a unique element of another set. If f denotes a function, then the first set is called the domain of f , and the second set is its range. We write

f : S 1 → S 2

to indicate that the domain of f is a subset of S 1 and that the range of f is a subset of S 2. If the domain

of f is all of S 1 , we say that f is a total function on S 1 ; otherwise f is said to be a partial function.

In many applications, the domain and range of the functions involved are in the set of positive integers. Furthermore, we are often interested only in the behavior of these functions as their arguments become very large. In such cases an understanding of the growth rates may suffice and a common order of magnitude notation can be used. Let f ( n ) and g ( n ) be functions whose domain is a subset of the positive integers. If there exists a positive constant c such that for all sufficiently large n

we say that f has order at most g. We write this as

If

then f has order at least g , for which we use

Finally, if there exist constants c 1 and c 2 such that

f and g have the same order of magnitude , expressed as

In this order-of-magnitude notation, we ignore multiplicative constants and lower-order terms that

become negligible as n increases.

Example 1.

Let

Then

In order-of-magnitude notation, the symbol = should not be interpreted as equality and order-of- magnitude expressions cannot be treated like ordinary expressions. Manipulations such as

are not sensible and can lead to incorrect conclusions. Still, if used properly, the order-of-magnitude arguments can be effective, as we will see in later chapters.

Some functions can be represented by a set of pairs

where xi is an element in the domain of the function, and yi is the corresponding value in its range. For

such a set to define a function, each xi can occur at most once as the first element of a pair. If this is

not satisfied, the set is called a relation. Relations are more general than functions: In a function each element of the domain has exactly one associated element in the range; in a relation there may be several such elements in the range.

One kind of relation is that of equivalence , a generalization of the concept of equality (identity). To indicate that a pair ( x , y ) is in an equivalence relation, we write

x ≡ y.

A relation denoted by ≡ is considered an equivalence if it satisfies three rules: the reflexivity rule

the symmetry rule

to υ 2. The sequence of edges ( υ 1 , υ 3 ), ( υ 3 , υ 3 ), ( υ 3 , υ 1 ) is a cycle, but not a simple one. If the edges of a

graph are labeled, we can talk about the label of a walk. This label is the sequence of edge labels encountered when the path is traversed. Finally, an edge from a vertex to itself is called a loop. In Figure 1.1, there is a loop on vertex υ 3.

Figure 1.

On several occasions, we will refer to an algorithm for finding all simple paths between two given vertices (or all simple cycles based on a vertex). If we do not concern ourselves with efficiency, we can use the following obvious method. Starting from the given vertex, say υi , list all

outgoing edges ( υi , υk ), ( υi , υl ),…At this point, we have all paths of length one starting at υi. For all

vertices υk , υl ,…so reached, we list all outgoing edges as long as they do not lead to any vertex

already used in the path we are constructing. After we do this, we will have all simple paths of length two originating at υi. We continue this until all possibilities are accounted for. Since there are only a

finite number of vertices, we will eventually list all simple paths beginning at υi. From these we

select those ending at the desired vertex.

Trees are a particular type of graph. A tree is a directed graph that has no cycles, and that has one distinct vertex, called the root , such that there is exactly one path from the root to every other vertex. This definition implies that the root has no incoming edges and that there are some vertices without outgoing edges. These are called the leaves of the tree. If there is an edge from υi to υj , then υi is said

to be the parent of υj , and υj the child of υi. The level associated with each vertex is the number of

edges in the path from the root to the vertex. The height of the tree is the largest level number of any vertex. These terms are illustrated in Figure 1.2.

At times, we want to associate an ordering with the nodes at each level; in such cases we talk about ordered trees.

Figure 1.

More details on graphs and trees can be found in most books on discrete mathematics.

Proof Techniques

An important requirement for reading this text is the ability to follow proofs. In mathematical arguments, we employ the accepted rules of deductive reasoning, and many proofs are simply a sequence of such steps. Two special proof techniques are used so frequently that it is appropriate to review them briefly. These are proof by induction and proof by contradiction.

Induction is a technique by which the truth of a number of statements can be inferred from the truth of a few specific instances. Suppose we have a sequence of statements P 1 , P 2 ,…we want to prove to

be true. Furthermore, suppose also that the following holds:

1. For some k ≥ 1, we know that P 1 , P 2 ,…, Pk are true.

2. The problem is such that for any n ≥ k , the truths of P 1 , P 2 ,…, Pn imply the truth of Pn +1.

We can then use induction to show that every statement in this sequence is true.

In a proof by induction, we argue as follows: From Condition 1 we know that the first k statements are true. Then Condition 2 tells us that Pk +1 also must be true. But now that we know that

the first k + 1 statements are true, we can apply Condition 2 again to claim that Pk +2 must be true, and

so on. We need not explicitly continue this argument, because the pattern is clear. The chain of reasoning can be extended to any statement. Therefore, every statement is true.

The starting statements P 1 , P 2 ,… Pk are called the basis of the induction. The step connecting Pn

with Pn+1 is called the inductive step. The inductive step is generally made easier by the inductive

assumption that P 1 , P 2 ,…, Pn are true, then argue that the truth of these statements guarantees the truth

of Pn + 1. In a formal inductive argument, we show all three parts explicitly.

Example 1.