Download OCaml Programming – Lecture Notes & Exercises and more Study notes Computer Science in PDF only on Docsity!
Foundations of Computer Science (2024-2025)
This course has two aims. The first is to teach programming. The second is to present some fundamental principles of computer science, especially algorithm design. Most students will have some programming experience already, but there are few people whose programming cannot be improved through greater knowledge of basic principles. Please bear this point in mind if you have extensive experience and find parts of the course rather slow.
The programming in this course is based on the language OCaml and mostly concerns the functional programming style. Functional programs tend to be shorter and easier to understand than their counterparts in conventional languages such as C. In the space of a few weeks, we shall cover many fundamental data structures and learn basic methods for estimating efficiency.
The first thing you will notice about this course is that there is an interactive version hosted online at https://hub.cl.cam.ac.uk/, where you can login with your Cambridge Raven identity and edit the code fragments in your browser. You are encouraged to do so – such edits will only persist in your session, and will help you to explore the world of functional programming. If you are using the web-based version, then you need to know a few concepts:
- The notebook consists of a sequence of textual and code snippets.
- The code snippets can be executed individually, and will ―remember‖ the results of the previous snippets.
- To begin with, click on Cell / Run All in the menu to execute the entire notebook.
- You can later double click on any cell and modify its contents, and press Shift+Enter to reevaluate its contents. This will only modify the current cell, so you will have to Run All again to see the effects on the whole notebook.
- While editing longer snippets, you can also press Shift+Tab while typing to get more docu- mentation hints about the code you are writing.
This course is lectured by Anil Madhavapeddy, with the practical exercises managed by Jonathan Ludlam. These notes are translated from Lawrence C. Paulson‘s earlier course on Standard ML, which had credits to David Allsopp, Stuart Becker, Gavin Bierman, Chloë Brown, Silas Brown, Qi Chen, David Cottingham, William Denman, Robert Harle, Daniel Hulme, Frank King, Jack Lawrence-Jones, Joseph Lord, Dimitrios Los, Farhan Mannan, James Margetson, David Morgan, Alan Mycroft, Sridhar Prabhu, Frank Stajano, Alex Trifanov, Thomas Tuerk, Xincheng Wang, Philip Withnall and Assel Zhiyenbayeva for pointing out errors. The current notes were ported to OCaml in 2019 by Anil Madhavapeddy, David Allsopp, and Jon Ludlam and subsequently edited by Jeremy Yallop. We thank Richard Sharp, Srinivasan Keshav, Ambroise Lafont, Vojtěch Tvrdík and Jeremy Yallop for further feedback and corrections since 2020.
Some books that are complementary to this course are:
- OCaml from the Very Beginning by John Whitington.
Chapter 1
Lecture 1: Introduction to
Programming
1.1 Basic Concepts in Computer Science
- Computers: a child can use them; nobody can fully understand them!
- We can master complexity through levels of abstraction.
- Focus on 2 or 3 levels at most!
Recurring issues:
- what services to provide at each level
- how to implement them using lower-level services
- the interface that defines how the two levels should communicate
A basic concept in computer science is that large systems can only be understood in levels, with each level further subdivided into functions or services of some sort. The interface to the higher level should supply the advertised services. Just as important, it should block access to the means by which those services are implemented. This abstraction barrier allows one level to be changed without affecting levels above. For example, when a manufacturer designs a faster version of a processor, it is essential that existing programs continue to run on it. Any differences between the old and new processors should be invisible to the program.
Modern processors have elaborate specifications, which still sometimes leave out important details. In the old days, you then had to consult the circuit diagrams.
1.1.1 Example 1: Dates
- Abstract level: dates over a certain interval
- Concrete level: typically 6 characters: YYMMDD (where each character is represented by 8 bits)
- Date crises caused by inadequate internal formats: - Digital‘s PDP-10: using 12-bit dates (good for at most 11 years) - 2000 crisis: 48 bits could be good for lifetime of universe!
Digital Equipment Corporation‘s date crisis occurred in 1975. The PDP-10 was a 36-bit mainframe computer. It represented dates using a 12-bit format designed for the tiny PDP-8. With 12 bits,
- to do so efficiently and correctly , giving the right answers quickly
- to allow easy modification as needs change - Through an orderly structure based on abstraction principles - Such as modules or classes
Programming in-the-small concerns the writing of code to do simple, clearly defined tasks. Pro- grams provide expressions for describing mathematical formulae and so forth. This was the original contribution of FORTRAN, the FORmula TRANslator. Commands describe how control should flow from one part of the program to the next.
As we code layer upon layer, we eventually find ourselves programming in the large : joining large modules to solve some messy task. Programming languages have used various mechanisms to allow one part of the program to provide interfaces to other parts. Modules encapsulate a body of code, allowing outside access only through a programmer-defined interface. Abstract Data Types are a simpler version of this concept, which implement a single concept such as dates or floating-point numbers.
Object-oriented programming is the most complicated approach to modularity. Classes define con- cepts, and they can be built upon other classes. Operations can be defined that work in appropri- ately specialised ways on a family of related classes. Objects are instances of classes and hold the data that is being manipulated.
This course does not cover OCaml‘s sophisticated module system, which can do many of the same things as classes. You will learn all about objects when you study Java. OCaml includes a powerful object system, although this is not used as much as its module system.
1.3 Why Program in OCaml?
Why program in OCaml at all?
- It is interactive.
- It has a flexible notion of data type.
- It hides the underlying hardware: no crashes.
- Programs can easily be understood mathematically.
- It distinguishes naming something from updating memory.
- It manages storage for us.
Programming languages matter. They affect the reliability, security, and efficiency of the code you write, as well as how easy it is to read, refactor, and extend. The languages you know can also change how you think, influencing the way you design software even when you‘re not using them.
What makes OCaml special is that it occupies a sweet spot in the space of programming language designs. It provides a combination of efficiency, expressiveness and practicality that is difficult to find matched by any other language. ―ML‖ was originally the meta language of the LCF (Logic for Computable Functions) proof assistant released by Robin Milner in 1972 (at Stanford, and later at Cambridge). ML was turned into a compiler in order to make it easier to use LCF on different machines, and it was gradually turned into a full-fledged system of its own by the 1980s.
The modern OCaml emerged in 1996, and the past twenty five years have seen OCaml attract a significant user base with language improvements being steadily added to support the growing commercial and academic codebases. OCaml is therefore the outcome of years of research into
programming languages, and a good base to begin our journey into learning the foundations of computer science.
Because of its connection to mathematics, OCaml programs can be designed and understood with- out thinking in detail about how the computer will run them. Although a program can abort, it cannot crash: it remains under the control of the OCaml system. It still achieves respectable efficiency and provides lower-level primitives for those who need them. Most other languages allow direct access to the underlying machine and even try to execute illegal operations, causing crashes.
The only way to learn programming is by writing and running programs. This web notebook provides an interactive environment where you can modify the example fragments and see the results for yourself. You should also consider installing OCaml on your own computer so that you try more advanced programs locally.
1.4 A first session with OCaml
In [1]: let pi = 3.
Out[1]: val pi : float = 3.
The first line of this simple session is a value declaration. It makes the name pi stand for the floating point number 3.14159. (Such names are called identifiers .) OCaml echoes the name (pi) and type (float) of the declared identifier.
In [2]: pi *. 1.5 *. 1.
Out[2]: - : float = 7.
The second line computes the area of the circle with radius 1.5 using the formula 2. We use pi as an abbreviation for 3.14159. Multiplication is expressed using *., which is called an infix operator because it is written between its two operands.
OCaml replies with the computed value (about 7.07) and its type (again float).
In [3]: let area r = pi *. r *. r
Out[3]: val area : float -> float =
To work abstractly, we should provide the service ―compute the area of a circle,‖ so that we no longer need to remember the formula. This sort of encapsulated computation is called a function. The third line declares the function area. Given any floating point number r, it returns another floating point number computed using the area formula; note that the function has type float -> float.
In [4]: area 2.
Out[4]: - : float = 12.
The fourth line calls the function area supplying 2.0 as the argument. A circle of radius 2 has an area of about 12.6. Note that brackets around a function argument are not necessary.
Now for a tiresome but necessary aside. In most languages, the types of arguments and results must always be specified. OCaml is unusual that it normally infers the types itself. However, sometimes it is useful to supply a hint to help you debug and develop your program. OCaml will still infer the types even if you don‘t specify them, but in some cases it will use a more inefficient function than a specialised one. Some languages have just one type of number, converting automatically between different formats; this is slow and could lead to unexpected rounding errors. Type constraints are allowed almost anywhere. We can put one on any occurrence of x in the function.
In [7]: let square (x : float ) = x *. x
Out[7]: val square : float -> float =
Or we can constrain the type of the function‘s result:
In [8]: let square x : float = x *. x
Out[8]: val square : float -> float =
OCaml treats the equality and comparison test specially. Expressions like if x = y then … are allowed provided x and y have the same type and equality testing is possible for that type. (We discuss equality further in a later lecture.) Note that x <> y is OCaml for 𝑥 ≠ 𝑦.
A characteristic feature of the computer is its ability to test for conditions and act accordingly. In the early days, a program might jump to a given address depending on the sign of some number. Later, John McCarthy defined the conditional expression to satisfy if true then x else y = x and if false then x else y = y.
OCaml evaluates the expression if then else by first evaluating. If the result is true then OCaml evaluates and otherwise. Only one of the two expressions and is evaluated! If both were evaluated, then recursive functions like npower above would run forever.
The if-expression is governed by an expression of type bool, whose two values are true and false. In modern programming languages, tests are not built into ―conditional branch‖ constructs but can just be part of normal expressions. Tests, or Boolean expressions, can be expressed using relational operators such as < and =. They can be combined using the Boolean operators for negation (not), conjunction (written as &&) and disjunction (written as ||). New properties can be declared as functions: here, to test whether an integer is even, for example:
In [9]: let even n = n mod 2 = 0
Out[9]: val even : int -> bool =
1.6 Efficiently Raising a Number to a Power
In [10]: let rec power x n = if n = 1 then x else if even n then power (x *. x) (n / 2) else x *. power (x *. x) (n / 2)
Out[10]: val power : float -> int -> float =
Mathematical Justification 𝑥^1 = 𝑥 𝑥^2 𝑛^ = (𝑥^2 )𝑛 𝑥^2 𝑛+1^ = 𝑥 × (𝑥^2 )𝑛.
For large n, computing powers using 𝑛+1^ 𝑛^ is too slow to be practical. The equations above are much faster. Example:
212 = 4^6 = 16^3 = 16 × 256^1 = 16 × 256 = 4096.
Instead of n multiplications, we need at most 2 lg 𝑛 multiplications, where lg 𝑛 is the logarithm of 𝑛 to the base 2.
We use the function even, declared previously, to test whether the exponent is even. Integer division (/) truncates its result to an integer: dividing 2 𝑛 + 1 by 2 yields 𝑛.
A recurrence is a useful computation rule only if it is bound to terminate. If 𝑛 > 0 then 𝑛 is smaller than both 2 𝑛 and 2 𝑛 + 1. After enough recursive calls, the exponent will be reduced to 1. The equations also hold if 𝑛 ≤ 0 , but the corresponding computation runs forever.
Our reasoning assumes arithmetic to be exact. Fortunately, the calculation is well-behaved using floating-point.
Computer numbers have a finite range, which if exceeded results in the integer wrapping around. You will understand this behaviour more as you learn about computer architecture and how modern systems represent numbers in memory.
If integers and floats must be combined in a calculation, OCaml provides functions to convert between them:
In [11]: int_of_float 3.
Out[11]: - : int = 3
In [12]: float_of_int 3
Out[12]: - : float = 3.
Chapter 2
Lecture 2: Recursion and Efficiency
2.1 Expression Evaluation
Expression evaluation concerns expressions and the values they return. This view of computation may seem to be too narrow. It is certainly far removed from computer hardware, but that can be seen as an advantage. For the traditional concept of computing solutions to problems, expression evaluation is entirely adequate.
Starting with , the expression is reduced to until this process concludes with a value. A value is something like a number that cannot be further reduced.
We write 𝐸 → 𝐸′^ to say that 𝐸 ′
is reduced to 𝐸′. Mathematically, they are equal: 𝐸 = 𝐸′, but the
computation goes from 𝐸 to 𝐸 and never the other way around.
Computers also interact with the outside world. For a start, they need some means of accept- ing problems and delivering solutions. Many computer systems monitor and control industrial processes. This role of computers is familiar now, but was never envisaged in the early days. Com- puter pioneers focused on mathematical calculations. Modelling interaction and control requires a notion of states that can be observed and changed. Then we can consider updating the state by assigning to variables or performing input/output, finally arriving at conventional programs as coded in C, for instance.
For now, we remain at the level of expressions, which is usually termed functional programming.
2.2 Summing the first n integers
In [13]: let rec nsum n = if n = 0 then 0 else n + nsum (n - 1)
Out[13]: val nsum : int -> int =
The function call nsum n computes the sum 1 + … + nz rather naively, hence the initial n in its name:
nsum 3 ⇒ 3 + (nsum 2) ⇒ 3 + (2 + (nsum 1) ⇒ 3 + (2 + (1 + nsum 0)) ⇒ 3 + (2 + (1 + 0))
The nesting of parentheses is not just an artifact of our notation; it indicates a real problem. The function gathers up a collection of numbers, but none of the additions can be performed until nsum 0 is reached. Meanwhile, the computer must store the numbers in an internal data structure, typically the stack. For large n, say nsum 10000, the computation might fail due to stack overflow.
We all know that the additions can be performed as we go along. How do we make the computer do that?
2.3 Iteratively summing the first n integers
In [14]: let rec summing n total = if n = 0 then total else summing (n - 1) (n + total)
Out[14]: val summing : int -> int -> int =
Function summing takes an additional argument: a running total. If n is zero then it returns the running total; otherwise, summing adds to it and continues. The recursive calls do not nest; the additions are done immediately.
A recursive function whose computation does not nest is called iterative or tail-recursive. Many functions can be made iterative by introducing an argument analogous to total, which is often called an accumulator.
The gain in efficiency is sometimes worthwhile and sometimes not. The function power is not iterative because nesting occurs whenever the exponent is odd. Adding a third argument makes it iterative, but the change complicates the function and the gain in efficiency is minute; for 32-bit
integers, the maximum possible nesting is 30 for the exponent 231 − 1.
2.4 Recursion vs Iteration
- ―Iterative‖ normally refers to a loop, coded using while for example (see the final lecture)
- Tail-recursion is only efficient if the compiler detects it
- Mainly it saves space (memory), though iterative code can also run faster
- Do not make programs iterative unless the gain is worth it
A classic book by Abelson and Sussman, which describes the Lisp dialect known as Scheme, used iterative to mean tail-recursive. Iterative functions produce computations resembling those that
2
Why do we need let bindings? Fast hardware does not make good algorithms unnecessary. On the contrary, faster hardware magnifies the superiority of better algorithms. Typically, we want to handle the largest inputs possible. If we double our processing power, what do we gain? How much can we increase 𝑛, the input to our function?
With sillySum, we can only go from 𝑛 to 𝑛 + 𝑛
- We are limited to this modest increase because the function‘s running time is proportional to 2. With the function npower defined in the previous
section, we can go from 𝑛 to 2 𝑛: we can handle problems twice as big. With power we can do much better still, going from 𝑛 to 𝑛.
The following table (excerpted from a 50-year-old book!) illustrates the effect of various time complexities. The left-hand column (dubbed ―complexity‖) is defind as how many milliseconds are required to process an input of size. The other entries show the maximum size of that can be processed in the given time (one second, minute or hour).
complexity 1 second 1 minute 1 hour gain
𝑛 lo
g 𝑛 𝑛^2 𝑛^3 2 𝑛
1000 60 000 3 600 000 ×
140 4 895 204 095 ×
31 244 1 897 ×
10 39 153 ×
The table illustrates how large an input can be processed as a function of time. As we increase the computer time per input from one second to one minute and then to one hour, the size of the input increases accordingly.
The top two rows (complexities and lg ) increase rapidly: for , by a factor of 60 per column. The bottom two start out close together, but 3 (which grows by a factor of 3.9) pulls well away from 𝑛 (^) (whose growth is only additive). If an algorithm‘s complexity is exponential then it can never
handle large inputs, even if it is given huge resources. On the other hand, suppose the complexity has the form 𝑐, where is a constant. (We say the complexity is polynomial .) Doubling the argument then increases the cost by a constant factor. That is much better, though if the algorithm may not be considered practical.
2.6 Comparing Algorithms: O Notation
- Formally, define provided as
- is bounded for some constant and all sufficiently large.
- Intuitively, look at the most significant term.
- Ignore constant factors as they seldom dominate and are often transitory
For example: consider 𝑛^2 instead of 3 𝑛^2 + 34 𝑛 + 433.
The cost of a program is usually a complicated formula. Often we should consider only the most significant term. If the cost is 𝑛^2 + 99 𝑛 + 900 for an input of size 𝑛, then the 𝑛^2 term will eventually dominate, even though 99 𝑛 is bigger for 𝑛 < 99. The constant term 900 may look big, but it is soon dominated by 𝑛^2.
Constant factors in costs can be ignored unless they are large. For one thing, they seldom make a difference: 2 will be better than 3 in the long run: or asymptotically to use the jargon. Moreover, constant factors are seldom stable. They depend upon details such as which hardware, operating system or programming language is being used. By ignoring constant factors, we can make comparisons between algorithms that remain valid in a broad range of circumstances.
The ―Big O‖ notation is commonly used to describe efficiency—to be precise, asymptotic complexity. It concerns the limit of a function as its argument tends to infinity. It is an abstraction that meets the informal criteria that we have just discussed. In the definition, sufficiently large means there is some constant 𝑛 0 such that |𝑓(𝑛)| ≤ 𝑐|𝑔(𝑛)| for all 𝑛 greater than 𝑛 0. The role of 𝑛 0 is to ignore
finitely many exceptions to the bound, such as the cases when 99 𝑛 exceeds 𝑛^2.
2.7 Simple Facts About O Notation
𝑂(2𝑔(𝑛)) is the same as 𝑂(𝑔(𝑛))
𝑂(𝑛^2
𝑂(log 10 𝑛) is the same as 𝑂(ln 𝑛)
- 50𝑛 + 36) is the same as 𝑂(𝑛^2 )
𝑂(𝑛^2 ) is contained in 𝑂(𝑛^3 ) 𝑂(2𝑛) is contained in 𝑂 ( 3 √
𝑂(log 𝑛) is contained in 𝑂( 𝑛)
𝑂 notation lets us reason about the costs of algorithms easily.
- Constant factors such as the 2 in 𝑂(2𝑔(𝑛)) drop out: we can use 𝑂(𝑔(𝑛)) with twice the value
- Because constant factors drop out, the base of logarithms is irrelevant.
- Insignificant terms drop out. To see that 𝑂(𝑛^2 + 50 𝑛 + 36) is the same as 𝑂(𝑛^2 ), consider that 𝑛^2 + 50 𝑛 + 36/𝑛^2 converges to 1 for increasing 𝑛. In fact, 𝑛^2 + 50 𝑛 + 36 ≤ 2 𝑛^2 for 𝑛 ≥ 51 , so
If 𝑐 and 𝑑 are constants (that is, they are independent of 𝑛) with 0 < 𝑐 < 𝑑 then - 𝑂(𝑛𝑐) is contained in 𝑂(𝑛𝑑) - 𝑂(𝑐𝑛) is contained in 𝑂(𝑑𝑛) - 𝑂(log 𝑛) is contained 𝑖𝑛𝑂(𝑛𝑐)
To say that 𝑂(𝑐𝑛) is contained in 𝑂(𝑑𝑛) means that the former gives a tighter bound than the latter. For example, if 𝑓(𝑛) = 𝑂(2𝑛) then 𝑓(𝑛) = 𝑂(3𝑛) trivially, but the converse does not hold.
2.8 Common Complexity Classes
- 𝑂(1) is constant
- 𝑂(log 𝑛) is logarithmic
- 𝑂(𝑛) is linear
- 𝑂(𝑛 log 𝑛) is quasi-linear
- 𝑂(𝑛^2 ) is quadratic
- 𝑂(𝑛^3 ) is cubic
- 𝑂(𝑎𝑛) is exponential (for fixed 𝑎)
Logarithms grow very slowly, so log complexity is excellent. Because notation ignores constant factors, the base of the logarithm is irrelevant!
of in the definition.
can double the constant factor
2
Equation Complexity 𝑇 (𝑛 + 1) = 𝑇 (𝑛) + 1 𝑂(𝑛) 𝑇 (𝑛 + 1) = 𝑇 (𝑛) + 𝑛 𝑂(𝑛 ) 𝑇 (𝑛) = 𝑇 (𝑛/2) + 1 𝑂(log 𝑛) 𝑇 (𝑛) = 2 𝑇 (𝑛/2) + 𝑛 𝑂(𝑛 log 𝑛)
To analyse a function, inspect its OCaml declaration. Recurrence equations for the cost function can usually be read off. Since we ignore constant factors, we can give the base case a cost of one unit. Constant work done in the recursive step can also be given unit cost; since we only need an upper bound, this unit represents the larger of the two actual costs. We could use other constants if it simplifies the algebra.
For example, recall our function nsum:
In [17]: let rec nsum n = if n = 0 then 0 else n + nsum (n - 1)
Out[17]: val nsum : int -> int =
Given 𝑛 + 1 , it performs a constant amount of work (an addition and subtraction) and calls itself recursively with argument 𝑛. We get the recurrence equations 𝑇 (0) = 1 and 𝑇 (𝑛 + 1) = 𝑇 (𝑛) + 1. The closed form is clearly 𝑇 (𝑛) = 𝑛 + 1 , as we can easily verify by substitution. The cost is linear.
This function, given 𝑛 + 1 , calls nsum, performing 𝑂(𝑛) work. Again ignoring constant factors, we can say that this call takes exactly 𝑛 units.
In [18]: let rec nsumsum n = if n = 0 then 0 else nsum n + nsumsum (n - 1)
Out[18]: val nsumsum : int -> int =
We get the recurrence equations 𝑇 (0) = 1 2
and 𝑇 (𝑛 + 1) = 𝑇 (𝑛) + 𝑛. It is easy to see that
𝑇 (𝑛) = (𝑛 − 1) + ⋯ + 1 = 𝑛(𝑛 − 1)/2 = 𝑂(𝑛 ). The cost is quadratic.
The function power divides its input 𝑛 into two, with the recurrence equation 𝑇 (𝑛) = 𝑇 (𝑛/2) + 1. Clearly 𝑇 (2𝑛) = 𝑛 + 1 , so 𝑇 (𝑛) = 𝑂(log 𝑛).
2.10.1 Exercise 2.
Code an iterative version of the function power.
2.10.2 Exercise 2.
Add a column to the table of complexities from The Design and Analysis of Computer Algorithms with the heading 60 hours:
complexity 1 second 1 minute 1 hour 60 hours
𝑛 lo
g 𝑛
2.10.3 Exercise 2.
𝑛^2 31 244 1 897
𝑛^3 10 39
2 𝑛^9 15
Let 𝑔 1 , …, 𝑔𝑘 be functions such that 𝑔𝑖(𝑛) ≥ 0 for 𝑖 = 1, …, 𝑘 and all sufficiently large 𝑛.
Show that if 𝑓(𝑛) = 𝑂(𝑎 1 𝑔 1 (𝑛) + ⋯ + 𝑎𝑘𝑔𝑘(𝑛)) then 𝑓(𝑛) = 𝑂(𝑔 1 (𝑛) + ⋯ + 𝑔𝑘(𝑛)).
2.10.4 Exercise 2.
Find an upper bound for the recurrence given by 𝑇 (1) = 1 and 𝑇 (𝑛) = 2 𝑇 (𝑛/2) + 1. You should be able to find a tighter bound than 𝑂(𝑛 log 𝑛).
