Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
This course focuses on the theory and practice of software analysis, including diagnosing bugs, testing, debugging, and more. The course aims to enhance existing programming skills and build better software. The course covers modern methods for improving software quality, encompassing reliability, security, and performance. The document also discusses the importance of program analysis and the different types of analyses. It also highlights the problem of security vulnerabilities in programming and the need for better testing and debugging tools.
Typology: Lecture notes
1 / 25
This page cannot be seen from the preview
Don't miss anything!
Welcome to Software Analysis and Testing. In this course, we will be diving deep into the theory and practice of software analysis, which lies at the heart of many software development processes such as diagnosing bugs, testing, debugging, and more. What this class won’t do is teach you basic concepts of programming. Instead, through a mixture of basic and advanced exercises and examples, you will learn techniques and tools to enhance your existing programming skills and build better software.
So why should you take this course? ** Visual for Bill Gates Quote ** Bill Gates once said and I quote “We have as many testers as we have developers. And testers spend all their time testing, and developers spend half their time testing. We're more of a testing, a quality software organization than we're a software organization." In this course, you will learn modern methods for improving software quality in a broad sense, encompassing reliability, security and performance. This will enable you to become a better and more productive software developer, as the aspects that we will address in this course, such as software testing and debugging, comprise over 50 % of the cost of software development. You will also be able to implement these methods in specialized tools for software diagnosis and testing tasks. An example task is systematically testing an Android application in various end-user scenarios. But let’s face it: you’re really here for the war stories.
The cause of the disaster was diagnosed to be a kind of programming error called a numeric overflow error, in a program running on the Ariane rocket’s onboard computer. The error resulted from an attempt during takeoff to convert one piece of data -- the sideways velocity of the rocket -- from a 64 - bit format to a 16 - bit format. The number was too big to fit and resulted in an overflow error. This error was misinterpreted by the rocket’s onboard computer as a signal to change the course of the rocket. This failure translated into millions of dollars in lost assets and several years of setbacks for the Ariane Program. The methods that we will learn in this course could have prevented this error. To read more about this disaster access the link provided in the instructor notes. [http://www.around.com/ariane.html] Now let’s look at another problem that is more earthly and affects everyday users of software.
While the Ariane disaster was a consequence of a programming error, at least the damage was an unintended consequence. On the other hand, malicious hackers can exploit these errors in everyday mobile and web applications to compromise the security of the underlying systems and data. This is a widespread problem, and it has been since the early days of the Internet. Several examples of programming bugs leading to security vulnerabilities you may have heard of include:
And the problem has only gotten worse with the advent of smartphones; now you too can make yourself vulnerable to programming disasters simply by installing an app. [Picture of smartphone “do you want to install” message pops up]
Dynamic program analysis infers facts about a program by monitoring its runs. Here are four examples of well-known dynamic analysis tools. Purify is a dynamic analysis tool for checking memory accesses, such as array bounds, in C and C++ programs. Valgrind is a dynamic analysis tool for detecting memory leaks in x 86 binary programs. A memory leak occurs when a program fails to release memory that it no longer needs. Eraser is a dynamic analysis tool for detecting data races in concurrent programs. A data race is a condition in which two threads in a concurrent program attempt to simultaneously access the same memory location, and at least one of those accesses is a write. Data races typically indicate programming errors, as the order in which the accesses in a data race occur can produce different results from run to run. Finally, Daikon is a dynamic analysis tool for finding likely invariants. An invariant is a program fact that is true in every run of the program.
Static program analysis infers facts about a program by inspecting its code. Here are four examples of well-known static analysis tools. Tools such as Lint, FindBugs, and Coverity inspect the source code of C++ or Java programs for suspicious error patterns. SLAM is a tool from Microsoft that checks whether C programs respect API usage rules. This tool is used by Windows developers to check whether device drivers use the API of the Windows kernel correctly. Facebook Infer is a more recent static analysis tool developed by Facebook for detecting memory leaks in Android applications. Finally, ESC/Java is a tool for specifying and verifying invariants in Java programs. We will look at an example of an invariant next.
The value of c is 42. To see why, we need to reason about only two cases over all runs of this program. In the runs where the true branch is taken, the value of z is p( 6 ) + 6 , which is 6 * 6 + 6 , which is 36 + 6 , which is 42. [Highlight line z = p( 6 ) + 6 ] In the runs where the false branch is taken, the value of z is p(- 7 ) - 7 , which is (- 7 *- 7 ) - 7 , which is 49 - 7 , which is 42 again. [Highlight line z p(- 7 ) - 7 ] Thus, the value of c is 42. We have thus shown that (z == 42 ) is a program invariant at the exit of this program. Now let us slightly change this program to call disaster whenever the value of z is not equal to 42. Then, notice that the invariant we just discovered is a useful fact for proving that this program can never call disaster!
Now let’s see how the different kinds of program analyses fare at discovering program invariants. Let’s first consider dynamic analysis. For simplicity, the shown program has only two paths. But in general, programs have loops or recursion, which can lead to arbitrarily many paths. Since dynamic analysis discovers information by running the program a finite number of times, it cannot in general discover information that requires observing an unbounded number of paths. As a result, a dynamic analysis tool like Daikon can at best detect likely invariants. From any run of the shown program, Diakon can at best conclude that (z == 42 ) is a likely invariant. It cannot prove that z will always be 42 , and that the call to disaster can never happen. This is not to say that dynamic analysis is useless. For one, the information that z might be 42 could be a useful fact. More importantly, Daikon can conclusively rule out entire classes of invariants even by observing a single run. For instance, from any run of this example program, Daikon can conclude that (z == c) is definitely not an invariant for any c other than 42. On the other hand, to conclusively determine that (z == 42 ) is an invariant, and therefore showing that the program will never call disaster, we need static analysis.
Let’s first introduce common terminology. Static analysis typically operates on a suitable intermediate representation of the program. One such representation shown here is a control-flow graph. It is a graph that summarizes the flow of control in all possible runs of the program. Each node in the graph corresponds to a unique statement in the program, and each edge outgoing from a node denotes a possible successor of that node in some execution. To achieve its stated goal, our static analysis tracks the constant values of the three variables in this program, x, y, and z, at each program point. This is called an abstract state, in contrast to a concrete state which tracks the actual values in a particular run. Since static analysis does not run the program, it does not operate directly over concrete states. Instead, it operates over abstract states, each of which summarizes a set of concrete states. As a result of this summarization, the static analysis may fail to accurately represent the value of a variable in an abstract state, which we denote using a question mark. While this ensures the termination of the static analysis even for programs with an unbounded number of paths, it can also lead the static analysis to miss variables that have a constant value. For this reason, we say that the static analysis sacrifices completeness. Conversely, whenever the analysis concludes that a variable has a constant value, this conclusion is indeed correct in all runs of the program. For this reason, we say that the static analysis is sound.
Now let’s examine how static analysis discovers invariants of the form (z == 42 ), even for programs that have an unbounded number of paths. We can pose this question in terms of a classic static analysis problem. This problem aims to find variables that have a constant value at a given program point. Consider the following example program which contains a loop. We will explain step-by-step how a static analysis discovers that variable y has the constant value 7 at the exit of this program.
Consider the following program. The analysis begins with an unknown value for variable b at the start of this program. In each of the three boxes shown, fill in the value of variable b that the analysis infers at the corresponding program point after completing its analysis. We will call these program points the loop header, the entry of the loop body, and the exit of the loop body. If the analysis cannot infer a definite value for b, enter a question mark into the box.
The value of b in the first box is 1. This is because immediately after the assignment of 1 to b, our static analysis knows that the value of b is 1. As the analysis proceeds, it discovers that the value of b at the entry of the loop body is still 1. Similarly, it discovers that the value of b at the exit of the loop body is 2. But the analysis is not done yet. It must analyze the loop again to ensure that these values are indeed sound. The analysis revisits the entry of the loop body. This time, it notices that the value of b can be 1 or 2. So it updates the value of b at the entry of the loop body to unknown. Continuing further, the analysis updates the value of b at the exit of the loop body to unknown as well. Due to these updates, the analysis analyzes the loop yet again. But this time, it concludes that the values of b at the entry and exit of the loop body have saturated. Therefore, the correct value of b in the 2 nd and 3 rd boxes is the unknown value.
Let’s review the answers. First we will focus on cost. Since dynamic analysis gathers information by running the program, its cost is proportional to the execution time of the program. A longer run thus costs more than a shorter one. Static analysis, on the other hand, gathers information by inspecting the program’s code, and therefore its cost is proportional to the size of the program’s source code. A larger program thus costs more than a smaller one. Now let’s look at effectiveness. As we saw in the example about program invariants, a dynamic analysis may miss errors, as it inspects only a finite number of runs whereas the program may contain an unbounded number of paths, some of which are not covered by those runs. We say that a dynamic analysis is “unsound”: in other words, it may produce false negatives. Static analysis, on the other hand, does not miss errors but it may report spurious issues. We say that a static analysis is “incomplete”: in other words, it may produce false positives.
You might be wondering whether it is possible for a program analysis to guarantee both soundness and completeness: no false positives nor false negatives. The answer is: not if we want the analysis to eventually finish! Even seemingly simple program properties for realistic programming languages like C and Java are undecidable. An example such property is whether a given point in a given program is reachable on some input to that program. You can find a link to recommended reading on the topic of undecidability in the instructor notes on this page. https://en.wikipedia.org/wiki/Undecidable_problem Designing a program analysis is thus an art that involves striking a suitable tradeoff between termination, soundness, and completeness. This tradeoff is typically dictated by the consumer of the program analysis. Let’s look at the primary consumers of program analysis next.
