Download Research Methods in Software Engineering: Deductive Models & Experiments - Prof. Victor R. and more Study notes Computer Science in PDF only on Docsity!
Experimentation in Software Engineering:
Reading Studies
The Software Engineering Discipline
Software techniques, methods, models. etc. need to be validated via experimentation refined and tailored to the application environment logically or physically integrated be easily transferred into practice
There is a need to
understand the relationships between process and product learn and evolve our knowledge based upon experience
We need an experimental, evolutionary software development framework that deals with the symbiotic relationship between research and development so that learning can take place in a practical way
Experimentation can take many forms
Research Paradigms
Need research to establish a scientific and engineering basis for software engineering
Required research methods involve the need to build, analyze, and evaluate
models of the software processes and products various aspects of the environment, e.g the people, the organization and the interactions of these models.
The goal is to develop the conceptual scientific foundations of software engineering upon which future researchers can build.
This is often a process of
discovering and validating small but important concepts that can be applied in many different ways and that can be used to build more complex and advanced ideas rather than merely providing a tool or methodology without
experimental validation of its underlying assumptions or careful analysis and verification of its properties
Research Paradigms
Definitions
A fact is information obtained through direct observation
A hypothesis is an educated guess that precedes an experiment
An experiment is -- a test, trial or tentative procedure policy; -- an act or operation for the purpose of discovering something unknown or of testing a principle, supposition, etc.; -- an operation carried out under controlled conditions in order to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law
Research Paradigms
The Scientific Method is an inductive paradigm which can be used to: understand the software process, product, people, environment extract models from the world that explain underlying phenomena, and evaluate if the model is representative of the phenomenon observed
Example: an attempt to understand the way software is developed by an organization to see if their process model can be abstracted or a tool can be built to automate the process
There are two variations of the inductive paradigm which we will call the
engineering method empirical method
Research Paradigms
The Engineering Method:
observe existing solutions, propose better solutions, build/develop, measure and analyze, and repeat the process until no more improvements appear possible.
This version of the paradigm is
an evolutionary improvement oriented approach assumes models exist modifies model to improve the thing being studied
Example: study improvements to methods or demonstrate that a tool performs better than its predecessor relative to certain characteristics
Research Paradigms
The Empirical Method:
propose a model, develop statistical/qualitative methods, apply to case studies, measure and analyze, validate the model and repeat the procedure.
This version of the paradigm is
a revolutionary improvement oriented approach proposes a new model studies effects of process or product suggested by the new model
Example: proposal of a new method or tool and validation that the model or tool is an advance over current models or tools
Research Paradigms
There must be a rationale for collecting data.
Experiments must be designed to acquire information useful for the building of a suitable description (model or theory) of the systems under study. It is an approach to model/theory/law building.
Experimentation alone is of no value if there is no underlying framework or context where experimental results can be interpreted.
Other issues involved in these inductive, experimental methods include
- the types of experimental design appropriate for different environments,
- whether the experiment is exploratory or confirmatory,
- the validity of the data collected,
- the cost of the experiment,
- the problems of reproducibility, etc.
The Experimental Discipline
Experimental Classifications
Level of variable relationship
Descriptive: there may be patterns in the data but the relationship among the variables has not been examined
Correlational: the variation in the dependent variable(s) is related to the variation of the independent variable (s)
Cause-effect: the treatment variable(s) is the only possible cause of variation in the dependent variable(s)
The Experimental Discipline
Experimental Classifications
Experience of Subjects novice: students or individuals not experienced in domain experts: practitioners or people with experience in domain
Experimental Setting In vivo: in the field under normal conditions In vitro: in the laboratory under controlled conditions
Type of Study Experiment: at least one treatment or controlled variable Observational study: no treatment or controlled variables
The Experimental Discipline
Experimental Classifications
Types of Analysis
Quantitative Analysis
- obtrusive controlled measurement
- objective
- verification oriented
Qualitative Analysis
- naturalistic and uncontrolled observation
- subjective
The Experimental Discipline
Experimental Classifications
Study
- an act to discover something unknown or of testing a hypothesis
- can include all forms of quantitative and qualitative analysis
Studies can be
- experimental
- driven by hypotheses; quantitative analysis
- controlled experiments
- quasi-experiments or pre-experimental designs
- observational
- driven by understanding; qualitative analysis dominates
- qualitative/quantitative study
- pure qualitative study
Factors Jeopardizing Validity
- There are several factors that can jeopardize the validity of an experimental design
- They can be broken into internal and external validity
- Internal validity is the basic minimum without which an experiment is uninterpretable - Did in fact the experimental treatments make any difference in this specific experimental instance?
- External validity deals with the issue of generalizability
- To what populations, settings, treatment variables, and measurement variables can this effect be generalized? - Campbell & Stanley, Experimental and Quasi-experimental Designs for Research
Internal Validity
Eight different classes of extraneous variables, which, if not controlled in the experimental design, might produce effects confounded with the effect of the experimental stimulus.
- History - the specific events occurring between the first and second measurement in addition to the experimental value, creating rival hypotheses (O1 X O2)
- Maturation - processes within the respondents operating as a function of the passage of time per se (not specific to the particular events), including growing older, hungrier, more tired, etc.
- Testing - the effects of taking a first test upon the scores of a second testing
- Instrumentation - changes in the calibration of a measuring instrument or changes in the observers or scorers used, may produce changes in the obtained measurements. - Campbell & Stanley, Experimental and Quasi-experimental Designs for Research
Internal Validity
- Statistical Regression - operating where groups have been selected on the basis of their extreme scores, i.e., tendency toward the mean
- Selection - biases resulting in differential selection of respondents for the comparison groups (X O1, O2)
- Experimental Mortality - differential loss of respondents from the comparison groups
- Selection-Maturation Interaction, etc. - any of the extraneous variables can have a combined effect that can be mistaken for the effect of the experimental variable - Campbell & Stanley, Experimental and Quasi-experimental Designs for Research
External Validity
The factors jeopardizing external validity or representativeness are:
- Testing and X : Reactive/Interaction Effect of Testing - a pretest might increase or decrease the respondent’s sensitivity or responsiveness to the experimental variable, thus making the results obtained for a pre-tested population unrepresentative of the effects of the experimental variable for the unpretested universe from which the experimental respondents were selected
- Selection and X : Interaction effects of selection biases and the experimental variable - Campbell & Stanley, Experimental and Quasi-experimental Designs for Research
True Experimental Designs
Design 4: The pretest post test Control Group design R O 1 X O 2 R O 3 O 4
Design 5: The Solomon Four group design
R O 1 X O 2 R O 3 O 4 R X O 5 R O 6
Design 6: Posttest Only Control Group Design
R X O 1 R O 2
- Campbell & Stanley, Experimental and Quasi-experimental Designs for Research
True Experimental Designs
Factorial Designs: Several treatments (ala Design 6) R X 1 O 1 R X 2 O 2 R X 3 O 3 … R Xn On
Can be done with Design 4 and 5 also
Can be done with a control group as well
- Campbell & Stanley, Experimental and Quasi-experimental Designs for Research
Quasi-Experimental Designs
When the experimenter lacks full control over the scheduling of experimental stimuli, something like an experimental design can be introduced
Time Series Design O 1 O 2 O 3 O 4 X O 5 O 6 O 7 O 8
Equivalent Time Samples Design X 1 O 1 , X 2 O 2 , X 1 O 3 , X 2 O 4 , (^) .. …
Non-Equivalent Control Group Design O X O O O
Campbell & Stanley, Experimental and Quasi-experimental Designs for Research
The Experimental Discipline
How do we combine experiments?
There are several different approaches to experimenting in the software domain
The approaches vary in
•Level of variable relationship
- Level of confidence in results, insights gained
- Experience of subjects
- Environmental setting
- Balance between quantitative/qualitative research
•Cost
The Experimental Discipline
Experimental Study Classifications
Observational studies
- qualitative/quantitative study
- pure qualitative study
Qualitative/quantitative analysis: observer has identified, a priori, a set of variables for observation There are a large number of case studies and some field studies
- in vivo
- descriptive
- experts Pure qualitative analysis: no variables isolated a priori, open
- deductions made using non-mathematical formal logic e.g., verbal propositions Found only one pure qualitative study, a Field Qualitative Study, in vivo, descriptive, experts
observation
The Experimental Discipline
Classes of Observational Studies
Observational Studies
Variable Scopes
A priori defined No a priori defined variables variables
# of
Sites
One Case Study Case Qualitative Study
More than Field Study Field Qualitative One Study
The Experimental Discipline
Sign of maturity in a field: level of sophistication of the goals of an experiment understanding interesting things about the discipline
For software engineering that might mean:
Can we build models that allow use to measure and differentiate processes and products?
Can we measure the effect of a change in a particular process variable on the product variable?
Can we predict the characteristics of a product (values of product variable) based upon the model of the process (values of the process variables), within a particular context?
Can we control for product effects, based upon goals, given a particular set of context variables?
The Experimental Discipline
Sign of maturity in a field: a pattern of knowledge built from a series of experiments
Does the discipline build on prior (knowledge, models, experiments).
Was the study an isolated event?
Did it lead to other studies that made use of the information obtained from it Have studies been replicated under similar or differing conditions?
Does the building of knowledge exist in one research group or environment, or has it spread to others - researchers building on each other's experimental work?
For example, inspections, in general, are well studied experimentally
However, there has been very little combining of results, replication, analysis of the differentiating variables
Dimensions of a Reading Technique
- Input object: Requirements, specification, design, code, test plan,…
- Output object: Set of anomalies
- Approach: Sequential, path analysis, stepwise abstraction,…
- Formality: Reading, correctness demonstrations,…
- Emphasis: Fault detection, traceability, performance,…
- Method: Walk-throughs, inspections, reviews,…
- Consumers: User, designer, tester, maintainer,…
- Product qualities: Correctness, reliability, efficiency, portability,…
- Process qualities: Adherence to method, integration into process,…
- Quality view: Assurance, control,...
Reading Techniques
Early experiments (Hetzel, Meyers) showed very little difference between reading and testing
But reading was simply reading, without a technological base
We discuss a series of experiments at the University of Maryland and at NASA used to learn about, evaluate, and evolve reading techniques
This example
- shows multiple experimental designs
- provides a combination of evaluation approaches
- offers insight into the effects of different variables on reading
The experiments start with the early reading vs. testing experiments to various Cleanroom experiments to the scenario based reading techniques currently under study
EXPERIMENTAL LEARNING MECHANISMS
Series of Studies
# Projects
One More than one
# of Teams
per Project
One 3. Cleanroom 4. Cleanroom (SEL Project 1) (SEL Projects, 2,3,4,...)
More than 2. Cleanroom 1. Reading vs. Testing one at Maryland 5. Scenario reading vs. ...
EVALUATION OF A PROCESS
When introducing any form of process, method or tool, the organization needs to evaluate its effectiveness
That effectiveness of a process can be measured by higher than normal quality cheaper development costs improved cycle time to delivery improved product functionality more predictable behavior … It is important to understand the relationship between the process and the product It is important to have a data as a basis of comparison
