Experimental Evaluation - Multimedia Networking - Lecture Slides, Slides of Computer Science

Download Experimental Evaluation - Multimedia Networking - Lecture Slides and more Slides Computer Science in PDF only on Docsity!

Experimental Evaluation in

Computer Science: A Quantitative

Study

Outline

• Motivation
• Related Work
• Methodology
• Observations
• Accuracy
• Conclusions
• Future work

Related Work

• 1979 surveys say experiments lacking
  • 1994 say experimental CS under funded
• 1980, Denning defines experimental CS
  • “Measuring an apparatus in order to test a hypothesis ”
  • “If we do not live up to traditional science standards, no one will take us seriously”
• Articles on role of experiments in various CS disciplines
• 1990 experimental CS seen as growing,
  • but in 1994  “ Falls short of science on all levels”
• No systematic attempt to assess research

Methodology

• Select Papers
• Classify
• Results
• Analysis
• Dissemination (this paper)

Select Comparison Papers

• Neural Computing (72 papers)
  • Neural Computation, volume 5
  • Interdisciplinary: bio, CS, math, medicine …
  • Neural networks, neural modeling …
  • Young field (1990) and CS overlap
• Optical Engineering (75 papers)
  • Optical Engineering, volume 33, no’s 1 and 3
  • Applied optics, opto-mech, image proc.
  • Contributors from: EE, astronomy, optics…
  • Applied, like CS, but longer history

Classify

• Same person read most
• Two read all, save NC

Subclasses of Design and Modeling

• Amount of physical space (pages) for experiments - Setup, Results, Analysis
• 0-10%, 11-20%, 21-50%, 51%+
• To shallow? Assumptions:
  • Amount of space proportional to importance by authors and reviewers
  • Amount of space correlated to importance to research
• Also, concerned with those that had no experimental evaluation at all

Assessing Experimental Evaluation

• Look for execution of apparatus, techniques or methods, models validated - Tables, graphs, section headings…
• No assessment of quality
• But count only ‘true’ experimental work
  • Repeatable
  • Objective (ex: benchmark)
• No demonstrations, no examples
• Some simulations
  • Supplies data for other experiments
  • Trace driven

Observation of Major Categories

• Majority is design and modeling
• The CS samples have lower percentage of empirical work than OE and NC
• Hypothesis testing is rare (4 articles out of 403!)

Observation of Major Categories

(Combine hypothesis testing with empirical)

Observation of Design Sub-Classes

• Many more NC+OE with 20%+ than in CS
• Software engineering (TSE and TOPLAS) worse than random

Observation of Design Sub-Classes

• Shows percentage that have 20%+ or more to experimental evaluation

Accuracy of Study

• Deals with humans, so subjective
• Psychology techniques to get objective measure - Large number of users  Beyond resources (and a lot of work!) - Provide papers, so other can provide data
• Systematic errors
  • Classification errors
  • Paper selection bias

Systematic Error: Classification

• Classification differences between 468 article classification pairs (93 had difference, 20%)