Experimental Evaluation - Multimedia Networking - Lecture Slides, Slides of Computer Science

These are the Lecture Slides of Multimedia Networking which includes Variations, Layer Encoded Videos, Internet, Typical, Encoding, Commercial Streaming, Layered Encoding, Allows Easier Scaling, Variation in Quality etc.Key important points are: Experimental Evaluation, Computer Science, Quantitative, Motivation, Related Work, Methodology, Observations, Accuracy, Conclusions, Future Work

Typology: Slides

2012/2013

Uploaded on 03/27/2013

ekana
ekana 🇮🇳

4

(44)

370 documents

1 / 26

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
Experimental Evaluation in
Computer Science: A Quantitative
Study
Docsity.com
pf3
pf4
pf5
pf8
pf9
pfa
pfd
pfe
pff
pf12
pf13
pf14
pf15
pf16
pf17
pf18
pf19
pf1a

Partial preview of the text

Download Experimental Evaluation - Multimedia Networking - Lecture Slides and more Slides Computer Science in PDF only on Docsity!

Experimental Evaluation in

Computer Science: A Quantitative

Study

Outline

  • Motivation
  • Related Work
  • Methodology
  • Observations
  • Accuracy
  • Conclusions
  • Future work

Related Work

  • 1979 surveys say experiments lacking
    • 1994 say experimental CS under funded
  • 1980, Denning defines experimental CS
    • “Measuring an apparatus in order to test a hypothesis ”
    • “If we do not live up to traditional science standards, no one will take us seriously”
  • Articles on role of experiments in various CS disciplines
  • 1990 experimental CS seen as growing,
    • but in 1994  “ Falls short of science on all levels”
  • No systematic attempt to assess research

Methodology

  • Select Papers
  • Classify
  • Results
  • Analysis
  • Dissemination (this paper)

Select Comparison Papers

  • Neural Computing (72 papers)
    • Neural Computation, volume 5
    • Interdisciplinary: bio, CS, math, medicine …
    • Neural networks, neural modeling …
    • Young field (1990) and CS overlap
  • Optical Engineering (75 papers)
    • Optical Engineering, volume 33, no’s 1 and 3
    • Applied optics, opto-mech, image proc.
    • Contributors from: EE, astronomy, optics…
    • Applied, like CS, but longer history

Classify

  • Same person read most
  • Two read all, save NC

Subclasses of Design and Modeling

  • Amount of physical space (pages) for experiments - Setup, Results, Analysis
  • 0-10%, 11-20%, 21-50%, 51%+
  • To shallow? Assumptions:
    • Amount of space proportional to importance by authors and reviewers
    • Amount of space correlated to importance to research
  • Also, concerned with those that had no experimental evaluation at all

Assessing Experimental Evaluation

  • Look for execution of apparatus, techniques or methods, models validated - Tables, graphs, section headings…
  • No assessment of quality
  • But count only ‘true’ experimental work
    • Repeatable
    • Objective (ex: benchmark)
  • No demonstrations, no examples
  • Some simulations
    • Supplies data for other experiments
    • Trace driven

Observation of Major Categories

  • Majority is design and modeling
  • The CS samples have lower percentage of empirical work than OE and NC
  • Hypothesis testing is rare (4 articles out of 403!)

Observation of Major Categories

(Combine hypothesis testing with empirical)

Observation of Design Sub-Classes

  • Many more NC+OE with 20%+ than in CS
  • Software engineering (TSE and TOPLAS) worse than random

Observation of Design Sub-Classes

  • Shows percentage that have 20%+ or more to experimental evaluation

Accuracy of Study

  • Deals with humans, so subjective
  • Psychology techniques to get objective measure - Large number of users  Beyond resources (and a lot of work!) - Provide papers, so other can provide data
  • Systematic errors
    • Classification errors
    • Paper selection bias

Systematic Error: Classification

  • Classification differences between 468 article classification pairs (93 had difference, 20%)