Understanding the Systematic and Random Errors in Video ..., Lecture notes of Computer Vision

Download Understanding the Systematic and Random Errors in Video ... and more Lecture notes Computer Vision in PDF only on Docsity!

Understanding the Systematic and Random Errors in Video Sensor

Data

Gerda Kamb erova

GRASP Lab oratory Department of Computer and Information Science University of Pennsylvania

i

Abstract The purp ose of this rep ort is to help computer vision researchers to understand the video sensor data, and hence, utilize b etter the data in vision algorithms, and also evaluate correctly and metho dically the results of the algorithms. The vision sensors are complex electronic systems and as such exhibit systematic and random errors. Unfortunately imp ortant sp eci cations regarding cameras are not standardly and unambiguously provided by manufacturers. Such sp eci cations are necessary for some scienti c and engineering applications. In this rep ort we present the ma jor comp onents of the imaging system, and give the main parameters and noise sources. There is no agreement in the literature (and manufacturers' do cumentation) on the nomenclature, de nitions, measurement units and/or the conditions under which these parameters and noise levels are measured. As a general rule, video cameras are for qualitative imaging (this do es not apply for scienti c and sp ecial purp ose cameras). In order to p erform quantitative measurements radiometric correction of the data has to b e p erformed. We use at eld correction pro cedure, [PH96], to account for systematic errors. We show the p ositive e ect this pro cedure has on multicamera applications (in particular on disparity map computation). In addition we have also reviewed the noise mo dels and the radiometric correction pro cedure of Healey amd Kondepudy [HK94].

ii

• 1 Intro duction
• 2 The Imaging System Overview
  • 2.1 The Camera : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
  • 2.2 The Image Transmission: form the Camera to the Framegrabb er : : : : : : : : : : :
  • 2.3 The Framgrabb er and Analog-to-Digital Converter (ADC) : : : : : : : : : : : : : : :
  • 2.4 A Camera Mo del : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
• 3 Parameterization of the Vision Sensor and Its Uncertainties
  • 3.1 Camera Related Noise : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.1.1 Photon Shot Noise : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.1.2 Read Noise : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.1.3 Pattern Noise : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
  • 3.2 Camera Related Parameters : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.1 Quantum Eciency of the CCD : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.2 Sp ectral Resp onsitivity of the CCD : : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.3 Charge Transfer Eciency of the CCD : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.4 Minimal and Maximal Signals : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.5 Spatial Resolution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.6 Temp oral Resolution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.7 Signal-to-Noise ratio (SNR) : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.2.8 Dynamic Range of the Camera and the Video Sensor : : : : : : : : : : : : : :
  • 3.3 Framegrabb er Related Parameters and Noise : : : : : : : : : : : : : : : : : : : : : :
    • 3.3.1 Geometric Distortions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.3.2 Radiometric Distortions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
  • 3.4 Discretization Related Noise and Distortions : : : : : : : : : : : : : : : : : : : : : :
    • 3.4.1 Aliasing : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
    • 3.4.2 Quantization error : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
• 4 Examples of Sensor Parameters and Pattern Noise
  • 4.1 Background and Pattern Noise in Dark Images : : : : : : : : : : : : : : : : : : : : :
  • 4.2 Photoresp once Nonuniformities in Flat Fields : : : : : : : : : : : : : : : : : : : : : :
  • 4.3 Parameters Rep orted in some Manufacturer's Data Sheets : : : : : : : : : : : : : : :
• 5 Noise Mo dels and Estimation Pro cedures of Healey and Kondepudy
  • 5.1 The Image Mo del of Theoretical Imp ortance : : : : : : : : : : : : : : : : : : : : : : :
  • 5.2 The Image Mo del Used in the Computations : : : : : : : : : : : : : : : : : : : : : :
  • 5.3 Estimation Pro cedures : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : - Level : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5.3.1 An Estimation of Total Noise Variance for a Flat Field of \Reasonably" High
    • 5.3.2 Estimation of the ampli er gain and the signal-indep end ent noise variance : :
    • 5.3.3 Estimating FPN in dark images and its variance : : : : : : : : : : : : : : : :
    • 5.3.4 Estimating the factor mo deling PRNU | K (a; b) : : : : : : : : : : : : : : : :
• 6 Radiometric Correction Pro cedures
  • 6.1 Flat- eld Correction Used in the Disparity Map Exp eriment : : : : : : : : : : : : : :
  • 6.2 The Radiometric Correction of Healey and Kondepudy : : : : : : : : : : : : : : : : :
  • 6.3 Radiometric Correction Metho ds of Beyer : : : : : : : : : : : : : : : : : : : : : : : :
• 7 A Disparity Map Computation | an Example of a Multicamera Algorithm
  • 7.1 Exp erimental setup : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
  • 7.2 Tests conducted : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
  • 7.3 Evaluation : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
• 8 Conclusions

1 Intro duction

We study the video sensor as a measuring device, in particular, its systematic and random measure- ment errors. The purp ose of this rep ort is to help computer vision researchers: (i) to understand video sensor data, and hence utilize b etter the data in vision algorithms; and (ii) to evaluate cor- rectly and metho dically the results of the algorithms. Our nal goal is to present a digital image mo del, accounting b oth for systematic and random measurement errors. These mo dels will b e used in the p erformance evaluation and characterization of subsequent vision algorithms. We will identify the calibration steps necessary for the joint \equalization" of multiple cameras. These exp eriments and pro cedures should b e repro ducible in a typical vision engineering lab oratory. In most of the engineering applications in rob otics and machine vision it is desirable to guarantee satisfactory accuracy and precision in minimum (real) time at moderate cost. Thus by de ning an appropriate cost functional, a balance b etween precision/accuracy | monetary cost | processing time has to b e struck. The most p opular sensor in computer vision applications is the analog charge-coupled device camera (CCD camera/solid-state camera) paired with a framegrabber^1. The sensing surface of the camera is an array of photo detecting elements. The imaging pro cess is based on the principle of converting photons into electric current. The main stages in the CCD imaging are: (i) the generation and capture of charges induced by the incoming photons; (ii) the collection of these charges into charge packets; (iii) the transp ort of all charge packets to an output no de; (iv) the conversion of the total charge of each packet into an ampli ed measurable quantity (voltage). The output of the CCD camera is an analog signal. This signal is rst ampli ed by an o -chip ampli er, and then resampled and digitized by a framegrabb er. The output of the framegrabb er is a digital image^2 which is a discrete representation of the scene b eing imaged (discrete, b oth, spatially and in intensity). The digital image is stored in a computer memory for subsequent pro cessing by computer vision mo dules. Digital images are input data to vision algorithms. When the digital images are used as data for inferring accurate and precise information ab out a real scene it is imp ortant that the video sensor (i.e. CCD camera and framegrabb er) is geometrically and radiomertrically calibrated. Algorithms for geometric calibration (recovering the intrinsic and extrinsic camera parameters, and their accuracy) have b een sub ject of extensive research in the computer vision community, see [HS93], [Fau93] for a textb o ok intro duction; [DA89] and [WCH92] for a review of pap ers; and [GM96] for recent results. The work on radiometric calibration and noise mo dels of CCD cameras is limited, [HK94]. Radiometric calibration pro cedures are studied in videometric applications [LF90], photogrammetry [Bey92] , astronomy [SHL+^95 ], [Mcl89], video microscopy [Ino89]. Charged-coupled devices and solid-state imagers are analyzed by designers, and test pro cedures for their evaluation, with the use of precise measuring equipment, is discussed in [Bar75], [BL80], [JEC+^84 ], [JKE85], [JEW+^95 ]. For recent, complete, and comprehensive intro duction to solid-state imaging and related issues see [The95] and [Hol96]. An indepth treatment of the video signal formats can b e found in [Enn71, Poy96]. A short and clear intro duction is given in [LF90, Bey90]. When multiple sensors are used in applications requiring hard p erformance guarantees, cor- recting for errors and obtaining ob jective con dence measures for the uncertainty of the results cannot b e neglected. The use of vision algorithms with physically di erent sensors, necessitates

(^1) In the text we use as synonyms frame grabb er and digitizer (^2) The digital image is a two dimensional array of numb ers.

CAMERA

LENS

DIGITIZER

A/D

COMPUTER MEMORY

digitized image

low pass

CCD CAMERA:

CCD CAMERA AND DIGITIZER:

LOSS OF INFORMATION AND SYSTEMATIC AND RANDOM NOISE

OPTICS: blur geometric dstortions

photon (shot) noise fixed pattern noise (systematic) read noise

• trapping
• charge transfer
• reset
• output amplifer
• background (dark current, internal luminance)

horizontal scale (systematic, due to mismatch in the camera pixel clock and digitizer clock frequencies) line jitter quantization error systematic noise introduced by the clocking of electronic components

; ; ;

; ccd image ;

The Output: discretized spatialy and in intensity; noisy.

Figure 1: The imaging system: in { incoming light from a scene; out { a digitized image

columns of photosensitive area

1 2 1 2 3 4

(shielded from light)

First move charge packets from pixels to vertical shift register, and then, in parallel, to readout

First move charge packets from image area to storage area, and then from storage to readout

a) Interline transfer b) Frame transfer

image area

storage area

vertical CCD shift register

horizontal CCD shift register (shielded^ from^ light)

Figure 2: Charge transfer organizations: a) interline; b) frame

of the digital image a pel. In the text, we use pixel to denote sel or p el, which one it will b e clear from the context. A row of sels is mapp ed into a row of p els. Ideally, a p el should corresp ond to a an exact sel. Often this is not the case. Clearly, if the camera pixel clo ck and the digitizer clo ck have di erent p erio ds, the numb er of sels p er line in the camera and p els, p er row in the digitized image may di er, also the resolution of the camera and digitized image will di er (we explain this later in more detail). In digital cameras the A/D conversion is done in the camera system, so there is one to one corresp ondence b etween sels and p els. In this rep ort we will not discuss digital cameras (they more reliable, and advanced, but also exp ensive { and for the last reason not very p opular yet in computer vision applications.)

2.4 A Camera Mo del

The camera output is analog, in [Hol96] the following mo del is given,

Vcamer a = ne

GG 1 q C

where q = 1 : 6 E 19 coul is the electron charge, G and G 1 denote the gains of the on-chip and o -chip ampli ers, resp ectively, ne is the total numb er of electrons in the charge packet, and C is the capacitance of the output no de. The device output conversion gain (OCG), GqC , is measure in units V/electron. \Charge conversion values typically range from 0.1 Ve^ to 10 Ve^ ", [Hol96].A charge packet contains a photo electrons and also \noise" electrons (thermally generated, induced from cross talk b etween electronic comp onents, or simply b ecause of physics of the device). The camera output dep ends on the illumination (the typ e of source and the wavelength, , of the photons), optics (f-numb er, ap erture, fo cal length, magni cation), and detector quantum eciency (generated electrons p er photon). Figure 3 (page 6) represents the transfer of the signal from the scene, through the detector (characterized by its quantum eciency/ sp ectral resp onsitivity), the on-chip ampli er, the o -chip ampli er and A/D converter. The signal is measurable quantity while the numb er of photo electrons,

Integration time Detector

Output node amplifier

Off-chip amplifier

Digital AnalogOutput Output

R G G ADC M

q q

Vcamera

ne Vsignal 1

DN

Figure 3: From photon input to camera output( numb er of electrons, voltage from the on-chip ampli er or from the o -chip ampli er, or digital numb er, DN, after digitization.

npe , is calculated using the mo del, [Hol96],

npe =

Z  2

Rq ()

Lq (; T )AD tint F 2 (1 + Moptics )^2

optics ()Tatm d;

where AD is the area of the detector, Lq is the illumination/radiance, Rq is the quantum e- ciency/sp ectral resp onsitivity, tint is the integration time, Moptics is the magni cation of the optics^5 , F is the f-numb er^6 , Tatm is the transform function of the atmosphere and optics is the transmittance of the optical system, and T is the absolute temp erature.

3 Parameterization of the Vision Sensor and Its Uncertainties

We review parameters and factors which characterize the video sensor^7 (camera and digitizer) and limit its p erformance. Along with this, we p oint out geometric and radiometric uncertainties and discrepancies in the digital image. These are due to the optics, the CCD camera, the joint op eration of the camera and the digitizer, and to the discretization pro cess. The geometric distortions related to optics has b een analyzed extensively [Sla80]. Discretization e ects are in the center of the signal pro cessing research [Jah93]. Both of these are imp ortant, but out of the scop e of our rep ort. Any undesired factors which cause discrepancies in the output signal we consider noise. The noise may b e deterministic, systematic, or random. We fo cus on systematic and random noise which originate in the CCD camera and the frame grabb er. We want to study all noise comp onents. This rep ort shows that not only random but also systematic errors have to b e accounted for. \The magnitude of each noise comp onent must b e quanti ed and its e ect understo o d. ... predicted system p erformance may deviate signi cantly from actual p erformance if signi cant noise is present. It is essential to understand what limits the system p erformance, so that intelligent improvements are made.", [Hol96], page 90.

3.1 Camera Related Noise

The total random noise^8 of the CCD has three ma jor comp onents: photon (shot) noise, read noise (has many di erent comp onents), and pattern noise. Often the total noise is mo deled as sum of shot noise, read noise and pattern noise

< ncamer a >=

q < n^2 shot > + < n^2 r ead > + < n^2 patter n >;

where < n^2 ::: > denotes the variance of the corresp onding noise comp onent, and < n::: > the standard deviation. A graph of the noise vs the signal level is called a photon transfer curved (intro duced by Janesick). From here the noise levels and saturation levels can b e derived and the dynamic range sp eci ed [Hol96], p 115. The total noise is dominated by the read noise at low signal levels, by the pattern noise at levels close to saturation, and by the photon shot noise in b etween [JEC+^ 84].

3.1.1 Photon Shot Noise

This noise is related to the quantum nature of light, the natural variation of the incident photon ux. The total numb er of photons emitted by a steady light source over a time interval varies according to a Poisson distribution. Any \shot noise limited" pro cess exhibits Poisson distribution.

(^5) Moptics = the distance, R 2 , from the detector to the lens divided by the distance, R 1 , from the source to the lens. (^6) For a circular ap erture F = f l =D where f l is the e ective fo cal length, f l = 1 =R 1 + 1 =R 2 , and D is the diameter of the ap erture. (^7) The video sensor is a camera plus a frame grabb er. (^8) In the literature the noise rep orted is rms in electrons

In some cameras certain numb er of pixels are shielded from light and are used to estimate the average dark current which then is subtracted from the signal. This is not very precise since the dark current varies from pixel to pixel and also the average of dark current over the shielded (so called dark pixels) is di erent than the average dark current over the \active" sels (removing the average do es not remove the variability in dark current). Anyway by removing the average dark current what is achieved is full use of the dynamic range of the sensor, but the shot noise in the dark current electrons cannot b e removed { it is alway present. Dark current generation is pixel dependent. By averaging dark images a systematic component, so cal led xed pattern in the dark images is make prominent. This component can be subtracted from the images, thus removing the systematic component due to dark current generation, but the variation due to shot noise in dark current cannot be removed | always is present.

2. Fat zero: Fat zero is intro duced to aid the charge transfer eciency and the consistency of the quantum eciency. Optically generated fat zero follows Poisson distribution. If the fat zero is electrically intro duced, on input, it is less than shot noise.
3. Internal luminance: The internal luminance in the CCD device may have variety of sources. One source is the clo cking of the voltages to the gates which control the p otential well levels. It is manifested in an exp onential decline in average line intensity could b e detected. (This is con rmed by our exp eriments with dark images.) The phenomena is explained by generation of long wave photons by the clo cking of the register, which photons get absorb ed in the lines close to the register, thus increasing the background charges [JEC+^84 ]. Statistically this noise is mo deled by Poisson distribution. Second source of luminance is di usion. It is related to input-output mechanisms. This phenomena explains the \radiation" of light in the CCD dark image from the p osition of the output ampli er. To prevent the contribution to dark current from output register and ampli er, some cameras have additional dark pixels along the horizontal shift register and the ampli er. The most damaging source of luminance are blemishes. These are single sels that get saturated fast. This is a result from defects in the sel gates [JEC+^84 ].

 Trapping noise. Trapping noise is caused by random variations in the \trapping" states of CCD: charges get trapp ed, and also they are kept trapp ed for some random p erio d. This typ e of noise is also very much technology dep endent, and for the so called buried-channel CCD is very low, on the order of 5 electrons [Mcl89]^10.

 Reset (k T C ) noise. When a charge packet arrives at the output no de, it pro duces a voltage change. To measure the voltage of a charge packet a reference voltage level is needed. The readout capacitors are reset to a nominal voltage level at each readout cycle. The reset noise relates to the uncertainty this voltage level. The rms reset noise in electrons is

< nr eset >=

p k T C q e^ r ms;

(^10) One should keep in mind that in astronomy application s, where very low level signals have to b e detected, high p erformance digital co oled cameras are usually used. We assume that the numb ers cited are for digital cameras.

where q is the electron charge, k is the Boltzmann constant, T is the absolute temp erature, and C is the capacitance in picofarads. The reset noise represent the uncertainty at the measuring stage. Reducing the capacitance, reduces the reset noise (also the output conversion gain, Gq =C , increases). Co oling also reduces the reset noise but it is not that ecient as the reduction on the dark current generation. The reset noise is e ectively removed by a electronic pro cessing technique called correlated double sampling [Mcl89].

 Ampli er noise The ampli er noise is asso ciated entirely with the output no de. It may have two comp onents: a white noise (due to thermal generation) and a comp onent intro duced by interaction of charges and \traps" present in the transistor channel (called 1/f noise). Ana ampli er noise is asso ciated with the on-the chip ampli er, also there may b e an ampli er noise comp onent asso ciated with the o chip ampli er (if such an ampli er is present). By go o d manufacturing, this noise can b e reduced substantially, \typically ab out 6 electrons or less" [Mcl89].

3.1.3 Pattern Noise

Both, the xed pattern noise in dark current and the photoresp once nonuniformity are called pattern noise. They are manifested as spatial nonuniformities most prominent in averaged dark images and averaged at elds (see Sections 4.1 and 4.2). The pattern noise contributes to the nonuniformity of the individual sel resp onses to a scene of uniform brightness. We use a at elding pro cedure to correct for the pattern noise (this takes care of the systematic comp onents in the noise).

1. Fixed Pattern Noise (FPN) of the Dark Current Dark current can originate at di erent lo cations in the CCD but has, in all cases, to do with the irregularities in the crystal structure of the silicon (metal impurities and crystal defects). This gives rise to the so called xed pattern noise in the dark current. This xed pattern noise is signal-indep en dent and is additive.
2. Photoresp once Nonuniformities (PRNU) of the CCD Array Di erences in the re- sp onsitivity of individual pixels in the presence of light lead to Photoresp once nonuniformity. This is also a typ e of pattern noise, but it is signal dep endent and is mo deled with multi- plicative factor of the photo electrons shot noise ([Hol96], p 113),

< nP RN U >= U npe ; (3:1)

where U varies with the pixel.

3.2 Camera Related Parameters

The following parameters are commonly used to measure and compare the p erformance of the CCD arrays: sp ectral resp onse, minimum signal, maximum signal, dynamic range, pixel to pixel uniformities, and output conversion gain. Other parameters, for full characterization, include the sp ectral quantum eciency, the charge transfer eciency, full well capacity, linearity, pixel nonuniformity, signal to noise ratio. Read noise is an imp ortant parameter which was discussed in Section 3.1.2. read noise which determines the lowest level of detectable signal. For consumer applications the read noise, full well capacity and resp onsitivity are usually of most interest, [Hol96], still even these are not rep orted by all manufacturers.

Holst, [Hol96], cautions that in order to obtain the transformation curve, most often the exp o- sure is varied (illumination times integration time), during these exp eriments the source should not b e changed (the same light bulb should b e used for example, but neutral density lters could b e used to change the source intensity). It is imp ortant not to compare blindly average resp onsitivity for di erent devices since the sp ectral quantum eciency di er among devices. Comparing arrays based on spectral responsitivity makes since if the il lumination conditions are similar and also similar to the calibration condition, otherwise responsitivity should be used only as a qualitative descriptor of array performance. \Sensitivity" parameter is mentioned in some camera data sheets. Unfortunately, there is no unique de nition of sensitivity, and the related units. It is generally understo o d that the sensitivity dep ends on the sp ectral quantum eciency and the noise level. At high signal levels the sensitivity is prop ortional to the exp osure (thus quantum eciency), and at low signal levels it is b ounded by the noise.Thus \sensitivity" if not clari ed, is also a qualitative parameter, not a reliable ground for comparing di erent cameras. An evaluation of a camera should be done with an application in mind.

3.2.3 Charge Transfer Eciency of the CCD

A basic limitation of the p erformance of the CCD is the eciency with which a charge packet can b e transferred from one p otential well to the next (with minimal addition or loss of charges). It is necessary that any charge packet passes through the CCD structure in a time p erio d so short that the amount of additional charge picked on the way is minimal. (This limits the size of the CCD registers, and the clo ck frequencies). On the other hand, the limited time for transp ort and the trapping of charges by \surface states" results in loss of charge during transp ort. A way of minimizing the loss due to trapping is to keep the p otential wells always semi-empty (fat zero is intro duced), so charges from passing packets will not b e trapp ed. Typical CTE for consumer applications CCD is ab out 0.9999. For high-p erformance contemp orary CCDs the loss could b e 7 electrons for 4096 transfers. A charge transfer eciency (CTE) of 0.9999995 is rep orted [JEW+^ 95]. The net eciency is CTEn^ , where n is the maximal numb er of transfers a packet may undergo. CTE dep ends on the charge packet size, it decreases for small packets (due to trapping), it also decreases for near maximal size packets (due to spills) [Hol96], page 81.

3.2.4 Minimal and Maximal Signals

Both, minimal and maximal signals are technology dep endent and varies with the CCD architecture. The minimal signal is the one which pro duces signal-to-noise ratio one [Hol96], or, equivalently, signal equal to the noise level in rms electrons. The minimal signal is some times rep orted as noise equivalent exp osure (NEE). The read noise puts a lower b ound on the minimal signal. NEE should not b e used for comparing arrays with di erent architecture [Hol96]. The minimal illumination implies the signal-to-noise ratio one. The de nition of minimal signal also varies with the author. The maximal signal is the one which lls up the p otential wells (saturates them). It is called saturation equivalent exp osure (SEE). The size of the well is directly related to the size of the sel and the CCD architecture (back illuminated CCDs have smaller wells). In [PH96], for the Photometrics digital camera it is rep orted that the full well capacity is 800 times the pixel area in micrometers. In the presence of anti-blo oming, the \white-clip" level is taken as the maximum

signal. In case of minimal dark current

S E E = Vmax Rav e

The saturation level dep ends on the CCD technology (material prop erties and pixel size), it is indep endent of the noise. The maximal signal for the camera corresp onds to full well

Vmax =

G 1 Gq Nw ell C

where G and G 1 are the gains of the on-chip and o -chip ampli er, q is the electron charge, Nw ell is the full well capacity and C is the capacitance of the measuring no de in picofarads. The numb er in electrons is inversely prop ortional to the square of the f-numb er of the optics, thus f-number must be selected when reporting minimal or maximal signal. Usually the minimal and maximal signals are rep orted with di erent optic f-numb ers, 5.6 for the maximum, and 1.4 for the minimum, if not otherwise sp eci ed. For consumer/industrial cameras the integration time is selected to comply with the broadcasting standards (i.e. 1/60 for the EIA 170 broadcasting standard). Varying the integration time or iris the signal levels may b e changed. The minimal/maximal parameters dep end on the illumination, and the sp ectral resp onse of the detector. It is imp ortant that the source color temp erature is rep orted (most often an incandescent source, color temp erature 2856 K is used, but this do es not have to b e the source used by the manufacturers). The reported parameters minimal signal/maximal signal/sensitivity/il lumination are quantities which de nition varies with the author, thus, cameras cannot be compared blind ly based on these parameters.

3.2.5 Spatial Resolution

The resolution of an imaging sensor is its ability to resolve spatial variations in the incoming signal. It is the sensor's ability to discriminate b etween closely spaced p oints in the image. The spatial frequency of an image fo cused on the device and the modulation transfer function (MTF) of the output are standard ways of characterizing the resolution. Spatial resolution dep ends on the geometry of the sensor { numb er of pixels, their shap e, and organization of the pixels in the imaging array. The resolution is sp eci ed by measuring the highest spatial frequency which can b e distinguished by the device for a given contrast. One metho d employed for determining sp ecial resolution is the shrinking raster metho d [Mad96]. In this metho d, a pattern of contrast vertical bar is imaged on the CCD. The distance b etween the bars is gradually decreased till a human observer cannot distinguish the bars as separate entities in the image. The resp onse of the imaging system to changing spatial frequencies is characterized by its modulation transfer function: de ned as the resp onse of the system to a sinusoidally changing spatial frequency fsig of the input signal. The individual sels sample the optical signal. Let x denotes a sel center, Ppix horizontal pitch (i.e., the center to center horizontal distance b etween sels), and
x is the pixel width, then the geometric mo dulation transfer function is

M T FG = sin (

x Ppix

fsig fN

where fsig is the signal frequency, and fN is the Nyquist frequency [The95]. High M T FG may not always b e advantage though { this leads to aliasing e ect which is out of the scop e of our discussion. For treatment of the Mo dulation Transfer Function Theory for the video sensor see [Hol96], Chapters 9 and 10.

resolution (or gain) of the video sensor. It is directly related to the saturation level (full well- capacity) and the bit depth of the framegrabb er. The saturation level divided by 2 N^ (where N is the bit depth of ADC) gives the quantization step. The video sensor dynamic range is restricted by the quantization noise when the video sensor noise is dominated by the quantization noise. As it will b e discussed later, under uniform distribution mo del, the variance of the quantization noise equals the quantization step squared divided by 12. For N bit ADC, the maximum signal is 2 N times the quantization step, thus the video sensor dynamic range is 2 N^

p 12, so 8-bit ADC cannot have dynamic range greater than is 59dB. There are di erent tradeo s which relate spatial, temp oral, and intensity resolution, which we will not discuss further here. Again, as with other parameters, the dynamic range varies largely with conditions depending on the optics (f-number of the lens, iris setting), integration time, spectral characteristics of the source of il lumination, spectral response of the CCD array. To be conclusive, when reported, the conditions under which it was measures should be clear.

3.3 Framegrabb er Related Parameters and Noise

3.3.1 Geometric Distortions

There are two p ossible typ es of geometric discrepancies which could arise due to the frame grabb er: (i) the individual lines in the digitized image are not aligned prop erly, this is termed linejitter (it is a result of the failure of the frame grabb er to detect the HSYNC); and (ii) the framegrabb er undersamples or oversamples the \lo cked" line (the sampling frequency is di erent from the camera pixel clo ck frequency) and thus a scaling factor for conversion b etween pixels and sels is intro duced. This parameter, horizontal scale, is one of the ob jects of geometric calibration. A geometric distortion in the digital image which relates to VSYNC detection is that the top, up to 100, lines in the image have relatively higher jitter. A detailed study of di erent frame grabb er architectures and synchronization mechanisms, and related geometric and radiometric distortions is given in [Bey92]. In [Bey90] three di erent metho ds for detecting and accommo dating for linejitter are prop osed. Two of them require very sp ecialized equipment and measuring pro cedures. The most appropriate metho d for general computer vision applications is the plumb line method [Bro71] in which an image of vertical strip es is taken, and the linejitter and radial distortions due to optics are estimated together. In [LT87] a metho d for estimating linejitter based on Fourier analysis is presented, it is appropriate for cases in which it is proven that the jitter is no more than a pixel.

3.3.2 Radiometric Distortions

An e ect in radiometric distortion from interlacing is the shift in gray level b etween o dd and even elds. Yet another source of radiometric distortion: in the pro cess of digitization, the framegrabb er has to b e able to restore the zero reference level from which to measure (this pro cess is called DC-restoration); failure to detect the reference level correctly results in shift in the gray values, and thus radiometric distortions. \The fall-o of the sample-and-hold mechanism used in many DC-restoration circuits" leads to uniform (for all images with this con guration) comp onent in the background and could b e easily removed [Bey92]. One of the most noticeable manifestations of distortions due to the camera/frame grabb er interface is a systematic error component in the background noise of the dark images as discussed

earlier due primary to the mismatch b etween the frequency of the camera pixel clo ck and the digitizer sampling frequency.

3.4 Discretization Related Noise and Distortions

3.4.1 Aliasing

The e ect of spatial discretization is a main ob jective in signal pro cessing. Severe distortions (Moire-e ect) o ccur when the sampling theorem^11 is violated. The sel spacing puts limitation on the highest frequency in the input b eyond which severe distortions o ccur, and the area of the CCD chip limits the lowest frequencies which can b e detected.

3.4.2 Quantization error

A typ e of radiometric uncertainty is captured by the quantization error. It results from the analog to digital conversion of the signal. There are di erent schemes for conversion. Usually, all gray values are considered equally probable, and the distribution of the quantization error is assumed to b e uniform over the interval [ 12 q ; 12 q ], where q is the analog to digital quantization unit. Under this mo del the variance of the quantization error is q 2 =12. The value of q dep ends on the dynamic range of the CCD camera and the bit depth of the digitizer, its magnitude is the voltage corresp onding to the least signi cant bit in the ADC scale. For ADC with N bits, q = Vmax = 2 N^ , and in terms of the array output

< nAD C >=

C

GG 1 q

q p 12

e^ rms:

If we want to pick up an ADC converter to match the dynamic range of the array, Vmax corresp onds to the full capacity of the well,

< nAD C >= Nw ell 2 N^

p 12

It is desirable that the quantization noise is less than the read noise, for this an ADC with large enough numb er of bits has to b e selected.

4 Examples of Sensor Parameters and Pattern Noise

4.1 Background and Pattern Noise in Dark Images

Manifestation of the background noise can b e observed in the dark images. A dark image is an image taken with no access of light to the video sensor. First the camera is warmed up, next images are taken with a tight, opaque cap on the lens. Figure 5 (page 17) shows a typical dark image for the con guration of black and white camera SONY XC-77RR and a frame grabb er DT1451. Sp eci cations from the data sheet of the camera: 493(V)x768(H) sels; pixel clo ck frequency 14 :318MHz, within 1% error. For the frame grabb er DT1451: e ective digital image size 480(V)x512(H) p els; sampling frequency about 10MHz. In image A, Figure 5 (page 17), the dark current random noise is visible. Observe internal luminance. Systematic comp onent in the background noise is prominent in image

(^11) The input signal can b e fully reconstructed from the samples if the input signal frequency, f , is at most half of the the sampling frequency.