Using Color to Separate Reflection Components, Study notes of Linear Algebra

Download Using Color to Separate Reflection Components and more Study notes Linear Algebra in PDF only on Docsity!

ROCHESTER UNIV NY DEPT OF COMPUTER SCIENCE S A SHAFER FE92 APR^84 TR-136^ N@@@14-82-K-@i UNCLSSIFIDi F/G^ 9/2^ N

EEERhh

EE

E~l".i

SI

li-i

....... _. / III= (^) -Ia+ +

HI,,/

MICROCOPY RESOLUTION TEST CHART NATIONAL BUREAU (^) OF STANDAROS-1963-A

%F

Sii

Sal^ °.t °

i'. ,.-.+.'e ..; .. .. ... ..- (^5)....... ..-.-.. *S. , .....,...'.-,.,-..****. .... ,.,. •. - JI -. f.. ... .... ,. ... ....- + ... .+... ... +... " .- ,, 5•. ..****.

,i,77.N.^ o..

Using Color^ to Separate

Reflection Components

Steven A. Shafer Computer Science Department University of^ Rochester Rochester, NY^14627 TR^136 April 1984

*The author's permanent address is:^ Computer^ Science^ Department Carnegie-Mellon University

Pittsburgh, PA^15213 Z

Accession For N DTICTIS (^) TABGRA&I Unnnounced JUstificatio

:.,4,,',-.:.

. -Distri

Availability Codes

- porpra oe^ D~Th~ON^ public STA~1Avail^ i:e CmDist^^ Specialand/or DCstaibutiO- Uivemited^ o L~.

*Q. -s *',* ;;-:.

D-l T*^ t..^ *._^ -^ -^ '-

.' SECURITY^ CLASSIFICATION^ OF^ THIS^ PAGE^ (When^ D.e^ Enterd)

REPORT DOCUM ENTATION PAGE BEFOREREAD COMPLETINGINSTRUCTIONS FORM

I. REPORT NUMBER 2.^ GOVT^ ACCESSION^ NO.^ 3.^ RECIPIENT'S^ CATALOG^ NUMBER TR 136 [A _ ,_-.

4. TITLE (and Subile) i.^ TYPE^ OF^ REPORT^ &^ PERIOD^ COVERED

Using Color to Separate Reflection^ Components^ technical^ report

6. PERFORMING ORG. **REPORT NUMBER

7. AUTHOR(a) 8.**^ CONTRACT^ OR^ GRANT NUMBER(s)

Steven A. Shafer N00014-82-K-

9. PERFORMING ORGANIZATION NAME AND^ ADDRESS^ I.^ PROGRAM^ ELEMENT. PROJECT.^ TASK Computer Science^ Department^ AREA^ AWORK^ UNIT^ NUMBERS

University of Rochester

Rochester, NY 14627 "_

II. CONTROI.ING OFFICE NAME AND ADDRESS 12. REPORT DATE

Defense Advanced Research Projects Agency April 2, 1984

1400 Wilson Blvd. 13.^ UMUER^ OF^ PAGES

Arlington, VA 22209 -

14. MONITORING AGENCY NAME 6 ADDRESS(it diltemrnt from CoamfllInd Office) IS. SECURITY^ CLASS.^ (of^ this^ report)

Office of Naval Research n

Information Systems • unclassifled_

Arlington, VA^^22217 Is.^ DECLASSIFICATION/DOWGRADINOG SCHEDU E

16. DISTRIBUTION STATEMENT^ (of^ thia^ Repori)

Distribution of this document is unlimited..

17. DISTRIBUTION STATEMENT (of the abstract entered in block 20, it different from Report)

IS. SUPPLEMENTARY NOTES

none

19. KEY WORDS (Conllnue on reverssi It U neceear end ldenOtly by block number)

20. ABSTRACT (Continue on reveres aide It neceeea.y and Identify by block number)

This paper presents an algorithm for analyzing a standard color image to

determine intrinsic images of the amount of interface ('specular") and body

("diffuse") reflection at each pixel. The interface reflection represents the

highlights from the original image, and the body reflection represents the

original image with highlights removed. Such intrinsic images are of interest

because the geometric properties of each type of reflection are simpler than

the geometric properties of intensity in a black-and-white image.

(over)

FORM143 ",OND O ,SO,,.T"-"

DD , 1473 EDITIONOFINOVSSISOBSOLETE unclassified SECURITY CLASSIFICATION OF THIS PAGE (Phon Data Entered)

0 -****.****. - .,.. , :.. .,.... ,. -.

-:-

Using Color to Separate Reflection Components

Steven A.^ Shafer*

Computer Science Department University of Rochester Rochester, New York 14627 2 April 1984

Abstract

) of This the paper amount presents of interface an algorithm ('pecularland for analyzing body a standard ("iffuse color I reflection image to at determine each pixel. intrinsic The interface images

reflection represents the highlights from the original image, and the body reflection represents the original image with highlights removed. Such intrinsic images are of interest because^ the^ geometric properties of each type of reflection are simpler than the geometric properties of intensity in a black- and-white image. The algorithm is based upon a physical model of reflection which states that two distinct types of reflection -- interface and body reflection -- occur and that each type can be decomposed into a relative spectral distribution and a geometric scale factor. This model is far more general than typical - models used in computer vision and computer graphics, and includes most such models as special cases. In addition, the model does not assume^ a^ point^ light^ source^ or^ uniform^ illumination distribution over the scene. The properties of spectral projection into color space are used to derive a new model of pixel-value color distribution, and this model is exploited in an algorithm to derive the intrinsic images. Suggestions are provided for extending the model to deal with diffuse illumination and for analyzing " the intrinsic images of^ reflection.^ 6LCd'l^ r,^ -^ c^ ,,^^ f!. -L II

(0) The author's permanent address is: Computer Science Department, Carnegie-Mellon University, Pittsburgh, Pennsylvania 15213. This research was sponsored by the Defense Advanced Research Projects Agency (DOD), monitored by the Office of Naval Research under Contract NOOO1 4-82-K-01 93. .-. -. The views and conclusions contained in this document are those of the author and should not be Interpreted as representing the official policies, either expressed or implied, of^ the^ Defense^ Advanced Research Projects Agency or the US Government. *.. ..... : -

Table of Contents

 - 1.1 Previous Work in Color Image Understanding 1. Introduction I 

• 2. Reflection and the Dichromatic Model
     • 2.1 Physical Properties of Reflection
       • 2.1.1 An Aside on Terminology
     • 2.2 The Dichromatic Reflection Model
     • 2.3 Previously Used Reflection Models
• 3. Pixel Values in Color Space
  • 3.1 Spectral Projection
    • 3.2 The Dichromatic Model in Color Space
  • 3.3 Analyzing Color Pixel Values - 3.3.1 Assumptions of the Dichromatic Analysis Algorithm
  • 3.4 Summary
• 4. Applying the Dichromatic Analysis Algorithm
  • 4.1 Extending the Model for Diffuse Illumination
  • 4.2 Analysis of the Intrinsic Reflection Images
• 5. Conclusions
     • 5.1 Acknowledgements
• 6. Bibliography

1. Introduction

When we look around us, the surfaces we see are typically glossy.^ They^ may seem^ to^ be^ very^ shiny, fairly matte, or anywhere in between, but virtually all^ the surfaces^ around^ us^ exhibit^ highlights^ to varying degrees. These highlights are most pronounced when the surface normal bisects the angle between the direction of illumination and the direction of view, making the position^ and^ intensity^ of highlights very sensitive to viewing geometry. This causes problems with many low.level computer vision methods such as segmentation (which typically assumes uniform or smoothly varying intensity across a surface) or stereo and motion analysis (which attempt to match images taken from different viewpoints).

Highlights are not the only source of^ intensity^ variation^ across^ a^ surface.^ Even^ with^ uniform illumination of a matte, uniformly colored surface, there will be smooth shading due to the angle of incidence of the incoming illumination relative to the surface normal. - -

It would be very useful to be able to separate the effects of shading from highlights. This might result in intrinsic images telling, at each pixel, the intensity of the shading and the intensity of the highlight at that point. This separation was first suggested by Barrow and Tenenbaum (1], and would effectively produce an image of the highlights, and an image with the highlights removed. These would be useful for analysis since each of these phenomena is more simply related to the angles of illumination and viewing than is their sum (which is measured in a black-and-white image). In addition, the relative insensitivity of shading to viewpoint would make the shading intrinsic image an ideal candidate for stereo or motion analysis.

It is frequently observed that highlights have a different color from the characteristic color of a surface (which is related to shading). In this paper, we show how a simple, rather general model of reflection, called the "Dichromatic Reflection Model", can be .usedto determine intrinsic images of the two types of reflection from a standard color image (i.e. red-green-blue separation).

The analysis uses the properties of spectral projection, the process whereby color pixel values are determined from the spectral power^ distribution^ (SPD)^ of^ incoming^ light.^ Combining^ the^ Dichromatic Reflection Model with spectral projection results in a new model of pixel value distribution in R-G-B color space. The model predicts that pixel^ values^ from^ pixels^ on^ a^ single^ surface^ will^ lie^ on^ a parallelogram in color space, and that the position of any pixel's color within that parallelogram yields the coefficients of the two types of reflection.

I~

2

A simple algorithm is presented which utilizes this model of pixel values to determine the desired intrinsic images. The model and the algorithm are then extended to deal with diffuse illumination and shadows. Additional work for the future includes^ implementation^ of^ the^ algorithm^ and^ verification^ of the model with real images.

For additional background information, the reader interested in radiometry in general is referred^ to [9], [12], and [22]; while appearance measurement (gloss and color) is discussed in (10], [14], [16], and [35].

1.1 Previous Work in Color Image Understanding

Color image understanding in^ the^ past^ has^ not^ been^ based^ on^ general^ models^ of^ reflection.^ Most^ of the work has been the application to three-dimensional color space of algorithms originally developed and used for analyzing monochrome images. This includes primarily edge detection [21] and clustering [5,18, 20, 23] (etc.). In such work, image regions or edges are^ identified^ by^ distances between pixel values in color space, without appeal to any model of color generation in^ the scene.

Color has also been used for object labelling based on known object colors or object colors measured in the image [19, 26,31,32,34, 36]. This approach, known as "spectral signature analysis" in remote sensing, uses the known reflection properties of materials of interest in a particular domain. It depends on^ having^ few^ different^ types of^ materials^ in^ the scene,^ and^ having^ prior^ knowledge^ of^ their spectral reflectance.

The properties of color space transformations have also been studied [17, 24], although no such transformations have sufficed to "solve" the image segmentation or labelling problems.

The only previous work in utilizing general properties of reflected color has been in the form of simple statements such as "far-away outdoor^ objects^ look^ bluish",^ "outdoor^ shadows^ are^ bluish", and "natural colors tend to be desaturated" [19, 31]. In the psychological field, Rubin and Richards proposed a method for using color to determine changes in material across^ an^ image^ [27];^ however, while the method is quite interesting, their assumptions appear to be restrictive.

The approach described in this paper is significant because it is applicable to many common types of materials, without prior^ knowledge^ about their^ colors,^ and^ because it^ is based^ on^ a^ very^ general model of reflection^ in the^ real^ world.

Section 2.3 presents a discussion of^ previous^ work^ in^ reflectance^ models^ for^ computer graphics^ and image understanding.

P. e Z e

medium. Because the medium's^ index of^ refraction^ differs^ from^ that^ of^ the^ air,^ some of^ the^ light^ will^ be reflected at the interface producing interface reflection as shown in figure 2-1. The direction of^ such reflection will be in the "perfect specular direction" relative^ to^ the local^ surface^ normal,^ i.e.^ reflected about the surface normal. Note that most materials are optically "rough", with local surface normals th.t differ from the macroscopic or reference surface normal. The local perfect specular direction is therefore somewhat different from the macroscopic perfect specular direction,^ so^ that^ the^ interface reflection will be somewhat scattered at the macroscopic level.

The amount of light reflected at the interface is governed by Fresnel's laws, which relate interface reflectance to the angle of incidence (relative to the local surface normal), the index of refraction of the material, and the polarization of the incoming illumination [15]. Interface reflectance is thus a function of wavelength of light, since index of refraction generally^ depends^ on^ wavelength,^ but^ the amount is typically constant to within a few percent across the visible spectrum; acrylic plastic, for example, has an index of refraction of 1.485 at 400 nm (the blue end of the visible spectrum) and 1.505 at 700 nm (the red end) [3, 16, 35], producing Fresnel reflection coefficients of^ 3.8%^ and^ 4.1%^ at the ends of the spectrum. Interface^ reflection^ is^ frequently^ assumed to be^ constant^ with^ respect^ to wavelength because of the small magnitude difference, and is thus said to have the same color (relative SPD) as the illuminant [7, 14, 16]. The effect of polarization is more severe, and interface reflection tends to be highly polarized, especially for large angles of incidence ("grazing angles") [1-5,35].

The light that penetrates through the interface passes through the medium, where it undergoes scattering from the colorant,^ and^ eventually^ is^ either^ transmitted^ through^ the^ material^ (if it^ is^ not opaque), absorbed ,y the colorant, or re.emitted through the same interface by which it entered, producing body reflection as shown^ in^ figure^ 2-1.^ The^ geometric^ distribution^ of^ this^ body^ reflection^ is sometimes assumed to be isotropic, i.e. independent of viewing direction, although some^ work^ is under way to produce^ improved^ models^ of^ scattering^ [7,^ 10].^ The^ color^ of^ the^ body^ reflection is generally different from that of the illumination, since interactions with colorant particles result in absorption with a probability dependent on wavelength. Body reflection is usually considered to be unpolarized.

Real reflection tends to be more complex than described above. For example, real body reflection -" is not isotropic (figure 2-2 shows curves of reflection as a function of viewing direction) [13], and - some materials exhibit several interfaces producing interface reflection (Figure 2-3) [30]. However, the explanation provided^ above^ is^ a^ very^ useful^ approximation.

- (^) .sw. -s fi -..v all'-1' 111;1it 0.1f)to to Figuro 2-2 -ojnelcto i-. tjo n.c;.a, tn t i (fom [1] Thisov- fij~r_--,Ti%,j C~rC3 f rflQ ic~ asa uncionof iewingango fr (^) vrio s m tkeial,.I 2 ardrci,-wO Uiato shl on ata lo itiuinofrfeto i nr-rd

~*~> \ ~.r

K~jK>K ~N}

Z \ i-..

Fige2-3 A 2-2nt LodyePrdcnTwReflectionsio Itrfel~ or~ (from13]) j

..... '., 7

the off-specular angle.^ s^.^ the^ angle^ between^ V^ and^ the^ direction^ of^ (macroscopic) perfect specular reflection J.--" We also use the standard symbol X to refer to the wavelength of light.

2.2 The Dichromatic Reflection Model

We now propose a simple mathematical model of reflectance, based on the above discussion, called the Dichromatic Reflection Model.

The Dichromatic Reflection Model states:

L(X,i,e,g) = Li(X,i,e,g) + Lb(X,i,e,g) (2-1)

= m i (i, e, g) c, (X) + m b (i, e, g) Cb (,) (2-2)

This model represents two statements about reflected light, as expressed by the two parts of the equation:

1. Equation (2-1) says that the total radiance L^ of^ the^ reflected^ light^ is^ the^ sum^ of^ two independent parts: the radiance Li of the light reflected at the interface and the radiance Lb of the light reflected from the surface body.

2. Equation (2-2) says that each of these components of the light can be decomposed into two parts: - a. composition -- a relative spectral power distribution c, or Cb which depends only on wavelength but is independent of geometry, and b. magnitude .- a geometric scale factor mi or mb which depends only on geometry and is independent of wavelength. Intuitively, the Dichromatic Reflectance Model says^ that^ there^ are^ two^ independent^ reflection processes, and that each has a characteristic color whose magnitude, but not spectral distribution, varies with the directions of illumination and view.

In the remainder of this chapter we will address the scope of this model and its validity; in the next - chapter we will see how this model may be exploited to determine intrinsic images of mi and mb, the amount of interface and body reflection at each pixel.

The Dichromatic Reflection Model assumes the following: o The surface Is an opaque, inhomogeneous medium with one significant interface. e The surface is not optically active, i.e. it has no fluorescence or thin-film properties, and it is uniformly colored, i.e. it has a uniform distribution of the colorant. Reflection from the surface is Isotropic with respect to rotation about the surface normal.

.4.; ~ ** 4 4~* 4 ' ..... '.. ... ... * ..-

. * o.O"^.^.^.^ ^ q 4.. '.-°;.°. °*

.^7 7 7 -^ -^ -

• There is no inter-reflection among surfaces.
• There is a single light source, i.e. no diffuse ("ambient") illumination, and the relative spectral power distribution S(A) of the illumination is constant across the scene. The assumptions about^ the^ surface^ are^ typical^ for reflectance^ models^ and not^ too^ unrealistic.^ The assumption of no inter-reflection is also typical, but unfortunately not^ realistic^ at^ all.^ Finally,^ the assumption of no ambient light is not at^ all^ realistic,^ but^ will^ be^ relaxed^ (in^ fact,^ eliminated)^ in^ section 4.1 below.

Equally interesting is a list of assumptions not made by the model, which express the scope or generality of the model:

. Imaging geometry -- the model makes no assumption that orthography or perspective is being used. Either projection satisfies the model. o Planar surface -- the model applies equally to curved and planar surfaces. It also applies to textured surfaces, i.e. surfaces with macroscopic roughness (but see the note below about analyzing intrinsic images). o Specific reflectance model -. the model does not assume specific functions mi, ci, N or Cb; in particular, there is no specific geometric model of highlights, no assumption that the highlights have the same color as the illumination, and no assumption that the body roflection is isotropic. o Point light source -- the model applies equally well to a point light source, an extended light source, or a light source infinitely far away. o Uniform distribution of illumination .- the model does not assume that the amount of illumination is the same everywhere in the scene; only that the SPD is the same (see above). This is important, since real (especially extended) light sources produce nonuniform amounts of illumination in different areas of the scene. It must be pointed out that any complexity of the forms mentioned above will cause great difficulty in analyzing the resulting intrinsic images of m (^) i and mb. However, the Dichromatic Reflection Model will make it possible to compute these intrinsic images regardless of such complexity.

In spite of the apparent simplicity of the Dichromatic Reflectance Model, it suffers from some flaws. In practice, they ought to have only a minimal impact on the usefulness of the model.

• There is no obvious way to decide how to scale the magnitude functions m, and mb against the composition functions ci and cb. Therefore, the resulting intrinsic images may be only of relative, rather than absolute, reflection magnitudes. At least, it is reasonable to require that O < mi, m (^) b 5 1 for all i, e, and g. e Interface reflection exhibits an interdependence between wavelength and geometry, as expressed by Fresnel's equations. As stated (^) above, this is a small effect; the author's

. *ba .-

- n -. r '- "-

i.e. mb = COS i. These models of reflection are also examples of the Dichromatic Reflection Model,

with the exception of the use of Fresnel's equations to govern the color of the interface reflection

(usually a small effect, as noted above).

In computer vision, Shibata and Frei [29] used the following model to predict reflection in aerial

images:

L (i,e, g) = t 1 cosn^ s + t 2 cosm g + (1 - - t 2 ) COSi

Here, the first and last terms are similar to Phong's, and the middle term represents "backscattering"

of light in the direction of illumination. The constants t, t2, n, and m are parameters of the material

being viewed. Extended to color images, this model would also be an instantiation of the Dichromatic

Reflection Model, with the backscattering term added to mi.

V. \

3. Pixel Values in Color Space

This chapter begins with a discussion of spectral projection, the relationship between an SPD of

light and its color coordinates. When this process is applied to the D.chromaic Reflection Model, a

new model for pixel values in color space is the result. A simple algorithm is presented for exploitlIng this model to determine the intrinsic images of m (^) i and m (^) b.

3.1 Spectral Projection

Spectral projection is the process whereby pixel values are computed (^) from the spectral power distribution (^) (SPD) of the measured light. The process in a monochrome camera is quite simple, with the pixel value p being just a summation (^) of the amount of light at each wavelength X(,), weighted by the responsivity (^) of the camera to the various wavelengths, s(.):

p = J X(X) slA) d,

The interval of summation is determined (^) by s(,), which is non-zero over a bounded interval of wavelengths X.

In a color camera, (^) color filters are interposed between the incoming illumination and the camera Each filter has a transmittance function 'r(A), specifying the fraction of light transmitted at each wavelength; thus, spectral projection with a filter is specified by the above integral, with s(,) relplaced by i(A) s(A). Typically, three filters (^) (red, green, and blue) are used with transmittances t^ r' 1. and rb, resulting in a vector of three color values, C a [r, g, b]. If we (^) let r(A) be the reeponavity of the camera combined with the red filter, 7(A) = r(X) (^) s(k), (etc. for g and b), then the color value of any SPD X(X) is given by: xx 1 = [ x(A)i(A) (^) dh,

bx L X(X) b(A) dJ

Spectral projection is a linear transformation, (^) as shown in [281; in other words, C , **bY *** a +x " bCy where a and b are scalars and X() and Y(X) are SPD's. (^) To see this, consider first the red component rax + by; it is easily seen that-

reX+by = f[aX) + bY(,)] F(X)dA = a f X(,) T(X) dX + b f Y(A) T(A) d.

a N + b ry

With similar equations for green and blue, we have the complete result. The linearity property (^) of