Download Optical Flow: Understanding and Computing with Constraint Equations - Prof. Brian Potetz and more Study notes Electrical and Electronics Engineering in PDF only on Docsity!
<.=-,/.$>:%($?+@=59$89)A
Optical Flow
- Motion of brightness pattern in the image
- Ideally Optical flow = Motion field
Aperture Problem Aperture Problem
Optical Flow Constraint Equation
- Assume brightness of patch remains same in both images:
- Assume small motion: (First order Taylor expansion of E)
Optical Flow: Velocities
Displacement:
d : R 2 → R
E(d) = Edata(d) + λEsmooth(d)
Edata(d) =
x,y
C(Ilef t(x, y), Iright(x + d(x, y), y))
Esmooth =
x,y
φ(d(x + 1, y) − d(x, y))
x,y
φ(d(x, y + 1) − d(x, y))
φ(∆d)
∆!x w(L(#x), L(#x + ∆#x))w(R(#x), R(#x + ∆#x))d(L(#x + ∆#x), R(#x + ∆#x)) ∑ ∆!x w(L(#x), L(#x^ +^ ∆#x))w(R(#x), R(#x^ +^ ∆#x)) #x = (x, y)
ri = f ′ r^0 r 0 · z
vi =
∂ri
∂t
= f ′ (ro^ ·^ z)v^0 −^ (v^0 ·^ z)r^0 (r 0 · z)^2
δt → 0
≈
d : R
→ R
E(d) = Edata(d) + λEsmooth
Edata(d) =
x,y
C(Ilef t(x, y), Irig
Esmooth =
x,y
φ(d(x + 1, y) − d
x,y
φ(d(x, y + 1) − d(
φ(∆d)
∆!x
w(L(#x), L(#x + ∆#x))w(R(#x), R(#x + ∆#x))d(L
∆!x
w(L(#x), L(#x + ∆#x))w(R(#x), R
ri = f
r 0
r 0 · z
vi =
∂ri
∂t
= f
(ro · z)v 0 −
(r 0
δt → 0
Optical Flow Constraint Equation
Divide by and take the limit
Constraint Equation
ri = f
r 0
r 0 · z
vi =
∂ri
∂t
= f
(ro · z)v 0 − (v 0 · z)r 0
(r 0 · z)
δt → 0
1
Optical Flow Constraint Equation
Divide by and take the limit
Constraint Equation
NOTE: must lie on a straight line
We can compute using gradient operators!
But, (u,v) cannot be found uniquely with this constraint!
Optical Flow Constraint
- Intuitively, what does this constraint mean?
- The component of the flow in the gradient direction is
determined
- The component of the flow parallel to an edge is
unknown
Computing Optical Flow
- Formulate Error in Optical Flow Constraint:
- We need additional constraints!
- Smoothness Constraint (as in shape from shading and stereo):
Usually motion field varies smoothly in the image. So, penalize departure from smoothness:
- Find (u,v) at each image point that MINIMIZES:
weighting factor
Discrete Optical Flow Algorithm
Consider image pixel
- Departure from Smoothness Constraint:
- Error in Optical Flow constraint equation:
- We seek the set that minimize:
NOTE:
show up in more than one term
Discrete Optical Flow Algorithm
- Differentiating w.r.t and setting to zero:
Update Rule:
are averages of around pixel
Example
Reduce the Resolution!
image H image I
Gaussian pyramid of image H Gaussian pyramid of image I
image H image I u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine Optical Flow Estimation
image J image I
Gaussian pyramid of image H Gaussian pyramid of image I
image H image I
run iterative OF
run iterative OF
upsample
. . .
Coarse-to-fine Optical Flow Estimation
::
:B
• #-5@2@=59$.-C)D2$13$=)+,-./$E121)
- 2-5@2@=59$)D.92$)F$35-,/59$15G.
- 2-5@2@=59$13F./.3=.$5++/)5=C.2$-)$.3./GH$
• I)D./3$?JK.=-$L.=)G31@)3$#H2-.*
• #.G*.3-5@)
:&
#=59.OM3E5/153-$8.5-,/.$N/532F)/*$Q#M8NR
:S
#,+./+1P.92$Q?E./O#.G*.3-5@)3R
:T
:U
6/)J5J1912@=$I.-C)D2$F)/$0121)
• #.V3G$+5/5*.-./2$5==)/D13G$-)$-C.$2-5@2@=2$)F$
/.59$A)/9D$+/)J9.*
• 6/)E1D.2$35-,/59$A5H2$-)$D.=)+)2.$+/)J9.2$
13-)$2,J+/)J9.*
- 45H.2$/,9.W$=C513$/,9.W$=)3D1@)359$13D.+.3D.3=.
• X++/)5=C$)+@175@)3$A1-C$-.=C31Y,.2$F/)$
2-5@2@=59$13F./.3=.
:%
6/)J5J1912@=$5++/)5=C.2$5/.$,2.F,9$13$2.E./59$A5H2(
y = mx + b
b = mx − y
(x 0 , y 0 )
(x 1 , y 1 ) (m 0 , b 0 )
(m 1 , b 1 )
x sin φ − y cos φ = ρ
ρ = r sin(φ − θ)
r =
x^2 + y^2
tan θ = y/x
Eelastic(v) =
0
α(s) · |v′(s)|^2 ds
Estif f ness(v) =
0
β(s) · |v ′′ (s)| 2 ds
Eedge(v) = −
0
|∇I(x(s), y(s))| 2 ds
Euser (v) =
0
U ser(x(s), y(s))ds
y = mx + b
b = mx − y
(x 0 , y 0 ) (x 1 , y 1 )
(m 0 , b 0 )
(m 1 , b 1 )
x sin φ − y cos φ = ρ ρ = r sin(φ − θ)
r =
x^2 + y^2
tan θ = y/x
Eelastic(v) =
0
α(s) · |v ′ (s)| 2 ds
Estif f ness(v) =
0
β(s) · |v ′′ (s)| 2 ds
Eedge(v) = −
0
|∇I(x(s), y(s))| 2 ds
Euser (v) =
0
U ser(x(s), y(s))ds
y = mx + b
b = mx − y (x 0 , y 0 )
(x 1 , y 1 )
(m 0 , b 0 )
(m 1 , b 1 ) x sin φ − y cos φ = ρ
ρ = r sin(φ − θ)
r =
x^2 + y^2 tan θ = y/x
Eelastic(v) =
0
α(s) · |v ′ (s)| 2 ds
Estif f ness(v) =
0
β(s) · |v ′′ (s)| 2 ds
Eedge(v) = −
0
|∇I(x(s), y(s))|^2 ds
Euser (v) =
0
U ser(x(s), y(s))ds
NH+1=59$#35Z.$!3./G1.
• !952@=1-H
• #@[3.
• !DG.$6/)P1*1-H
• \2./$13-./5=@)
y = mx + b
b = mx − y
(x 0 , y 0 )
(x 1 , y 1 ) (m 0 , b 0 )
(m 1 , b 1 )
x sin φ − y cos φ = ρ
ρ = r sin(φ − θ)
r =
x^2 + y^2
tan θ = y/x
Eelastic(v) =
0
α(s) · |v′(s)|^2 ds
Estif f ness(v) =
0
β(s) · |v ′′ (s)| 2 ds
Eedge(v) = −
0
|∇I(x(s), y(s))| 2 ds
Euser (v) =
0
U ser(x(s), y(s))ds
Smoothness Constraint for SFS
- In nature, objects are cohesive, and typically have smooth
surfaces.
- Smoothness constraint relates surface normals of neighboring
surface points
Minimize
(penalize rapid changes in surface normals over the image)
Ereconstruction =
(I − R(p, q))
2
dxdy
(I(x, y) − R(p(x, y), q(x, y)))
2
dxdy
Eintegrability =
∫ ∫ (^
∂y
p −
∂x
q
dxdy
E = λREreconstruction + λI Eintegrability
Esmoothness =
p
2
x +^ p
2
y +^ q
2
x +^ q
2 y
dxdy
6/)J5J191-H$O$X$L.E1.A$)F$-C.$4521=
• <5/G.W$K)13-W$D12=/.-.$D12-/1J,@)
• I5/G
• ")3D1@)
• 45H.2$L,9.
• 85=-)/1713G$95/G.$+/)J5J191-H$D12-/1J,@)
BU
6)13-2$MbD$91Z.$-)$=)E./(
