Download Vector Quantization Lecture Notes (ECE 8873, Spring 2004) and more Study notes Electrical and Electronics Engineering in PDF only on Docsity!
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
ECE 8873
Data Compression & Modeling
Lecture 7:
Vector Quantization
School of Electrical and Computer Engineering
Georgia Institute of Technology
Spring, 2004
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Average or expected rate:
Notations & Terminology
)))]
[
ˆ)]
[
β α
α β
D
d E
d E
X
X
X
X
}
,
{
I i S
S
i^
∈
=
(^
x
x
x
x
α β
d
d
Distortion:Average or expected distortion:Encoder partition:
where
j
i
S
S
S
j
i
i^
≠
=
∩
⇒
if
disjoint
are
φ
A
S
i I ii
=
∪
∈,
}
) (
: {
I
i
S
i^
∈
′
=
x
x
α
))]
[^
x
l
E
Quantizer:
(^
l
Q
β α =
rate. the
determines
also
which
function"
length"
the is l
Encoder
Decoder
x
ˆ x
α
β
i
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Vector Quantization
Encoder
Decoder
x
ˆ x
α
β
i
Framing
Nearest-neighborsearch
codeword
index
Tablelookup
Synthesis
codebook
codebook
Grouping data intowaveform blocks orfor analysis toproduce repre-sentation vectors
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
-^
Gain from statistical dependence
-^
Gain from optimal space filling polytope
From Scalar to Vector - Advantages
Scalarquantizer
Vectorquantizer
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Non-uniform Quantizer
-^
Coding principle#1:
The best representation value in an interval isone that minimizes average distortion in thatinterval, i.e, the centroid
The best interval an input value is assigned to isthe one whose centroid is closest to the inputvalue.
-^
#1 relates to the decoder function, given the encoderresults.
-^
#2 relates to the encoder function, given the decoderdesign (the codebook).
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
•^
Use data rather than distribution to design the quantizer –i.e,, train the quantizer.
-^
Let
be a set of data from a
stationary source and
-^
The encoder
assigns every data point
to a region,
indexed
i
, which contains the subset
which can be considered a sample of the set
-^
The decoder uses
as the reproduction vector for
each region, incurring
as the distortion or
Empirical Design of Quantizer Codebook as the average distortion:
i
x^
i
i
d^
})) ( , (
{∑
Ω∈
β x
∑
∑ ∈^
Ω∈
I i^
i
i
d
L
x^
x
β
{^
I
i
S^ i
=^
x
x
α
L
j
j^
L
x
}
), ( {^
I i i^
∈
β
x
x
(^
I
I
Q
α α^
j
x
}
) (
and
: {^
I i
i^
∈ = Ω ∈ = Ω
x
x x
α
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Iterative (Hill Climbing) Procedure
x
x
ˆ
:
,
:
), , (^
→
→
=
I
I
Q
β α β α m C
) , ( ) , ( ) , ( β α β α β α
′
′
≥
′
≥
D
D
D
Given the set of reproduction codewords,
NearestNeighbor Partitioning
Centroid Computation
Convergence check
(^1) + m C
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
The Lloyd Algorithm
-^
Step 1: Get an initial codebook
-^
Step 2: Given the codebook
, find for each
training vector
the closest
codeword and its index
-^
Step 3: After the entire training set is exhausted,compute new centroid for each cell from the set
respectively;
-^
Step 4: Compute new average distortion. If it hasdropped by only an insignificant amount since lastiteration, stop; otherwise, repeat steps 2-4.
I i i j q j
i^
{ x
(^
j q i^
m C m C
{^
L
j
j^
L
x
x
j
j q i^
x for
codeword
closest the of
index ) (^
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Centroid with Itakura-Saito Distortion
1
log
2 |)
( |) (
1
| ,^ |
2 ,^2
2
2
2 2
−
−
=
^
∞
σ σ
ω π
ω
σ
σ
π π
ω^
x
j
IS
d e A X A X d
(^
)^
i
X
x
j
X
X^
IS
i^
i
i
i
d e A X A X d D
Ω −
−
=
=^
∫^
Ω∈
∞
−^
Ω∈
Ω∈
2 ,^2
2
2
2 2
log
2 |)
( | ) ( 1 |
,|
σ^ σ
ω π
ω
σ
σ
π π
ω
)
2
2
2
2 |)
( |) (
1
σ
ω π
ω
σ
π π
ω^
a R a^
p t
j^
d
e A
X
=
and
{^
I
i
X
X
X
i^
α
(^
)^
(^
2
,
2
,
2
2
2 |)
( | ) (
1
σ
σ
ω π
ω
σ
π π
ω^
a
R
a a
R a^
∫^
Ω∈
Ω∈
−^
Ω∈
=
=^
i
i
i
X^
p X
t
X^
p X t
j
X
d
e A
X
Same LPC equation but with (average) autocorrelations.
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Tree-Structured Vector Quantizers
Full VQ with Lloyd algorithmBinary-tree VQ
Each training token is compared to all the codewords in each iteration.Each training token is compared to two codewords in the respective regionduring each iteration.
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Binary-Tree VQ
00
10
00
10
Input vector
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Remarks on Tree-Structured VQ
-^
Overall distortion performance is suboptimal
-^
Much faster training due to division of training set(fewer tokens and fewer codewords to compare)
-^
Larger codebook space in training (almost twice)
-^
When coupled with code representation and thereceiver keeps a copy of the entire tree-structuredcodebook, it may offer some additional (minor)benefit in robust transmission and recovery of signal(rather than retrieving the codeword of finestresolution, get the intermediate one the receivedcode can reliably allow in the case of bit error orloss).
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Vector Quantizer with Product Code
-^
While going with longer vectors will get closer to theperformance bound, the complexity and/or (practical) latencybecome a problem.
-^
May need to compromise by using smaller vectors (not to grouptoo many together) or breaking the vectors (even in the case ofa complete set of parameters that define a model) into smallerchunks.
Framing
Nearest-neighborsearch
codeword
index
Tablelookup
Synthesis
codebooks
codebooks
Complexity comparison?
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Mean-Removed Vector Quantizers
-^
Often the mean of a vector may have different significance ineither statistics or in perception.
-^
Recall the structural component of information; mean can beconsidered a 1
st^
order structure. )
(^
r
x
r
x
(^
2 1
k
x
x
x^
L
(^
2 1
xk
x x^
L
x
k i
xi
k
1
μ
Code each component “separately.”
But
(^
2 1
rk
r r^
L
r^
with
1
k ∑= i
i
r^
r
k
μ
Therefore, can code the vector
in
k
-1 dimensions.
r
L
j
j
r^
L
r
Also consider the statistics of
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Alternative Structure in M-R VQ
x
Residualcodebook r
VQ
Samplemean
SQ
Mean
codebook
index
μ
x
Residualcodebook r
VQ
Samplemean
SQ
Mean
codebook
index
μˆ^ μ
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Gain-Shape Vector Quantizers
-^
Similar to mean, the root mean square value of the vector canbe considered a 1
st^
order structure, used to scale the vector.
(^
s
x
g
1 ( x
x
s^
−
g
(^
2 1
xk
x x^
L
x^
k i
xi
g
1
2
( x
Code each component separately.
is the shape vector
Distortion is
(^
2
2
2
s x x s x s x
t g g g g d
to maximize
to minimize
ˆ^ s
s
x
t ˆ
g ˆ
(^
s
x
t
g
Spring 2004
ECE 8873 B. H. Juang
Copyright 2004
Lecture 7, Slide #
Performance - PTVQ
1
2
3
4
5
6
7
8
9
RATE (bits/vector)
1.81.61.4 1.21.00.80.6 0.4Likelihood ratio distortion0.
Training performance
RATE (bits/vector)
Likelihood ratio distortion
1
2
3
4
5
6
7
8
9
1.81.61.41.21.0 0.80.60.40.
Testing performance
