Prepare for your exams
Get points
Guidelines and tips
Sell on Docsity
Docsity AI

Prepare for your exams

Study with the several resources on Docsity

Earn points to download

Earn points by helping other students or get them with a premium plan

Guidelines and tips

Sell on Docsity

Docsity AI

Log in Sign up

Prepare for your exams

Study with the several resources on Docsity

Find documents

Prepare for your exams with the study notes shared by other students like you on Docsity

Search for your university

Find the specific documents for your university's exams

Docsity AINEW

Summarize your documents, ask them questions, convert them into quizzes and concept maps

Explore questions

Clear up your doubts by reading the answers to questions asked by your fellow students

Earn points to download

Earn points by helping other students or get them with a premium plan

Share documents

20 Points

For each uploaded document

Answer questions

5 Points

For each given answer (max 1 per day)

All the ways to get free points

Get points immediately

Choose a premium plan with all the points you need

Study Opportunities

Choose your next study program

Get in touch with the best universities in the world. Search through thousands of universities and official partners

Community

Ask the community

Ask the community for help and clear up your study doubts

Free resources

Our save-the-student-ebooks!

Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors

ECE 6605 Information Theory Homework: Entropy, Mutual Information, and Processing, Assignments of Electrical and Electronics Engineering

Georgia Institute of Technology - Main Campus Electrical and Electronics Engineering

The fall 2003 homework assignment for the information theory course offered by the school of electrical and computer engineering at georgia institute of technology. The assignment covers topics such as entropy, joint entropy, mutual information, and processing of random variables. Students are required to find the entropy of random variables x and y, conditional entropy, joint entropy, and mutual information. They are also asked to consider examples of mutual information for a fair coin flip and a 6-sided die, and to prove whether the mutual information between a binary random variable and two independent observations is zero if the mutual information between the random variable and each observation is zero. The assignment also includes a problem on the effect of averaging multiple noisy observations on reducing noise.

Typology: Assignments

Pre 2010

Uploaded on 08/05/2009

koofers-user-2t0 🇺🇸

8 documents

1 / 2

This page cannot be seen from the preview

Don't miss anything!

School of Electrical and Computer Engineering

Georgia Institute of Technology

ECE 6605 Information Theory

Fall 2003

Homework #1

Assigned: August 20, 2003

Due: September 3, 2003

1) Example of joint entropy (entries in the following table are P(X=x,Y=y) )

Y=1

Y=2

Y=3

X=A

1/4

1/8

0

X=B

1/4

1/8

Find

(a) H(X), H(Y )

(b) H(X|Y), H(Y|X)

(c) H(X,Y)

(d) H(Y)-H(Y|X)

(e) I(X;Y)

(f) Draw a Venn diagram for the quantities in (a) through (e).

2) Mutual information examples

(a) Consider a fair coin flip. What is the mutual information between the top

side and the bottom side of the coin?

(b) A 6-sided fair die is rolled. What is the mutual information between the top

side and the front face (the side most facing you)?

3) Consider the following simple system

This problem is applicable to the situation where you get two independent

(noisy) observations Y1 and Y2 of the random variable X (think of the boxes as

communication channels). This is a simplified version of what happens in

some multiple antenna communication system.

Let X, Y1 and Y2 be binary random variables. If I(X; Y1) = 0 and I(X; Y2) = 0,

X

Y

1

2

Discover Assignments of Electrical and Electronics Engineering Georgia Institute of Technology - Main Campus

Partial preview of the text

Download ECE 6605 Information Theory Homework: Entropy, Mutual Information, and Processing and more Assignments Electrical and Electronics Engineering in PDF only on Docsity!

School of Electrical and Computer Engineering Georgia Institute of Technology ECE 6605 Information Theory Fall 2003 Homework # Assigned: August 20, 2003 Due: September 3, 2003

Example of joint entropy (entries in the following table are P(X=x,Y=y) ) Y=1 Y=2 Y= X=A 1/4 1/8 0 X=B 1/4 1/4 1/ Find (a) H(X), H(Y ) (b) H(X|Y), H(Y|X) (c) H(X,Y) (d) H(Y)-H(Y|X) (e) I(X;Y) (f) Draw a Venn diagram for the quantities in (a) through (e).
Mutual information examples (a) Consider a fair coin flip. What is the mutual information between the top side and the bottom side of the coin? (b) A 6-sided fair die is rolled. What is the mutual information between the top side and the front face (the side most facing you)?
Consider the following simple system This problem is applicable to the situation where you get two independent (noisy) observations Y 1 and Y 2 of the random variable X (think of the boxes as communication channels). This is a simplified version of what happens in some multiple antenna communication system. Let X, Y 1 and Y 2 be binary random variables. If I(X; Y 1 ) = 0 and I(X; Y 2 ) = 0,

X

Y

Y 1

2

does it follow that I(X; Y 1 ; Y 2 ) = 0? a) Yes or no? b) Prove or provide a counterexample.

Processing on a random variable. In many signal processing or communications applications an observation X is further processed by some operation g(X), e.g. filtering. Show that H(X) ≥ H(g(X)). Under what conditions on the function g() are they equal? Hints: 1) first think about what processing can do to the possible values of X and the resulting values g(X) and convince yourself this makes intuitive sense. 2) Form the joint entropy H(X,g(X)) and express it two different ways using the chain rule.
Multiple observations-part 2. In the figure of problem 3 assume that X is a random variable and

Y 1 = X + N 1

Y 2 = X + N 2

where

N 1 and N 2 are independent random variables with equal variance

s

2

. The purpose of this problem is to see why multiple independent observations are better than a single observation. Show that a simple averaging receiver that performs the following operation

Y = ( Y 1 + Y 2 )/ 2 has the effect of reducing the noise when

compared with just a single noisy of X. X Y

Y 1

2

Channel (^) Receiver