Information & Entropy, Summaries of Computer Fundamentals

Download Information & Entropy and more Summaries Computer Fundamentals in PDF only on Docsity!

Comp 595 DM

Professor Wang

• Information Equation

p = probability of the event happening b = base (base 2 is mostly used in information theory) *unit of information is determined by base base 2 = bits base 3 = trits base 10 = Hartleys base e = nats

• Another Example

Balls in the bin

The information you will get by choosing a ball

from the bin are calculated as following.

I(red ball) = - log(4/9) = 1.1699 bits

I(yellow ball) = - log(2/9) = 2.1699 bits

I(green ball) = - log(3/9) = 1.58496 bits

• Then, what is Entropy?
  • Entropy is simply the average(expected) amount of the information from the event.
• Entropy Equation

n = number of different outcomes

• Let’s look at this example again…

Calculating the entropy…. In this example there are three outcomes possible when you choose the ball, it can be either red, yellow, or green. (n = 3) So the equation will be following. Entropy = - (4/9) log(4/9) + -(2/9) log(2/9)

• • (3/9) log(3/9) = 1. Therefore, you are expected to get 1. information each time you choose a ball from the bin

Clear things up.

• Does Entropy have range from 0 to 1?
  • No. However, the range is set based on the number of outcomes.
  • Equation for calculating the range of Entropy: 0 ≤ Entropy ≤ log(n), where n is number of outcomes
  • Entropy 0(minimum entropy) occurs when one of the probabilities is 1 and rest are 0’s
  • Entropy log(n)(maximum entropy) occurs when all the probabilities have equal values of 1/n.