An Introduction to Information Theory
An Introduction to Information Theory. Instructor: Prof. Adrish Banerjee, Department of Electrical Engineering, IIT Kanpur. Information Theory answers two fundamental questions: what is the maximum data rate at which we can transmit over a communication link, and what is the fundamental limit of data compression. In this course we will explore answers to these two questions. We will study some practical source compression algorithms. We will also study how to compute channel capacity of simple channels. (from nptel.ac.in)
| Lecture 02 - Measure Information |
In this lecture, we first quantify information using Shannon's measure of information. We define entropy, conditional entropy, relative entropy, mutual information. We describe IT inequality and use it to prove some simple properties of entropy, relative entropy. We also state the chain rule for entropy, mutual information and divergence.
Go to the Course Home or watch other lectures: