An Introduction to Information Theory (Prof. Adrish Banerjee, IIT Kanpur): Lecture 03

An Introduction to Information Theory

An Introduction to Information Theory. Instructor: Prof. Adrish Banerjee, Department of Electrical Engineering, IIT Kanpur. Information Theory answers two fundamental questions: what is the maximum data rate at which we can transmit over a communication link, and what is the fundamental limit of data compression. In this course we will explore answers to these two questions. We will study some practical source compression algorithms. We will also study how to compute channel capacity of simple channels. (from nptel.ac.in)

Lecture 03 - Information Inequalities

In this lecture we state and prove Jensen's inequality, log sum inequality, data processing lemma and Fano's lemma. We also give some other conditions to check convexity of a function. Further we use these inequalities to prove some properties of entropy and relative entropy function.

Go to the Course Home or watch other lectures:

Lecture 01 - Introduction

Lecture 02 - Measure Information

Lecture 03 - Information Inequalities

Lecture 04 - Problem Solving Session I

Lecture 05 - Block to Variable Length Coding: Prefix Free Code

Lecture 06 - Block to Variable Length Coding: Bounds on Optimal Code Length

Lecture 07 - Block to Variable Length Coding: Huffman Coding

Lecture 08 - Variable to Block Length Coding

Lecture 09 - The Asymptotic Equipartition Property

Lecture 10 - Block to Block Coding of DMS (Discrete Memoryless Source)

Lecture 11 - Problem Solving Session II