Sidan finns bara på engelska
Information theory
Course manager: Harald Nautsch
Suggested credits: 8 hp
Course literature
The course will be based on the book Elements of Information Theory by Thomas Cover & Joy Thomas. The most recent edition is the 2nd edition, with ISBN 0471241954, but it is also possible to use the 1st edition, with ISBN 0471062596.
Organisation
Lectures will be given by Harald and by the course participants.
Examination
In order to pass, each participant should give one lecture. Subjects will be assigned before the first lecture.
After each lecture a number of problems to solve will be given. The problems should be solved individually by each participant, but it’s ok to discuss the problems with each other. You should turn in your solutions no later than a week after they are given.
Lectures
| No | Subject | Chapter(s) |
|---|---|---|
| 1 | Introduction, entropy, mutual information, AEP | 1-3 |
| 2 | Entropy rates of stochastic processes, differential entropy | 4, 8 |
| 3 | Data compression | 5 |
| 4 | Channel capacity, channelcoding theorem I | 2.10, 7 |
| 5 | Channel capacity, channel coding theorem II | 2.10, 7 |
| 6 | Gaussian channel | 9 |
| 7 | Information theory and statistics | 11 |
| 8 | Maximum entropy | 12 |
| 9 | Network information theory I | 15 (channel parts) |
| 10 | Network information theory II | 15 (channel parts) |
| 11 | Rate-distortion theory | 10 |
| 12 | Network source coding, universal source coding | 13, 15.4, 15.5, 15.8, 15.9 |
Chapter numbers are given according to the second edition of the book. The first edition has a slightly different chapter structure (see below).
Course material
Extra course material, such as lecture slides, will be published here.
| Slides from lecture 1 | |
| Slides from lecture 2 | |
| Slides from lecture 3 | |
| Slides from lecture 4 | |
| Slides from lecture 5 | |
| Slides from lecture 6 | |
| Slides from lecture 7, part 1 | |
| Slides from lecture 7, part 2 | |
| Slides from lecture 7, part 3 | |
| Slides from lecture 8 | |
| Slides from lecture 9 | |
| Slides from lecture 10 | |
| Slides from lecture 12 |
Problems
Problems will be published here after each lecture. All book references refer to the second edition. In order to pass the course you should solve at least 75% of the total problems and at least 60% of the problems for each subject.
| Lecture(s) | Problems |
| 1 | 2.4, 2.5, 2.6, 2.12, 2.35, 3.13ab (NOTE: The table on page 69 is wrong!) |
| 2 | 4.2, 4.6, 4.7, 8.1, 8.4 |
| 3 | 5.4 (also calculate the expected code length for (c)), 5.6, 5.8, 5.12, 5.25, 5.30 |
| 4-5 | 7.4, 7.8, 7.13, 7.16, 7.19 |
| 6 | 9.3, 9.4 (assume Xi>=0), 9.6, 9.9 |
| 7 | 11.12, 11.14, 11.15, 11.18 |
| 8 | 12.1, 12.3, 12.4 |
| 9-10,12 | 15.2, 15.8, 15.13, 15.17, 15.18 |
| 11 | 10.2, 10.5, 10.14 |
Comparison between book editions
The chapters in the two editions of the book correspond to each other as follows
| 2nd ed. | 1st ed. |
|---|---|
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
| 5 | 5.1-5.9, 5.11-5.12 |
| 6 | 6 |
| 7 | 8 |
| 8 | 9 |
| 9 | 10 |
| 10 | 13 |
| 11 | 12.1-12.9, 12.11 |
| 12 | 11 |
| 13 | 5.10, 12.10 + new material on Lempel-Ziv coding |
| 14 | 7 |
| 15 | 14 |
| 16 | 15 |
| 17 | 16 |