# Information theory

Course manager: Harald Nautsch

Suggested credits: 8 hp

## Course literature

The course will be based on the book Elements of Information Theory by Thomas Cover & Joy Thomas. The most recent edition is the 2nd edition, with ISBN 0471241954, but it is also possible to use the 1st edition, with ISBN 0471062596.

## Organisation

Lectures will be given by Harald and by the course participants.

## Examination

In order to pass, each participant should give one lecture. Subjects will be assigned before the first lecture.

After each lecture a number of problems to solve will be given. The problems should be solved individually by each participant, but it’s ok to discuss the problems with each other. You should turn in your solutions no later than a week after they are given.

## Lectures

No Subject Chapter(s)
1 Introduction, entropy, mutual information, AEP 1-3
2 Entropy rates of stochastic processes, differential entropy 4, 8
3 Data compression 5
4 Channel capacity, channelcoding theorem I 2.10, 7
5 Channel capacity, channel coding theorem II 2.10, 7
6 Gaussian channel 9
7 Information theory and statistics 11
8 Maximum entropy 12
9 Network information theory I 15 (channel parts)
10 Network information theory II 15 (channel parts)
11 Rate-distortion theory 10
12 Network source coding, universal source coding 13, 15.4, 15.5, 15.8, 15.9

Chapter numbers are given according to the second edition of the book. The first edition has a slightly different chapter structure (see below).

## Course material

Extra course material, such as lecture slides, will be published here.

 Slides from lecture 1 PDF Slides from lecture 2 PDF Slides from lecture 3 PDF Slides from lecture 4 PDF Slides from lecture 5 PDF Slides from lecture 6 PDF Slides from lecture 7, part 1 PDF Slides from lecture 7, part 2 PDF Slides from lecture 7, part 3 PDF Slides from lecture 8 PDF Slides from lecture 9 PDF Slides from lecture 10 PDF Slides from lecture 12 PDF

## Problems

Problems will be published here after each lecture. All book references refer to the second edition. In order to pass the course you should solve at least 75% of the total problems and at least 60% of the problems for each subject.

 Lecture(s) Problems 1 2.4, 2.5, 2.6, 2.12, 2.35, 3.13ab (NOTE: The table on page 69 is wrong!) 2 4.2, 4.6, 4.7, 8.1, 8.4 3 5.4 (also calculate the expected code length for (c)), 5.6, 5.8, 5.12, 5.25, 5.30 4-5 7.4, 7.8, 7.13, 7.16, 7.19 6 9.3, 9.4 (assume Xi>=0), 9.6, 9.9 7 11.12, 11.14, 11.15, 11.18 8 12.1, 12.3, 12.4 9-10,12 15.2, 15.8, 15.13, 15.17, 15.18 11 10.2, 10.5, 10.14

## Comparison between book editions

The chapters in the two editions of the book correspond to each other as follows

2nd ed. 1st ed.
1 1
2 2
3 3
4 4
5 5.1-5.9, 5.11-5.12
6 6
7 8
8 9
9 10
10 13
11 12.1-12.9, 12.11
12 11
13 5.10, 12.10 + new material on Lempel-Ziv coding
14 7
15 14
16 15
17 16