Information theory, inference, and learning algorithms

Vorlesung mit Übung (4 SWS)

Course description

This course presents a unified treatment of information theory and statistical inference, highlighting their shared foundations and complementary perspectives. Emphasis is placed on modern developments and interdisciplinary connections, particularly with statistical physics and theoretical computer science. Core topics include entropy and information measures, data compression and coding, the minimum description length principle, probabilistic graphical models, and inference algorithms such as belief propagation. The course also explores deep links between computation and physics, including the relationship between computational hardness, inference limits, and phase transitions in disordered systems.

By the end of the course, students will be able to apply core concepts of information theory, such as entropy and mutual information, to problems in statistical inference and learning. They will formulate inference problems within an information-theoretic and Bayesian framework, analyze data compression and representation schemes in light of fundamental limits, and model complex systems using probabilistic graphical models and inference algorithms such as belief propagation. Students will also develop an understanding of how ideas from statistical physics—such as free energy, entropy–energy tradeoffs, and phase transitions—illuminate the limits of inference and the emergence of computational hardness in high-dimensional systems.

Parameters

Time: Tue, from 8:00 to 12:15 (weekly lectures), Thu 13:00 to 15:00 (biweekly excercises)
Location: Robert-Mayer-Str. 11-15, Magnus Hörsaal (lecture), SR 307 (Exercises)
Language: English
Evaluation: Final oral exam

Schedule

TBD