ECE 7620

Information Theory and Generative Modeling

Course Description

Course information provided by the 2026-2027 Catalog.

A graduate-level introduction to information theory, data compression, and generative modeling. An introduction to information measures: entropy, mutual information, relative entropy, differential entropy, and their properties. Lossless compression and its connection to prediction and generative modeling. The Minimum Description Length (MDL) principle in model selection. Practical lossless compression using arithmetic coding. The rate-distortion theorem and its connection to lossy compression standards such as JPEG, mp3, and AAC as well as generative modeling techniques such as autoencoders and variational inference. The Nonlinear Transform Coding framework. Practical methods for lossy compression such as Trellis-Coded Quantization (TCQ) and entropy-constrained dithered quantization.

Prerequisites ECE 4110 or equivalent.

Enrollment Priority Enrollment intended for: students intending to undertake Ph.D.-level research in information theory, statistics, or machine learning.

Last 4 Terms Offered 2024FA

Learning Outcomes

• Demonstrate use of information measures including entropy, mutual information, relatively entropy, and their properties.
• Compute theoretical limits to compression for both lossless and lossy problems.
• Analyze the performance of lossless and lossy compression schemes, including comparing their performance against the theoretical limits.
• Design lossless and lossy compression algorithms for provided datasets that approach the theoretical limits.

View Enrollment Information