**Course Description**
In this course we consider important problems of representing sources of information efficiently (data compression aka source coding), recovering information from noisy yet redundant data (statistical inference, decoding), and protecting information via carefully added redundancy (error correction coding). This course is an introduction to how data processing systems are designed and analyzed. In particular, we consider a slew of compression algorithms without and with loss, algorithms for (optimal) inference/decoding, and some simple error correction codes. In addition to various algorithms, we also introduce two important quantities associated with measuring information. The first quantity, called (Shannon) entropy, characterizes the fundamental limit on how compressible a data source is. The second quantity is called mutual information and quantifies the fundamental limit on inference or recovery of information. The aim of this course is to build a solid introductory foundation for topics in information theory, data processing, and coding.

Consider watching this documentary about Claude Shannon, one of the most under-celebrated American scientists/technologists!


My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.' -- Claude Elwood Shannon, (1971) Scientific American, Vol 225, p. 180.