Start Here

WELCOME TO ECE159

Introduction to Information Theory and Data Processing.

This course covers the basic concepts in information theory and coding. In this course we consider three important problems: 1) how to represent a source of information efficiently? 2) how to infer information from corrupted signals and/or correlated yet imperfect observations, and 3) how to protect information against potential sources of distortion and corruption such as additive noise? Even more interestingly we ask how to quantify the reliability and efficiency of our schemes against fundamental limits such as Shannon entropy and mutual information.

These topics are of paramount importance in data science and communications engineering. The aim of this course is to build a solid (theoretical as well as intuitive) foundation for topics in information theory and data processing. In particular, we consider lossless compression of discrete sources (such as the compression of text and digital images) as well as lossy representation of continues variables (quantization). We use compare these schemes in terms of efficiency and accuracy against a fundamental limit known as Shannon entropy. Moving to inference, we also show how the expected reduction in entropy, i.e. mutual information, quantifies the Bayesian transform of prior to posterior and also provides a bound on the reliability of inference task. Finally, we look at some popular computationally efficient approximations to maximum likelihood and discuss simple forward error correction techniques to increase reliability.

Each week, you will be expected to log into the Learning Management System (Canvas) and participate in the weekly assignments: read the required text, watch lecture videos to enhance your learning, participate in discussions that probe your reaction to the topics considered, and hand in HWs to practice and demonstrate your knowledge. Your participation and engagement with the learning activities is crucial to succeeding in this course and will help prepare you for a final exam in Week 11. Lectures notes are available on Canvas. The students are strongly encouraged to read ahead as studies have proved this to be the most effective method for learning [source: Matthew Herbst, HILD 20R]

Now, please follow the steps below to continue your orientation to this course.

Step 1: Read the course syllabus and course schedule

The course syllabus will provide you with the course schedule, course objectives, explanations of assignments and assessments, grading policies, and instructor contact information. Please read it carefully. You should have a deep familiarity with the schedule and process of the course.

Step 2: Access your course materials

Textbooks:

- - Lecture notes are all posted online. The bulk of these notes were originally developed by late Prof. J. K. Wolf for his Introduction to Information Theory and Coding class.
  - For additional reading, I recommend:
    Information theory, inference, and learning algorithms
    David J.C. MacKay
    Cambridge Press, 2003

Lecture Videos:

I will record the lectures for your future use. The videos will be linked within the weekly module in the order you should watch them. You can also access all course videos in the course's Media Gallery. Please note that while the lectures are recorded for future reference, attending the lectures is still the most effective method of learning.

The recommended browser is Chrome; however, if you are using Safari, see the Technical Requirements page and follow the instructions to ensure access.

Zoom Video Conference:

We will be using ZOOM, a video conferencing tool to conduct office hours and virtual meetings.

OFFICE HOURS:

I will be holding regular office hours on TTH at 2:00-3:00pm.

To Join an Office Hour at the scheduled time, click the Zoom LTI PRO link in the left-hand navigation menu.

VIRTUAL MEETINGS

[I will present the material discussed in lecture notes on Tuesdays and Thursday @ 12:30-1:50pm.]

Please make sure to check the course calendar for any scheduled Zoom meetings. You can also access the Zoom scheduler by clicking Zoom LTI PRO in the left-hand navigation menu.

Step 3: Assess your readiness for remote learning

If you're new to Remote Learning, go through the Gear Up For Success activity. This activity is completely anonymous, and will provide tips and feedback to help you prepare for success.

Step 4: Read the Technical Requirements Page

The Navigating Canvas and Technical Requirements pages identify the browsers, operating systems, and plugins that work best with Canvas. If you are new to Canvas quickly review the Canvas Student Orientation materials.

Step 5: Read about academic integrity and guidelines for online interactions.

Online community and integrity are essential to a successful learning experience. Review the Community Guidelines & Academic Integrity page.

Step 6: Familiarize yourself with available academic support services

View the Student Support Section for more information on campus resources and community support.