Intro to AI 2 final terms (Lecture Information Theory, Shannon)

Information Content

The amount of surprise associated with an event.

Entropy

The average information content or uncertainty in a random variable.

Noisy Channel

A communication path where messages may be distorted.

Redundancy

Extra bits added to help correct or detect errors in noisy transmission.

Compression

Reducing the size of data by using patterns and predictions.

Information Gain

Difference in entropy before and after learning something new.

Channel Capacity

Maximum rate at which information can be reliably transmitted.

Surprisal

Negative log probability of an event; higher when the event is unlikely.

Symbol

A basic unit of communication (e.g., character, word, signal).