Information and Communication / System Theory Flashcards

Detailed practice vocabulary flashcards covering Information and Communications (entropy, coding, capacity) and System Theory (Fourier transforms, signal types, linear systems) based on the lecture transcript.

Average information content of a source with 512 equiprobable symbols

$$9\, \text{bits/symbol}$$

Entropy of throwing a perfect die

$$ld(6)\, \text{bits}$$

Condition for maximum mutual information

When $X$ and $Y$ are completely dependent

Efficiency of a code for 12 equiprobable symbols using 4 bits/symbol

$$ld(12) / 4$$

Dependency of syntactic information content

The number of symbols the source generates

Entropy of a binary source

Depends on the probabilities of the symbols

Condition for maximum source entropy

Equally probable symbols

Expression for source capacity ($C$)

$$C = \max I(x) = \max H(x) = ld(N)\, \text{bits/symbol}$$

Dependency of self-information ($I(x_i)$)

The probability of the symbol

Capacity of a binary source

$$1\, \text{bit/symbol}$$

Condition for minimal source entropy (N symbols)

Different probabilities for all symbols (per key 11C)

Capacity of the Croatian language alphabet

$$4.76\, \text{bits/symbol}$$

Property of tabular codes

Independence of the elements of the code word

Arithmetic coding

A method of coding by blocks of symbols

Non-instantaneous indistinguishable code

A reversible code

Expression for code efficiency ($E$)

$$E = H / C_{\text{koda}}$$

Lost information content in a channel with all equal matrix elements

$$H(x)$$

Capacity of a symmetric noisy channel

$$H(X) + H(Y|X)$$

Average information content of 256 equiprobable symbols

$$8\, \text{bits/symbol}$$

Mutual information $I(X;Y)$ boundary

At most equal to $H(X)$

Condition for zero mutual information

$X$ and $Y$ are completely independent

Entropy of a source with memory ($H_{\infty}(X)$)

Determined by the expression $H_{\infty}(X) = \lim_{n \rightarrow \infty} H(X_n | X_{n-1}, X_{n-2}, \dots)$

Information content of equiprobable symbols

Equal to the entropy of the source

Formula for average information content $I(X)$

$$I(X) = = -\sum p(x_i) ld\, p(x_i)\, \text{bits/symbol}$$

Average information content of 16 equiprobable symbols

$$4\, \text{bits/symbol}$$

Entropy of joint sources $X$ and $Y$ dependency

Depends on the entropy of both sources and their conditional entropy

Relation between information content and probability

Larger when the probability is smaller

Dependency of information source entropy

The probabilities of the symbols

Condition for perfect secrecy of message $X$ with cryptogram $Y$

$$H(X|Y) = H(X)$$

Substitution cipher outcome for MILENIJ with $Y = x + 13 \pmod{27}$

$$\text{ĆGAKBGZE}$$

Condition for maximum transferred information $I(X;Y)$

Maximal when $H(Y|X) = 0$

Dependency of language entropy

The redundancy of the language

Entropy of a continuous source $H(X)$

$H(X) = - \int p(x) ld\, p(x) dx$ given $\int p(x) dx = 1$

Binary source entropy dependency

Depends on the probability of a single symbol

Joint entropy formula

$$H(X,Y) = H(X) + H(Y|X)$$

Maximum condition for joint entropy $H(X,Y)$

When $X$ and $Y$ are completely independent

Probability condition for zero mutual information

$$p(x,y) = p(x) p(y)$$

Efficiency of a code for 32 equiprobable symbols using 6 bits/symbol

$$5/6$$

Hexadecimal representation of binary 10011111

$$9\text{F}$$

ASCII binary representation of the symbol '@'

$$1000000$$

Primary role of an information encoder

To reduce the required capacity of memory or channel

Characteristics of a reversible code

Code without loss of information

Gray code for number 7 with $m=4$

$$0101$$

Huffman coding result for a source with memory

Non-optimal

Basis of arithmetic coding

Coding by blocks of maximum length

Run-Length Coding (RLC) property

Appropriate for sources with memory

Facsimile message coding (Group G3) basis

Modified Huffman Code (MHC)

Code capacity ($C_{\text{koda}}$) expression

$$C_{\text{koda}} = \sum a_i p(x_i)$$

Dependency of code word lengths in an optimal code

The probabilities of the coded symbols

Optimal code for binary source $p(x_1)=0.001$ and $p(x_2)=0.999$

$$x_1 \rightarrow 0, x_2 \rightarrow 1$$

Public key cryptographic system basis

Substitution and permutation

Key length ($N_k$) for perfect protection of message length $N_x$

$$N_k \ge N_x$$

GSM protective coding basis

Scrambling with 3 LFSRs

Average info content of a LOTO number (1 to 39)

Depends on the values of the symbols

Substitution cipher decrypt of JOGPSNDJKB ($Y = x + 1 \pmod{27}$)

INFORMACIJA

MHC code for 200 dots on an A4 fax line

$$000011001001000101$$

Type of Fourier transform

Integral transformation

Mathematical property of Fourier transform

Linear transformation

Fourier transform of a real function $x(t)$

Complex function of frequency

Dirac comb signal $x(t) = \delta(t) + \sum \delta(t - nT)$

Periodic sequence of Dirac impulses

Fourier coefficient $A_0$ (period 4, amp 10, width 2)

$$5$$

Measured spectral value of component $A_5 = 5$

$$10$$

Spectral density of a sine signal

Imaginaria

Spectral density of a shifted Dirac function $\delta(t \pm \tau)$

Only the phase part changes

Spectral density of Dirac function $\delta(t)$

$$\Delta(f) = 1$$

Spectral density formula for rectangular pulse (width $B$)

$$X(f) = AB \frac{\sin(\pi f B)}{\pi f B}$$

Spectral density zeros for pulse width $T=2$

Function with zeros at $X(f) = \frac{A}{j2} E \delta(f-B) + \delta(f+B)$

Spectral density of DC signal $x(t) = -3\text{V}$

$$x(f) = -3\delta(f)$$

Fourier transform of complex function $x(t) = e^{j2\pi ft}$

Real

FT of sine function $x(t) = A \sin(2\pi f_0 t)$

$$X(f) = \frac{A}{2} [\delta(f - f_0) - \delta(f + f_0)]$$

Probability of independent symbols $x$ and $y$

$$p(x,y) = p(x) \times p(y)$$

Autocorrelation of a stationary process as $t \rightarrow \infty$

Tends to zero

Real part symmetry of FT for real function $x(t)$

$$X_r(-f) = X_r(f)$$

Imaginary part symmetry of FT for real function $x(t)$

$$X_i(-f) = -X_i(f)$$

Magnitude property of mirrored function FT

$$|Y(-f)| = |Y(f)|$$

Signal type for $x(t) = -a^{-t|t|} \sin(at)$

Aperiodic

Signal type defined by $x(t) = x(t + T)$

Periodic

Spectral density of a periodic signal

Discrete

Linear system output calculation

Determined by convolution of input and unit impulse response

Entropy (H) of a source with $N$ symbols

Average uncertainty of the symbols generated by the source

Unit of information per symbol

$$\text{bits/symbol}$$

Binary logarithm symbol in text

$ld$ (logaritmus dualis)

Source with memory property

Joint probability depends on previous states

Definition of a perfect die in entropy terms

A source with 6 equally probable outcomes

Syntactic content definition

Structural information related to symbol frequency and alphabet size

Reversible code benefit

Allows for perfect reconstruction of the original signal

Self-information ($I$) of a symbol with probability $1$

$$0\, \text{bits}$$

Relationship between Joint and Conditional Entropy

$$H(X,Y) = H(X) + H(Y|X)$$