SSP - Probability, random processes, random signals

Part 1 of SSP curriculum

frequentist definiton of probability, also known as relative probability

the probability based on how often an event occurs

the three probability axioms

• for any event A, it holds taht 0<=Pr[A]<=1

• it holds that Pr[S]=1

• for any countable collection of M disjoint events it holds taht Pr[A1 U A2 U … U Am] = Pr[A]+Pr[A2]+…+Pr[m]

consequences of the probability axioms

• it holds that Pr[]=0

• it holds that Pr[Ac]=1-Pr[A]

• for any events A and B it holds that Pr[A U B]=Pr[A]+Pr[B]-Pr[A * B]

• if A ←B it holds that Pr[A]=<Pr[B]

• for any event A and event space {B1, B2, …, Bm} it holds that Pr[A]=sum(Pr[A * Bi]) = sum(Pr[ABi])

priori probability

probability og an avent A occurring without having made any observation (prior=before)

• Pr[A]

posteriori probability

new probability after having obtained more information about the situation (post=after)

• Pr[A|B]

Bayes’ theorem

if two events are independent, the following holds Pr[AB]=… and Pr[A|B]=.. and Pr[B|A]=…

=Pr[A]Pr[B]

=Pr[A]

=Pr[B]

random variables

processes that involve uncertainty and therefore have a random outcome

random variable versus a realization

• random variable is a set possbile values of X(s)

• x is a possible value of random variable X(s) = an observation

probability mass function (PMF) for discrete random variables

probability that a discrete random variable X takes on the value x

cumulative distribution funciton (CDF) for discrete random variables

cumulative probability of the discrete random variable X having a value lower than or equal to x

• obtained from PMF by summing all values of the PMF

probability mass function (PDF) for continuous random variables

probability that a continuous random variable X takes on the value x

cumulative distribution funciton (CDF) for continuous random variables

cumulative probability of the continuous random variable X having a value lower than or equal to x

• obtained from PMF by putting an interval all values of the PMF

cumulative distribtuion function (CDF) properties

• 0=<Px(x)=<1 for all x

• Px(-inf)=0 and similarly Px()+inf=1

• Pxx(x2)>=Px(x1) holds when x2>=x1

• holds that Px(x2)-Px(x1)=Pr[x1<X<=x2] for x2>x1

probability density function (PDF) properties

• it holds that px(x)>=0 for all x

• by definition Px(x)=integral px(u)du

• it holds that integralpx(x)dx=1

moments

quantities that summarizes random variables’ characteristics

sample moments

average over data samples

expectation operator orders

• first order - expected value = mean

• second order - variance (central moment = found by subtracting from the random variable their first order moment)

• mth order normalized central moments - determied as the central moment divided by the mth order of the standard deviation

average=sample mean, median and mode

• average - determined by summing over all values and dividng by the number of observations

• median - value in the middle of the ordered data set

• mode - observation with the highest number of occurences

joint random variables

multiple random variables that are all associated with an event

• joint CDF

• joint PDF

• joint PMF

marginalization

When one of the conditions X ≤ x and Y ≤ y is always satisfied, the cumulative distribution does not depend on that variable anymore and therefore that variable can be removed from the equation, leaving a distribution that now depends on one single random variable.

covariance

a measure of the relationship between two random variables

correlation

a noncentralized version of the covariance

two random variables are called orthogonal…

…if the correlation between them equals 0

correlation coefficient

the relationship between two random variables independent of the variances

central limit therem

after averaging multiple realizations of a random variable generated from the same (arbi trary) distribution, the distribution of the averaged random variables converges to a Gaussian distribution

multivariate

= multiple variables are involved, X=[X1, X2, X3,…, XN]^T

• multivariate joint CDF

• multivariate joint PDF

• multivariate joint PMF

A probability distribution depending on multiple random vectors can be regarded in…

…all aspects as a probability distribution depending on a single random vector (which is a concatenation of all different random variables)

statironary random process

if its statistical properties do not change over time

strict-sense stationary

When stationarity applies to the statistics of any order

Wide-sense stationarity (WSS)

weaker criterion than strict-sense stationarity. In stead of requiring that all N-th order statistics be constant, only the first- and second-order statistics (mean and variance) must be independent of the time shift

average signal power Ps

the average signal energy per sample and where a total of N samples are averaged over

If a random process is ergodic,…

the statistical properties of the entire random process can be inferred from any of its realizations and it has to be stationary

spectral factorization

• states that most random processes with a continuous power spectral density (PSD) can be generated as the output of a causal filter driven by white noise, the so-called innovation representation of the random process

= the determination of a minimum-phase system from its magnitude response or from its autocorrelation function