Probability and Statistical Inference I

objective probability

the long-run frequency of the occurrence of an event (e.g., observing heads in coin flips)

(Frequentist methods/statistics are rooted in objective probability, which is independent of the observer and human belief)

subjective probability

one’s possibly informed belief in the rate of occurrence of an event

(Bayesian statistics are fundamentally based on subjective probability where probability is a rational belief, updated with evidence)

Frequentist/classical methods

methods that are based on the measure of relative frequencies are typically called this …

Bayesian statistics

a framework where probability measures an individual’s belief about an uncertain event, expressed as a prior, and systemically updates this belief with evidence through Bayes’ Theorem, to form a posterior belief

Population

the entire complete set of individuals/items you want to study

sample space

the set of all possible outcomes of a specific random experiment or variable (potential results of a probabilistic process)

sample space refers to the set of all experimental outcomes for a SINGLE EXPERIMENT applied to a GIVEN POPULATION, NOT for every experiment to exist

IMPORTANT NOTE: sample spaces define outcomes of an experiment, but DO NOT define relative frequencies of each of the listed events

Associative Law in Probability

the way you group together unions and intersections does not matter (the order of combining events/grouping does not change the probability)

Distributive Law in Probability

defines how AND distributes over OR (vice versa)

De Morgan’s Laws

Complement of a Union = Intersection of Complements

NOT (A U B) = (NOT A) AND (NOT B)

Complement of an Intersection = Union of Complements

NOT (A AND B) = (NOT A) OR (NOT B)

Disjoint sets

sets whose intersection is the empty set

Experiment

a process by which one or more observations are made

active experiment: you control the setting (e.g., experimental design)

passive experiment: you simply collect data

Probability model

assigns probabilities to each outcome in the sample space based on proportions in the population

simple events/elementary outcomes

the most basic, indivisible outcomes in the sample space (omega)

compound event

an event that consists of two or more simple events

probability measure on omega

a function P from subsets of the sample space omega to the real numbers that satisfies the following axioms:

1) P(omega) = 1

The probability that anything in the sample space happens is 1.

2) If A is a subset of the sample space omega, the probability of it occurring is between 0 and 1.

3) Additivity Axiom: If A1, A2, …, Ak are disjoint events, the probability of any of these events occurring is the sum of the event’s individual probabilities.

Additivity axiom

If A1, A2, …, Ak are disjoint events, the probability of any of these events occurring is the sum of the event’s individual probabilities.

Events A and B are said to be independent if any of the following conditions hold:

• P(A|B) = P(A)

• P(B|A) = P(B)

• P(A and B) = P(A) * P(B)

For three separate events A, B, and C in the sample space which have a non-zero probability of occurring, A and B are conditionally independent if …

P(A and B | C) = P(A|C) * P(B|C)

conditionally independent

If C has occurred, then learning about whether A happened tells us nothing new about whether B happened.

But outside of C, A and B may be dependent.