stat101 1st part

1. Fundamental Concepts of Probability

Probability is the mathematical framework used to quantify uncertainty. It involves assigning a numerical value between 0 and 1 to the likelihood of an event occurring.

• Random Experiment: A procedure that can be repeated infinitely many times and has a well-defined set of possible outcomes.

• Sample Space ($S$): The collection of all possible outcomes of a random experiment. For example, in a coin toss, $S = {H, T}$.

• Event: A subset of the sample space ($A \subseteq S$). An event is said to occur if the resulting outcome of the experiment is an element of that subset.

2. Kolmogorov Axioms

Any probability measure must satisfy three fundamental axioms:

1. Non-negativity: For any event $A$, $P(A) \ge 0$.

2. Normalization: The probability of the entire sample space is 1, denoted as $P(S) = 1$.

3. Countable Additivity: For a sequence of mutually exclusive events $A<em>1, A</em>2, A<em>3, \dots$ (where $A</em>i \cap A<em>j = \emptyset$ for $i \neq j$): $P(\bigcup</em>{i=1}^{\infty} A<em>i) = \sum</em>{i=1}^{\infty} P(A_i)$

3. Rules of Probability

• Complement Rule: The probability that an event does not occur is $P(A^c) = 1 - P(A)$.

• Addition Rule (General): For any two events $A$ and $B$, the probability that at least one occurs is:
  $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

• Mutually Exclusive Events: If events cannot happen at the same time ($A \cap B = \emptyset$), then:
  $P(A \cup B) = P(A) + P(B)$

4. Conditional Probability and Independence

• Conditional Probability: The probability of event $A$ occurring given that event $B$ has already occurred is defined as:
  $P(A|B) = \frac{P(A \cap B)}{P(B)}$, where $P(B) > 0$

• Multiplication Rule: From the definition of conditional probability, we derive:
  $P(A \cap B) = P(A|B)P(B)$

• Statistical Independence: Two events $A$ and $B$ are independent if the occurrence of one does not affect the probability of the other. This is mathematically expressed as:

  • $P(A|B) = P(A)$

  • $P(A \cap B) = P(A)P(B)$

5. Bayes' Theorem

Bayes' Theorem describes the probability of an event based on prior knowledge of conditions that might be related to the event:
$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$

• Law of Total Probability: If $A<em>1, A</em>2, \dots, A<em>n$ is a partition of the sample space, then for any event $B$: $P(B) = \sum</em>{i=1}^{n} P(B|A<em>i)P(A</em>i)$

• Combining these, the expanded form of Bayes' Theorem is:
  $P(A<em>k|B) = \frac{P(B|A</em>k)P(A<em>k)}{\sum</em>{i=1}^{n} P(B|A<em>i)P(A</em>i)}$

1. Fundamental Concepts of Probability

Probability theory serves as the mathematical foundation for analyzing random phenomena. A random experiment consists of a process where outcomes are uncertain but the set of all possible results is known. This set is called the Sample Space ($S$), which can be finite, like the outcomes of rolling a six-sided die ($S = {1, 2, 3, 4, 5, 6}$), or infinite, such as the duration of a phone call. An Event ($A$) represents a specific outcome or a collection of outcomes deemed of interest. The probability function assigns a magnitude to these events, reflecting the relative likelihood of their occurrence in a long-run sequence of trials.

2. Kolmogorov Axioms

The mathematical consistency of probability is guaranteed by the three Kolmogorov Axioms. The non-negativity axiom ensures that probabilities are meaningful as non-negative measures ($P(A) \ge 0$). The normalization axiom establishes the scale of certainty, where the probability of the entire sample space ($S$) is 1. One immediate consequence of these axioms is that the probability of the null set ($\emptyset$), which represents an impossible event, is 0. Furthermore, the axiom of countable additivity allows for the calculation of probabilities for complex events formed by the union of disjoint (mutually exclusive) events. Specifically, if a set of events has no overlapping outcomes, the probability of at least one occurring is simply the sum of their individual likelihoods: $P(\bigcup<em>{i=1}^{\infty} A</em>i) = \sum<em>{i=1}^{\infty} P(A</em>i)$.

3. Rules of Probability

Several derived rules facilitate probability calculations. The Complement Rule states that for any event $A$, the probability that $A$ does not occur is $P(A^c) = 1 - P(A)$. This is particularly useful when the event "at least one" is simpler to calculate via its opposite, "none." For non-disjoint events, the General Addition Rule accounts for overlapping outcomes by stating $P(A \cup B) = P(A) + P(B) - P(A \cap B)$. This subtraction is necessary because the intersection ($A \cap B$) is counted twice—once within $P(A)$ and once within $P(B)$. If the events are mutually exclusive, the intersection is empty ($P(A \cap B) = 0$), simplifying the formula to a direct sum.

4. Conditional Probability and Independence

Conditional probability assesses how the likelihood of an event $A$ changes when we obtain new information about an event $B$. This updated probability is defined as $P(A|B) = \frac{P(A \cap B)}{P(B)}$, effectively restricting the original sample space to the outcomes contained within $B$. A crucial derivative is the Multiplication Rule, $P(A \cap B) = P(A|B)P(B)$, which calculates the joint probability of both events occurring sequentially. Statistical Independence occurs when knowing that $B$ has happened provides no extra information about the probability of $A$, such that $P(A|B) = P(A)$. For independent events, the joint probability is simply the product of their marginal probabilities: $P(A \cap B) = P(A)P(B)$.

5. Bayes' Theorem

Bayes' Theorem is a powerful tool for inductive reasoning, allowing one to reverse conditional probabilities to find $P(A|B)$ using $P(B|A)$. It is written as $P(A|B) = \frac{P(B|A)P(A)}{P(B)}$, where $P(A)$ is the prior probability (initial belief) and $P(A|B)$ is the posterior probability (updated belief). The denominator, $P(B)$, acts as a normalizing constant and is often calculated using the Law of Total Probability. This law partitions the sample space into mutually exclusive and exhaustive causes $A<em>1, A</em>2, \dots, A_n$ and calculates