Study Notes for Basics of Probability Theory

Department of Mathematics and Statistics

STAT 023 - Basics of Probability Theory

Instructor: Perlyn Mae Dilla - Tattao

Polytechnic University of the Philippines

www.pup.edu.ph

Introduction to Probability in Statistical Inference

• Foundation of inference: Framework for reasoning about uncertainty in conclusions.
• Quantifies confidence: Measures certainty in conclusions, providing a numerical scale from 0 (impossible) to 1 (certain).
• Guides decision-making: Evaluates hypotheses, estimates parameters, predicts outcomes.
• Key questions answered:
  • How reasonable is this outcome?
  • What’s the chance our conclusion is wrong?

Sample Space, Events, Operations on Events

Definition of Sample Space

• A sample space, denoted by S, is the set that contains all possible outcomes of a process or situation under consideration. Each element of S is called a sample point, representing one distinct outcome.
  • Mathematical Representation:
    $S = { \omega \;|\; \omega \text{ is all possible outcome} }$

Ways to Describe a Set

• Descriptive Method: A set described using a short verbal statement.
• Listing (Roster) Method: The set is written by listing elements inside braces, separated by commas.
  Example:
  $S = {HH, HT, TH, TT}$
• Set-Builder Notation: Uses a variable, braces, and a vertical bar | meaning “such that.”
  Example:
  {x \,|\, x \text{ is an integer, } 1 < x < 10}

Types of Sample Spaces

1. Finite Sample Space

   • Limited number of outcomes.
     Example: Flipping a coin twice,
     $S = {HH, HT, TH, TT}$

2. Countably Infinite Sample Space

   • Outcomes can be listed but go on forever.
     Example: Number of tosses until first head occurs,
     $S = {1, 2, 3, \ldots}$

3. Uncountably Infinite Sample Space

   • Outcomes form a continuum.
     Example: Life in years (t) of a certain electronic component,
     $S = {t \,|\, t \in \mathbb{R}^{+}}$

Definition of Event

• An event is a subset of a sample space. An event A is said to have occurred if the outcome of the random experiment is a member of A.
  • Examples:
  • For the sample space $S = {HH, HT, TH, TT}$, an event A that exactly one head occurs in flipping a coin twice can be defined as
    $A = {HT, TH}$
  • In the sample space of number of tosses until the first head,
    $S = {1, 2, 3, \ldots}$,
    the event that head occurs after 3 tosses is a specific member of the outcomes.
  • With respect to life in years, the event A that the electronic component fails before the end of the fifth year is defined as
    A = {t \,|\, 0 \leq t < 5}

Operations on Events

• Definitions of important set operations on events:
  • Complement of an event A: The subset of all elements of S that are not in A, denoted by A′.
  • Intersection of two events A and B: Denoted by A ∩ B, represents the event containing all elements common to both A and B.
  • Union of two events A and B: Denoted by A ∪ B, is the event containing all elements that belong to A, B, or both.
  • Mutually Exclusive Events: Two events A and B are mutually exclusive or disjoint if
    $A ∩ B = ∅$, indicating that they have no elements in common.

Example in Network Protocol Setting

• If a packet transmission has two possible outcomes (Success S and Failure F):
  • Sample Space when sending two packets in sequence:
    $S = {SS, SF, FS, FF}$
  • Event A that at least one packet is successfully transmitted:
    $A = {SS, SF, FS}$
  • Event B that both packets fail:
    $B = {FF}$
  • Analyzing set operations:
  1. A′
  2. A ∪ B
  3. A ∩ B
  • Determine two events that are mutually exclusive.

Counting Rules

Multiplication Rule

• Counting Rule #1: If an operation can be performed in n1 ways, and if for each of these ways a second operation can be performed in n2 ways, then the total number of ways to perform both operations together is:
  $n1 imes n2$.
• More generally, if there are k operations where the i-th operation can be performed in ni ways, the total is:
  $n1 imes n2 imes … imes nk$.

Example Applications of the Multiplication Rule

1. Example 1: Determine the number of sample points in the set of all possible selections of three items marked as defective (D) or non-defective (N).
2. Example 2: How many three-digit codes can be created using digits 0-9?
   • (a) Without restriction: $10 imes 10 imes 10 = 1000$
   • (b) Without repetition: Here, the counting method changes.
3. Example 3: With three objects (x, y, z), the total number of arrangements is observed by listing all possibilities:
   • Arrangements are:
   • $xyz, yxz, zxy, xzy, yzx, zyx$
   • The total number of arrangements is $3! = 6$.
4. In general, the arrangement of “n” distinct objects can be understood as $\mathbf{n!}$.

Permutations

• Definition: A permutation is an arrangement of all or part of a set of objects.
• Theorem:1
  • Number of permutations of n objects is n!.
  • The number of permutations of n distinct objects taken r at a time:
    P_{n,r} = rac{n!}{(n - r)!}.

Probability

Definition of Probability

• Probability: The measure of the likelihood that an event occurs as a result of a statistical experiment, quantified using probabilities ranging from 0 to 1.
• Definition of Probability of Event:
• The probability of an event A, denoted as P(A), is summarized as the sum of the weights of all sample points in A.
  Therefore,
  $0 ≤ P(A) ≤ 1, P(∅) = 0, and P(S) = 1.$
• If A1, A2, A3, … is a sequence of mutually exclusive events, then
  $P(A1 ∪ A2 ∪ A3 ∪ … ) = P(A1) + P(A2) + P(A3) + …$.

Probability Example

• A fair coin is tossed twice. What is the probability that at least one head occurs?
  • Sample space:
    $S = {HH, HT, TH, TT}$

Equally Likely Outcomes Rule

• Rule for equally likely outcomes: In an experiment with N possible outcomes, if exactly n outcomes correspond to event A, then
  $P(A) = \frac{n}{N}$.

Additive Rule Theorem

• For two events A and B:
  $P(A ∪ B) = P(A) + P(B) − P(A ∩ B).$
• If A and B are mutually exclusive:
  $P(A ∪ B) = P(A) + P(B).$

Example Use Cases of Additive Rule

• E.g., drawing cards from a standard 52-card deck to calculate probabilities involving diamond cards or aces.

Conditional Probability

• The probability that event B occurs given event A has occurred is denoted by
  $P(B | A)$.

Multiplication Rule for Events

• If events A and B can occur, then
  $P(A ∩ B) = P(A)P(B | A).$

Bayes' Rule

• Derived from the Law of Total Probability, applying partitions of probabilities for given events.
  • For partitioning events: if events B1, B2, …, Bk constitute a partition of the sample space S, then for any event A in S,
    $P(A) = \sum_{i=1}^{k} P(Bi) P(A | Bi)$.

Conclusion

• Probability theory provides frameworks to understand and quantify uncertainties in various fields through systematic use of sample spaces, events, counting rules, and probability concepts.