Week 3: Predicting events

Week 3: Predicting Events

Probability Theory

• A probability is a number between 0 and 1 that tells us how likely a certain event is.

• Classical method of assigning probabilities:
  $\frac{\text{Number of outcomes in which an event occurs}}{\text{Number of possible outcomes}}$

• The empirical probability or relative frequency of occurrence method:
  $\frac{\text{Number of outcomes in which an event has occurred in the past}}{\text{Number of opportunities for an event to occur}}$

• Subjective probability method:

  • Using judgment and other subjective criteria to determine the probability an event occurs.

Outline

• Probability theory

• Law of addition

• Conditional probability

• Application of Events: Probability trees and Binomial probability (more on this next week)

Introduction to Probability Theory

• Probability theory is a prerequisite to statistics.

• Experiment: A random process that creates outcomes (e.g., the data-collection procedure).

• Sample space: The set of all possible outcomes.

• Event: A set of outcomes (can contain no outcome, single outcome, or multiple outcomes) of an experiment to which probability is assigned.

Multi-dimensional Data

• Example (Contingency Table or Cross-Tab):

  • Two random variables:

    1. Phone plan chosen by a client ($29, $49, $79)

    2. Day (weekday vs weekend) the client bought the plan

  • 6 potential outcomes: Every combination between phone plan chosen by a client and whether on weekday or weekend purchased.

• Sample space: {$29 on weekdays, $29 on weekends, $49 on weekdays, $49 on weekends, $79 on weekdays, $79 on weekends}.
  Example Events: {$29 on weekdays}, {customers bought $49 plan (at some day)}, {customers bought any plan on a weekday}, {$49 on weekdays, $79 on weekends}

Assigning Probabilities to Observed Outcomes

• Relative frequency: outcomes receive probability corresponding to their number of occurrences
  $P(\text{outcome}<em>i) = \frac{\text{number of occurrences of outcome}</em>i}{\text{total number of occurrences of all outcomes}}$

Law of Addition

Joint vs Marginal Probabilities

• Joint probability: denotes relative frequency when asking about all dimensions (e.g., what is the relative frequency that a customer bought a $49 plan on a weekday?)

• Marginal Probability: displays relative frequency when only asking about a single dimension. E.g., what is the relative frequency that a customer bought a $49 plan? (it’s single dimension, because here we do not care whether they bought it on a weekday or on weekends)

Venn Diagram: Visualization of Probabilities

• Venn Diagram (John Venn, 1880) shows logic relations across sets.

  • The external rectangle indicates the whole sample space.

  • The internal circle indicates some event $A$.

Joint Events

• The joint probability is represented by the overlapping part in the Venn Diagram.

• The symbol $\cap$ means ‘intersection’. $A \cap B$ can be thought of as $A$ & $B$.

Law of Addition

Joint Probability such as P(\text{weekdays and } $29) and P(\text{weekend and } $49). Joint probability describes the probability of outcomes associated with more than one random variable, e.g., Day and Price.

Marginal Probability such as $P(weekdays)$ and P($29). Marginal probability describes the probability of outcomes associated with only one random variable.

Some Notation: Event, Complement, Intersection of Events

• Mathematically, we can define an event by $A$, and define the complement of the event by $A'$ (pronounced as A prime) which means “not $A$”.

• Example:

  • Let $A$ denote the event “$29”, and $B$ denote “weekdays”.

  • $A'$ then indicates “not $29”, which in the previous example is “$49 or $79”.

  • $B'$ then indicates “Sat-Sun”.

Complement Rule of Probability

• Complement Rule of Probability
  $P(A)+P(A') = 1$

Law of Total Probability, Version 1

• Law of total probability, version 1
  $P(A \cap B) + P(A \cap B') = P(A)$

Union of Events

• $A \cup B$ indicates the event $A$ or $B$ happens. This is denoted by $A \cup B$, pronounced as the union of A and B, or A union B. So $P(A \cup B)$ indicates the probability that $A$ or $B$ is true, or that $A$ or $B$ occur.

General Rule of Addition

• What is the probability of the event that a plan is sold at a weekday or it is sold for $29 dollars?

  • Answer: $\frac{10}{800} + \frac{40}{800} + \frac{370}{800} =0.0125+0.05+0.4625$

  • Alternative: $\frac{50}{800} + \frac{380}{800} - \frac{10}{800} =0.0625+0.475-0.0125$

General rule of addition

$P(A \cup B) = P(A) + P(B) - P(A \cap B)$

It states that the probability that $A$ or $B$ occurs equals the probability that $A$ occurs plus the probability that $B$ occurs minus the probability that both $A$ and $B$ occur at the same time.

Mutually Exclusive Events

• If event $A$ occurs only if event $B$ does not occur (cannot occur at the same time), we say $A$ and $B$ are mutually exclusive (events).

  • Mutually exclusive events $P(A \cap B) = 0$. In the Venn diagram, these events do not intersect.

• Any event and its complement are mutually exclusive. Either “$A$ occurs” or “$A$ does not occur”. So $P(A \cap A') = 0$ always.

Collectively Exhaustive Events

• If the occurrence of events $A$ and $B$ covers the whole sample space, we say $A$ and $B$ are collectively exhaustive (events).

  • Collectively exhaustive events $P(A \cup B) = 1$

• Any event and its complement are collectively exhaustive. “$A$ occurs” and “$A$ does not occur” make up all possible outcomes. So $P(A \cup A') = 1$ always.

Conditional Probabilities and Independence

Conditional Probabilities

• In many cases, we are interested in conditional probabilities.

  • What is the probability of achieving growth in the next quarter conditional on the success of our advertisement campaign?

  • What’s the probability of passing this subject conditional on not attending lectures?

• $P(A|B)$ denotes the probability that event $A$ occurs, conditional on that $B$ occurs. The symbol $P(X = x | Y = y)$ denotes the probability of random variable $X$ taking value $x$, conditional on the random variable $Y$ taking value $y$.

• In our example of costs for phone plan, P($29 | \text{weekdays}) or P(\text{Price} = $29 | \text{Day} = \text{weekdays}) means the probability of a client choosing the $29 plan, conditional on she visits the store during weekdays.

Conditional Probability Example

• What is the probability of selling a $29 plan, conditional on a weekday?

  • Approach 1: $\frac{10}{380} = 0.026$

  • Approach 2: \frac{P($29 \cap Weekday)}{P(Weekday)} = \frac{0.0125}{0.475} = 0.026

Conditional Prob.: Formula

• Conditional probability can be computed as follows:

$P(A|B) = \frac{P(A \cap B)}{P(B)}; \quad P(B|A) = \frac{P(A \cap B)}{P(A)}$

So the prob of $A$ conditional $B$ equals the joint prob of both $A$ and $B$, divided (weighted) by the prob of $B$ (which is a marginal prob).

Bayes Rule

• From the conditional probability formula, we can write the general law of multiplication:
  $P(A \cap B) = P(A|B)P(B) = P(B|A)P(A)$

• We then rearrange this further to derive Bayes rule:
  $P(B|A) = \frac{P(A|B)P(B)}{P(A)}$

Law of total probability, version 2

$P(A|B) = \frac{P(A \cap B)}{P(B)}; \quad P(B|A) = \frac{P(A \cap B)}{P(A)}$

So joint prob equals conditional prob multiplied by marginal prob. This leads to Law of total probability, version 2

$P(A|B)P(B) + P(A|B')P(B') = P(A)$

Independent Events

• Events are said to be independent if the occurrence or non-occurrence of an event does not affect the occurrence or non-occurrence of another event(s).

  • For example, flipping a coin twice. We know that the probability of heads is 50%.

  • Now, regardless of what happens on the first toss, the probability of heads on the second toss is always 50%. That is, the outcome in the second toss is independent of the outcome of the first toss.

• Mathematically, independence implies: $P(Y|X) = P(Y)$

• Does the customer’s purchasing behavior depend on whether they buy the plan on a weekday or on weekends?

  • If customer behaves on the weekday the same way she behaves on weekends, we say purchasing behavior is independent of day. In other words, independence implies

P($29 | Weekdays) = P($29 | Not Weekdays) = P($29)

P($29 | Weekdays) = \frac{0.0125}{0.475} = 0.026

P($29 | Not Weekdays) = \frac{0.05}{0.525} = 0.095

P($29) = 0.0625

Since these probabilities are all different, they may not be independent events.

Independent Events: Formula

• If $A$ and $B$ are independent (events), whether or not $B$ occurs should not affect the probability that $A$ occurs; also, whether or not $A$ occurs should not affect the probability that $B$ occurs. This means

  • Independent events, version 1:
    $P(A|B) = P(A); \quad P(B|A) = P(B)$

  • Based on Bayes rule this also means

    • Independent events, version 2:
      $P(A \cap B) = P(A) \times P(B)$

Implications of Formulas

$P(A|B) = P(A); \quad P(B|A) = P(B) \implies P(A \cap B) = P(A) \times P(B) \implies \text{independent}$

$P(A|B) \neq P(A); \quad P(B|A) \neq P(B) \implies P(A \cap B) \neq P(A) \times P(B) \implies \text{not independent, so dependent}$

Example

Are the events “client choosing $29 plan” and “client purchasing during weekdays” independent?

We have P($29) = 0.0625, and $P(weekdays) = 0.475$,

So P($29) \times P(weekdays) = 0.0297, which is different from

P($29 \cap weekdays) = 0.0125.

So these two events may not be independent.

Open Questions

• Q: Why we are not 100% sure and say “may”?

• Q: If we collect the data to answer the question “Are choices of plan and day of purchase independent?”, is the data a sample or a population? If we choose other set of data, will the table look the same as the one above?

• To see if such a hypothesis holds true, we need to do statistical tests. We will study those tests later in the semester.

Binominal Experiments

• Suppose you toss a coin 3 times in a row and you are interested in how likely it is that you get exactly two heads.

• This is an instance of a binominal experiment.

• A binomial experiment assesses the number of a certain outcome from repeated independent trials.

• Each trial has two possible outcomes (e.g., heads or tail, success or failure,…)

• Research question is usually: What is the prob of $x$ successes out of $n$ trials? For example: what is the prob of detecting 2 defects out of 3 products? What is the prob of 1 heads out of 3 coin tosses?

Binominal Tree

• When two outcomes (e.g., success or failure, or binary outcomes) are independent, $P(A|B) = P(A)$. In such a case, we can draw a probability tree called a binomial tree.

• Suppose we have three products, each can be defect (D) with probability $p$ or functional (F) with probability $q = 1 − p$

Example

• What is the prob. of $x$ successes out of $n$ trials? For example: what is the prob of detecting 2 defects out of 3 products (2D,1F)?

  • Define the random variable $X$ as the number of successes (defects) out of $n$ trials (3 products). We are looking for $P(X = 2; n = 3, \text{success defect rate} = p)$.

    • Case 1: D,D,F; prob: $p \times p \times (1 − p) = p^2(1 − p)$

    • Case 2: D,F,D; prob: $p \times (1 − p) \times p = p^2(1 − p)$

    • Case 3: F,D,D; prob= $(1 − p) \times p \times p = p^2(1 − p)$

• Given outcomes across trials are independent, what is the probability of $x$ successes out of $n$ trials with probability of each success being $p$?

$P(X = x; n, p) = \text{number of cases or combinations} \times p^x(1 − p)^{n−x}$

Total prob. is the sum of all cases. Thus,

$P(2; 3, p) = 3p^2(1 − p)$

Binomial Distribution

• A random variable $X$ taking value in $0, 1, …, n$ is said to follow the binomial distribution, denoted by $X \sim Bin(n, p)$ if it describes the (random) number of successes out of $n$ trials in a binomial experiment (meaning that successes in different trials are independent).

• We can calculate the probability of $X$ successes from $n$ trials using the general formula:

$P(X = x; n, p) = \binom{n}{x} p^x (1 − p)^{n−x}$

• $p^x$: The probability of $x$ successes.

• $(1 − p)^{n−x}$: The probability of $n − x$ failures. So in total we have $n$ trials.

• The factor (combinatorial operator) $\binom{n}{x} = \frac{n!}{x!(n − x)!}$ computes the number of cases or combinations of choosing $x$ objects from the set of $n$ objects. Remember the factorial operator $m! = 1 \times 2 \times 3 \times … \times (m − 1) \times m$.

Binomial Distribution: Example

• Say the defect rate is 0.2, there are 3 trials, and the random variable $X$ is the number of defects out of 3 products (in mathematical notation we can write $X \sim Bin(3, 0.2)$

$P(X = x; n, p) = \binom{n}{x} p^x (1 − p)^{n−x} = P(X = 2; 3, 0.2) = \binom{3}{2} (0.2)^2 (1 − 0.2)^{(3−2)} = 0.096$

This calculation is straightforward in Excel using the BINOM.DIST function

Binominal Distribution

• Properties of binomial distribution:

  • Almost all distributions have expectation (i.e., mean) and variance (and thus standard deviation). We learn some other important distributions in weeks 4 and 5.

  • Every distribution is characterized by some parameters.

    1. The binomial distribution has two parameters, $n$ (the number of trials) and $p$ (the success probability or success rate).

    2. The mean (or expectation) and variance of $X \sim Bin(n, p)$ are given by

$\mu_X = E(X) = np$

$\sigma<em>X^2 = Var(X) = np(1 − p)$, so $\sigma</em>X = std(X) = \sqrt{np(1 − p)}$

Summary: Probability

• Summary for week 2:

  1. Probability theory

     • a) $P(A) + P(A') = 1$

     • b) $P(A \cap B) + P(A \cap B') = P(A)$

     • c) $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

     • d) $P(A|B) = \frac{P(A \cap B)}{P(B)}; \quad P(B|A) = \frac{P(A \cap B)}{P(A)}$

     • e) $P(A|B)P(B) + P(A|B')P(B') = P(A)$; $P(B|A)P(A) + P(B|A')P(A') = P(B)$

     • f) independence $\iff P(A|B) = P(A); \quad P(B|A) = P(B)$

     • g) independence $\iff P(A \cap B) = P(A) \times P(B)$

  2. Binomial distribution: prob of $x$ success in $n$ trials, with $p$ success rate for each trial. $X \sim Bin(n, p)$

$P(X = x; n, p) = \binom{n}{x} p^x (1 − p)^{n−x}$