Session 9

Here are deep study notes for Session 9-1 and 9-2: ITT, LATE, Non-compliance, and Instrumental Variables, based on your uploaded slides.    

Session 9: Main Idea

This session answers a very important experiment question:

What if people do not actually follow the treatment assigned to them?

Example:

You send emails to 2,000 people.

But not everyone reads the email.

So there are two different questions:

1. Effect of sending the email

2. Effect of reading the email

These are not the same.

1. Treatment assigned vs treatment taken

In experiments, we often randomly assign treatment.

Let:

• T = 1: email was sent

• T = 0: email was not sent

• X = 1: person actually read the email

• X = 0: person did not read the email

• Y: outcome, such as whether they tried the product

The problem:

We control T, but we may not control X.

A company can send an email, but it cannot force someone to read it.

Analogy:

A teacher can give students extra practice problems, but cannot force every student to actually solve them.

2. ITT: Intention to Treat

ITT means the causal effect of assigning or offering the treatment.

In the email example:

ITT = effect of sending the email

It does not matter whether people read it or not. We compare everyone who was sent the email with everyone who was not sent the email.

Example from slides:

• 2,000 people sent email

• 100 tried the product

• 2,000 people not sent email

• 20 tried the product

Treatment group try-out rate:

100 / 2000 = 0.05

Control group try-out rate:

20 / 2000 = 0.01

ITT:

0.05 − 0.01 = 0.04

So sending the email increased try-out rate by 4 percentage points.

Teaching line:

ITT answers: “What happens if we send the email to the population?”

3. Why ITT may be smaller than the real treatment effect

Think of ITT as “offer effect”

ITT = effect of being offered/sent treatment, not necessarily using it.

Example: email campaign.

• 1,000 people are sent an email.

• Only 200 actually read it.

• The email can only influence people who read it.

• The remaining 800 dilute the average because they were “treated” in assignment, but not really exposed.

So when we compare all 1,000 emailed people vs all 1,000 non-emailed people, the effect gets averaged over:

• people who read the email and may change behavior

• people who ignored the email and likely do not change behavior

That is why ITT can look smaller.

Simple numbers

Suppose reading the email increases product trial by 20 percentage points.

But only 20% of emailed people read it.

Then the overall effect of sending email is:

20% reader share × 20 percentage point reading effect
= 4 percentage points

So:

• Effect of reading email = 20 percentage points

• Effect of sending email = 4 percentage points

That 4 percentage points is ITT.

Analogy

Imagine giving 100 students access to a free tutoring app.

Only 25 students actually use it.

If grades improve only for those 25 users, then the average effect across all 100 students will look small.

So:

Tutoring app usage may be powerful, but offering access looks weaker because many students did not use it.

ITT is smaller when many assigned people do not actually take the treatment.
It measures the effect of the offer, not the effect of actual usage.

4. LATE: Local Average Treatment Effect

LATE means the causal effect of taking the treatment for the people whose behavior was changed by the assignment.

In email case:

LATE = effect of reading the email for people who read it because they were sent it.

It is called local because it applies only to a specific group, not the whole population.

Teaching line:

ITT is population-level. LATE is effect on the responsive subgroup.

Example:

If sending email increases product trial by 4 percentage points, but only 20% of people actually read the email, then the effect of reading email may be much larger than 4 percentage points.

Simple formula intuition:

ITT = LATE × compliance rate

So:

LATE = ITT / compliance rate

If:

• ITT = 0.04

• Compliance rate = 0.20

Then:

LATE = 0.04 / 0.20 = 0.20

Meaning:

For people who actually read the email because they received it, reading the email increased try-out by 20 percentage points.

5. Four types of people in experiments

The slides define four groups:

1. Compliers

They follow assignment.

• If assigned treatment, they take it.

• If assigned control, they do not take it.

Example:

If sent email, they read it. If not sent email, they do not read it.

Analogy:

Students who do exactly what the teacher assigns.

2. Never takers

They never take treatment, no matter what.

Example:

Even if they receive the email, they never open it.

Analogy:

A person who gets a gym membership but never goes.

3. Always takers

They take treatment regardless of assignment.

Example:

They find the product information somewhere else and read it even if they were not sent the email.

Analogy:

A student who studies extra problems whether or not the teacher assigns them.

4. Defiers

They do the opposite of assignment.

• If assigned treatment, they avoid it.

• If assigned control, they take it.

This is rare and usually ignored.

Analogy:

A person who refuses to do something only because they were told to do it.

6. One-sided non-compliance

One-sided non-compliance means only the treatment group can fail to comply.

Example:

Email case:

• If you are sent the email, you may or may not read it.

• If you are not sent the email, you cannot read it.

So non-compliance happens only on the treatment side.

Another example:

Medicine prescription:

• If prescribed medicine, patient may or may not take it.

• If not prescribed medicine, patient usually cannot take it.

Teaching line:

In one-sided non-compliance, control people cannot access the treatment.

7. Two-sided non-compliance

Two-sided non-compliance means both treatment and control groups can fail to follow assignment.

Example:

Job counseling program:

• Treatment group is invited to join, but some do not join.

• Control group is not invited, but some still find a way to join.

Teaching line:

In two-sided non-compliance, treatment people may not take treatment, and control people may still access it.

8. ITT vs LATE

ITT

Question:

What is the effect of assigning the treatment?

Example:

What is the effect of sending emails?

Use when:

• You care about real-world policy rollout

• You want to know whether the campaign works as implemented

• You cannot force people to comply

Business meaning:

If I send this email campaign to 1 million users, how much total effect should I expect?

LATE

Question:

What is the effect of actually taking the treatment among those who respond to assignment?

Example:

What is the effect of reading the email among people who read it because we sent it?

Use when:

• You want to know whether the treatment itself works

• You want to improve compliance

• ITT is small but treatment may still be powerful

Business meaning:

The email content works, but the delivery/open rate may be weak.

9. Why not simply regress Y on X?

You might think:

Just compare people who read the email vs people who did not read it.

But this is biased.

Why?

Because people who read emails may be different from people who do not read emails.

Email readers may be:

• More interested

• More active customers

• Higher income

• More health-conscious

• More tech-savvy

These factors may also affect the outcome.

So if readers try the product more, it may not be because of the email. It may be because they were already more likely to try.

Teaching analogy:

Comparing email readers to non-readers is like comparing students who attend office hours to students who do not. Office hours may help, but office-hour students may already be more motivated.

That is selection bias.

10. Backdoor bias intuition

The slides show:

X → Y
X ← unobserved factors → Y

Where:

• X = reading email

• Y = trying product

• Unobserved factors = interest, motivation, income, health-consciousness

The problem is that X is correlated with the error term.

In regression terms:

Cov(X, error) ≠ 0

So OLS estimate of X on Y is biased.

Simple explanation:

We cannot tell whether Y changed because of X or because of hidden characteristics that caused both X and Y.

11. Instrumental Variable idea

To estimate LATE, we need a special variable called an instrumental variable, or IV.

The slides call this variable Z.

A good instrument must satisfy two conditions:

1. Relevance

Z must affect X.

Example:

Being sent an email affects whether someone reads the email.

So:

T affects X

2. Exogeneity / exclusion intuition

Z should affect Y only through X, not through another path.

Example:

Being randomly sent an email should affect product trial only because it makes people read the email.

Because T was randomized, it is not correlated with hidden motivation, income, or interest.

Teaching analogy:

An instrument is like a random push. It pushes some people into treatment, but does not directly change their outcome except through that treatment.

12. Why treatment assignment T is a good instrument

In the email experiment:

• T = whether email was sent

• X = whether email was read

• Y = whether product was tried

T is a good instrument because:

1. T is random.

   • It is not correlated with unobserved motivation or interest.

2. T changes X.

   • People sent emails are more likely to read emails.

3. T affects Y through X.

   • The email assignment affects trial mainly by causing email reading.

So we use T as an instrument for X.

Teaching line:

Random assignment is useful even when people do not comply, because it gives us clean variation in actual treatment.

13. Two-stage least squares intuition

Instrumental variable regression works in two stages.

First stage

Predict actual treatment using random assignment:

X = c + dZ + u

In email example:

Read email = c + d(Sent email) + u

This tells us how much sending email increases the probability of reading email.

The predicted value is:

X-hat

X-hat is the part of reading behavior caused by random assignment.

Second stage

Use predicted treatment to explain outcome:

Y = a + bX-hat + e

In email example:

Product trial = a + b(Predicted email reading) + e

Because X-hat comes only from random assignment, it is not contaminated by selection bias.

Teaching analogy:

Instead of comparing all email readers to non-readers, we compare the extra readers created by the random email assignment.

14. Time-Shift TV example

Session 9-2 applies this to Time-Shift TV.

TSTV lets households watch programming from one hour earlier.

Causal questions:

ITT question

Does offering TSTV increase TV watch time?

Here treatment is availability.

LATE question

Does actually using TSTV increase TV watch time?

Here treatment is actual usage.

This is like email:

• Offering TSTV = sending email

• Using TSTV = reading email

• TV watch time = product trial outcome

15. TSTV experiment setup

The experiment randomly assigned households:

• Treatment group: offered TSTV

• Control group: not offered TSTV

Unit of analysis:

Household

Outcome variables:

• Total TV watch time

• Live TV watch time

• Time-shift TV watch time

Because assignment was random, comparing treatment and control gives ITT.

Teaching line:

ITT tells us whether making TSTV available changes viewing behavior.

16. Non-compliance in TSTV

Some households did not behave according to assignment.

Examples from slides:

• Control household got access to TSTV.

• Treatment household had TSTV even before treatment.

• Treatment household had TSTV available but did not use it.

So there is non-compliance.

This means:

• ITT estimates effect of offering TSTV.

• LATE estimates effect of using TSTV for households induced to use it by the offer.

17. Why actual TSTV use is biased

If we compare households that used TSTV with those that did not, the estimate may be biased.

Why?

People who use TSTV may be different:

• They may love TV more.

• They may have busy jobs.

• They may have families.

• They may watch more shows generally.

These hidden factors affect both:

• Whether they use TSTV

• How much TV they watch

So:

used90 → total TV hours
used90 ← unobserved lifestyle → total TV hours

That is the backdoor problem again.

Teaching analogy:

People who use Netflix downloads may already be heavy media consumers. If they watch more content, we cannot say downloads caused all of it.

18. IV solution in TSTV

Use random TSTV offer as the instrument.

• Z = offered TSTV

• X = used TSTV

• Y = total TV watch time

First stage:

Used TSTV = c + d(Offered TSTV) + u

Second stage:

Total TV hours = a + b(Predicted TSTV use) + e

This gives LATE:

Effect of TSTV use among households whose usage was changed by being offered TSTV.

19. Why LATE can be larger than ITT

ITT averages over everyone assigned treatment:

• People who use TSTV

• People who do not use TSTV

If many households do not use TSTV, the average effect of offering it may be small.

LATE focuses only on households who actually used TSTV because of the offer.

So LATE is often larger.

Analogy:

Offering coupons to 1,000 people may increase purchases only slightly because many ignore the coupon. But among people who actually use the coupon, the purchase effect may be strong.

20. Managerial meaning

This session is very useful for business decisions.

Suppose:

• ITT = 0

• LATE > 0

Bad conclusion:

“Emails do not work. Stop sending them.”

Better conclusion:

“Reading emails works, but people are not reading them. Improve open rates, targeting, subject lines, or delivery channel.”

Examples:

• Send SMS instead of email

• Personalize subject lines

• Target users more likely to open

• Improve timing

• Use push notifications

Teaching line:

ITT tells us whether the current implementation works. LATE tells us whether the treatment itself has potential.

21. Very simple exam-style comparison

22. Final simple summary

Session 9 is about experiments where assignment and actual treatment are different. ITT measures the effect of offering or assigning treatment, like sending an email. LATE measures the effect of actually taking the treatment, like reading the email, for the people whose behavior changed because of the assignment. Since actual treatment choice is often biased, we cannot simply compare takers and non-takers. Instead, we use random assignment as an instrumental variable and estimate LATE using two-stage least squares.