ma 416 - lecture 3

what is the motivating problem for Monte Carlo simulation?

standard statistics (Z, T, KS, CvM, AD) follow known distributions or have critical value tables

if you invent your own custom statistic, there is no table to look up critical values or p-values from

define monte carlo simulation

a technique where you…

numerically draw many random samples from the null distribution,

compute your test statistic on each simulated sample

and use the resulting collection of values to build an empirical approximation of the true distribution of that statistic

…allowing estimation of critical values and p-values with no closed-form formula

what is a monte carlo simulation in plain english?

instead of doing the math to derive the theoretical distribution of a weird statistic, simulate it thousands of times on a computer and look at the resulting histogram

What inputs does the toy ⋆_stat require?

Only the sample mean x̄ and the claimed population mean μ₀.

What is the "ideal" value of ⋆_stat if H₀ is exactly true, and why?

4/3

If H₀ holds, x̄ ≈ μ₀, so the ratio ≈ 1, giving (1+3)/(1+2) = 4/3

Why is knowing the ideal value 4/3 not enough to make a decision?

you still need to know how far from 4/3 the statistic must fall before the difference is statistically rather than randomly different, which requires the distribution of  ⋆_stat

List the steps for conducting a Monte Carlo simulation, in order.

(1) Set a random seed.

(2) Create an empty vector for simulated statistics.

(3) Choose a large number of monte carlo simulations.

(4) Loop nmc times: draw a simulated sample from the null at the same n, compute the statistic, append it.

(5) Visualize with a histogram.

How do you choose nmc, and what is the tradeoff?

Arbitrarily large is the guiding principle (e.g. 100,000). More simulations give a smoother, more accurate approximation but take longer to run.

What must match between your simulated samples and your real data?

The sample size.

Example: mcS = rnorm(length(S), 21, 1.5) matches n, the claimed μ₀, and an assumed σ.

What does the histogram of simulated statistics show you?

What typical values of your statistic look like if H₀ really were true.

How do you extract critical values from a Monte Carlo simulation?

y taking quantiles (percentiles) of the vector of simulated statistic values.

For a two-sided test at level α, how is α split, and what is the R code?

 α/2 in each tail. starcrits = quantile(mcstarstats, c(alpha/2, 1-alpha/2))

Monte Carlo decision rule for a two-sided test?

If the observed statistic falls outside the critical range (below the lower or above the upper critical value), you have evidence to reject H₀.

Define a simple distribution test.

Testing against ONE fully specified distribution, with both the family and its exact parameters pinned down (e.g. N(μ = 21, σ = 1.5)).

Define a composite distribution test.

Testing against an entire FAMILY of distributions without caring about specific parameter values (e.g. "is this sample normal at all").

Which is generally easier to satisfy (fail to reject), simple or composite, and why?

Composite, because you only need to match any member of a whole family rather than one specific exact distribution.

Do t-tests, ANOVA, and least-squares regression require simple or composite normality?

Composite.

You do not need to nail down the exact mean/variance in advance, just confirm the data is normal enough in general.

In the composite CvM workflow, why and how is the sample standardized?

Because it is a composite test, sort the data then compute zS = (S − mean(S))/sd(S), placing it on a standard normal (mean 0, sd 1) scale for fair comparison.

Which tail holds the rejection region for CvM-type statistics, and why?

Entirely the RIGHT tail, because the statistic is built from squared differences so it can only be zero or positive. Large values indicate poor fit and small values indicate good fit, so there is no meaningful "too small" direction.

R code for a one-sided CvM critical value?

cvm_crit = quantile(mymccvm, 1-alpha), putting all of α in the single right tail.

CvM decision rule and the worked example result?

If CvM_stat > CvM_crit, reject H₀. Here 200.4893 > 0.1254684, so the HourlyWage sample does NOT come from a normal population.

Important caveat about normality tests

Different tests (KS, CvM, AD, or eyeballing a histogram) can disagree, and their relative accuracy depends on sample size and data structure. There is no single infallible test.

Historically, why did statisticians design statistics that follow known distributions?

Because known distributions (Z, T, χ², F) had usable tables, even when a better custom statistic (more power, more robustness, better at small n) might exist without any known distribution.

What constraint does Monte Carlo free you from?

The need for your test statistic to match a textbook distribution family.

It approximates the sampling distribution for any statistic you can dream up.

What does qnorm(A, lower.tail=FALSE) return?

The critical value with area A in the RIGHT tail.

What does qnorm(A, lower.tail=TRUE) return?

The critical value with area A in the LEFT tail.

What does pnorm(q, lower.tail=FALSE) return?

The area/probability ABOVE the value q (right tail).

 What does pnorm(q, lower.tail=TRUE) return?

The area/probability BELOW the value q (left tail).

What does rnorm(n, mu, sigma) do?

Generates a random sample of size n from a Normal(mu, sigma) population.

General q/p/r naming pattern in R?

q-functions give a critical value from a probability/area. p-functions give a probability/area from a value. r-functions generate random samples.

t-distribution R functions and parameter?

Back: qt(), pt(), rt(n, v), where v = degrees of freedom.

Chi-squared R functions and parameter?

qchisq(), pchisq(), rchisq(n, v), where v = degrees of freedom.

F-distribution R functions and parameters?

qf(), pf(), rf(n, v1, v2), where v1 = numerator df and v2 = denominator df.