Computer Simulation Module 10: Comparing Systems

Computer Simulation Module 10: Comparing Systems

Instructor Information

• Dave Goldsman, Ph.D.

• Professor at Stewart School of Industrial and Systems Engineering

Module Overview

• Previous Module: Simulation output analysis techniques, primarily batch means and independent replications for a single system.

• Current Module: Discussion on how to compare different systems to identify the best one.

Key Topics

1. Introduction

2. Confidence intervals (CIs) for the mean

3. CIs for the difference in two means

4. Paired CIs for the difference in two means

5. CIs for mean differences in simulations

6. Variance Reduction Techniques

   • Common random numbers

   • Antithetic random numbers

   • Control variates

7. Ranking and Selection Methods

8. Normal means selection problem

9. Single-stage procedure

10. Normal extensions

11. Bernoulli probability selection

12. Bernoulli extensions

13. Multinomial cell selection

14. Single-stage procedure + extensions

Introduction

• Simulation experiments typically analyze or compare a small number of systems, typically less than 200.

• Analysis methods depend on the type of comparison and data properties:

  • For one system, traditional confidence intervals (CIs) can be applied using normal or t-distributions.

  • For two systems, CIs can also be derived, potentially employing paired observations.

  • For more than two systems, ranking and selection techniques are utilized.

Confidence Intervals

• Various types of CIs are relevant:

  • CIs can estimate:

  • Means

  • Variances

  • Quantiles

  • Apply to both one-sample and two-sample cases as well as multiple sample cases.

  • Importance of classical CIs in both statistical and simulation environments; observations aren't always identically and independently distributed (i.i.d.) normal.

Summary of Module Introduction

• Overview provided on the module's goals and comparison of competing systems.

• Next discussion: confidence interval for a single mean from a normal distribution which sets the stage for subsequent topics.

Confidence Interval for the Mean

Lesson Overview

• Reviewed the issue of comparing simulated systems and focused on deriving classical confidence interval for mean of i.i.d. normal data.

One-Sample Case

• To obtain a two-sided $100(1 - \beta)$ CI for unknown mean $u$ of a normal distribution:

  • Assume independent, identically distributed (i.i.d.) normal data: $X<em>1, X</em>2, ext{…}, X_n$

  • Assume unknown variance is $au^2$.

  • Utilize the well-known t-distribution-based CI.

Calculating Sample Mean and Variance

• Sample mean: $\bar{X} = rac{1}{n} ext{sum}(X_i)$

• Sample variance: $S^2 = rac{1}{n-1} ext{sum}((X_i - \bar{X})^2)$

• The sample mean and variance are used in subsequent CI derivations.

Deriving Confidence Interval

• Let $v$ denote the $(1 - \beta/2)$ quantile of a t-distribution with $n-1$ degrees of freedom.

• We can establish the following CI:
  ar{X} - rac{v imes S}{ ext{sqrt}(n)} <
  u < ar{X} + rac{v imes S}{ ext{sqrt}(n)}

Summary

• Derived classical confidence interval for the mean of a single normal distribution with unknown variance.

• Next step will involve comparing two systems using two-sample confidence intervals for the difference in their means.

Confidence Intervals for the Difference in Two Means

Lesson Overview

• Reviewed CIs for the mean of a single normal distribution with unknown variance, leading to the current focused content on comparing two systems.

Two-Sample Case

• Consider two samples:

  • For independent samples, if $X<em>1, X</em>2, ext{…}, X<em>n$ are from a normal distribution and $Y</em>1, Y<em>2, ext{…}, Y</em>m$ are from another distribution, CI for difference between means can be computed.

• Methods:

  • Pooled CI when $au<em>1^2$ and $au</em>2^2$ are equal but unknown.

  • Approximate CI when $au<em>1^2$ and $au</em>2^2$ are unequal and unknown.

  • Paired CI when observations are paired.

Pooled CI Formula

• The CI for means is calculated as follows when variance is pooled:
  $\bar{X} - \bar{Y} ext{±} t<em>{α/2, n+m-2} S</em>p ext{sqrt}\bigg( rac{1}{n} + rac{1}{m} \bigg)$

• Where $S<em>p$ is pooled variance: $S</em>p = rac{(n - 1)S<em>{X}^{2} + (m - 1)S</em>{Y}^{2}}{n + m - 2}$

Approximate CI Method

• When using arbitrary unknown variances for both populations,
  $\bar{X} - \bar{Y} ± t_{α/2, df} ext{approx.}$

• DF is calculated using the formula,
  $df = rac{(S<em>{X}^{2}/n + S</em>{Y}^{2}/m)^{2}}{ rac{(S<em>{X}^{2}/n)^{2}}{n - 1} + rac{(S</em>{Y}^{2}/m)^{2}}{m - 1}}$

Example:

• Situation: Time taken by individuals to parallel park two different cars, modeled as follows:

  • Population data for car A and car B when respective sample means and variances are assessed.

Summary

• Considered multiple CIs for the difference between two normal means under the assumption of independence.

• Next discussion will be focused on scenarios of correlated samples and methods to assess those cases.

Paired CI for the Difference in Two Means

Lesson Overview

• Previous session covered independent sample CIs; this session focuses on cases where paired samples exist.

Paired CI Concept

• When observations are collected in pairs (e.g., twins), one observes the effect of variability within pairs.

• Technical assumptions include that all pairs are jointly normal, with potentially unequal variances in populations.

Paired Differences

• Define pair-wise differences as $D<em>i = X</em>i - Y_i$ for paired observations; leading to finding the CI for $\bar{D}$ (mean of the differences).

Estimator for the Mean of the Differences

• It reduces to estimating a mean based on the differences. We then need to calculate:

• $\bar{D} = rac{1}{n} ext{sum}(D<em>i)$ where $D</em>i$ captures the difference in observations from respective pairs.

CI for Paired Differences

• Can then derive CI from differences using familiar means and t-distributions:
  $\bar{D} ext{±} t<em>{α/2, n-1} rac{S</em>D}{ ext{sqrt}(n)}$

Contextual Example

• Similar example with time taken for an individual to park two cars highlights the effectiveness of utilizing paired t-tests.

Summary

• Discussed paired-t confidence interval for differences in means, particularly effective with correlated data.

• Next topic: applying confidence recursion methods to compare simulated systems.

CIs for Mean Differences in Simulations

Lesson Overview

• Prior content covered the paired-t CI derived standardly; upcoming shifts focus to implementation within simulation contexts.

Applying CIs in Simulations

• Important to analyze how simulations compare competing systems (e.g., different strategies).

• Focus on: assumptions that simulation outputs are normally distributed or can be approximated as such through averaging results of multiple independent simulations.

Confidence Interval Construction

• For given strategies 1 and 2, denote observed metrics as $Z<em>{1j}$ from strategy 1 and $Z</em>{2j}$ from strategy 2.

Basic CI for Difference

• A CI for the difference in means is similar in formulation as the two-sample case leading to useful results in understanding which strategy performs better.

Example Application

• Example with cost comparison following different airline re-start strategies. Need to pay special attention to randomization in conditions to ensure results are valid.

Summary

• The established CIs for mean differences of two simulated systems resemble earlier discussions linking back to CIs.

• Transition into variance reduction techniques (VRT) which will follow next.

Variance Reduction Techniques

Common Random Numbers

• Utilizing common random numbers isolates randomness in simulations, effectively comparing outputs based on shared random variables across competing systems.

Conceptual Approach

• Subjects alternative systems to identical experimental conditions, addressing variability and distinctions clearly, lowering variance of estimators.

Demonstration Example: Queueing Analysis

• Evaluating customer flow in scenarios with common random numbers shows marked variances in estimation.

• Resulting confidence intervals from analysis highlight clear efficiency gains through the use of shared random numbers.

Summary

• Introduced variance reduction through common random numbers—effectively enhances confidence interval tightness in comparative analyses.

• Preparation for introducing Antithetic Random Numbers in the following session.

Antithetic Random Numbers

Lesson Overview

• Transition from common random numbers’ positive correlations to exploring the method of inducing negative correlation among estimators using antithetic random numbers.

Mechanism of Antithetic Numbers

• Describe the efficient correlation structure of estimates so average outcomes achieve lower variance. The lowering of variance from the mean of two negatively correlated estimators is a central focus.

Example Context: Monte Carlo Integration

• Show practical implementation of antithetic random numbers with an integral calculation introduced to illustrate effectiveness.

Results from Application

• Display improvements achieved in computation toward the true answer when the antithetic approach is employed, emphasizing its utility.

Summary

• The use of antithetic random numbers allows reducing estimators' variance significantly, enhancing the analysis and providing a pivotal methodology for future applications.

• Proceed to explore the final VRT, the control variates method, in subsequent discussions.

Control Variates

Lesson Overview

• Comprehensive discussion on control variates synthesizes with regression methods improving estimator variance.

Concept

• For a given simulation output process (observed output), if an additional known statistic serves a controllable metric, it can refine estimators of means.

Implementation

• Define new estimator employing a known variable through regression analogy, minimizing variance efficiently.

Example Applications

• Applications extend to diverse fields like economics, logistics, and simulations for accurate predictive analytics.

Summary

• Concluded discussions on VRTs with effective implementation of control variates since it reduces uncertainties within estimators.

• Next steps transition into Ranking and Selection Methods, focusing on how to choose the best system.

Ranking and Selection Methods

Lesson Overview

• Transition into the ranking and selection (R&S) methods, centering on selecting the best or optimal system based on comparative analysis.

R&S Techniques Introduction

• Techniques range from simultaneous CIs to ANOVA, focusing on determining superior systems and their metrics such as means and variances.

Key Objectives

• Guaranteeing a reliable selection probability while providing explicit comparisons aids experimenters in deriving outcomes of interest effectively.

Summary

• Established fundamentals in R&S methods targeting systematic assessment to determine optimal processes, facilitating precise statistical guarantees in selections.

• Transition into selecting the best normal distribution in next sessions which will enhance understanding of comparisons.

Normal Means Selection

Lesson Overview

• Focus on a structured approach to identifying which normal distribution holds the maximal mean through established probability thresholds.

Independent vs Preference Zones

• Discuss preference zones where variance in detection probabilities are particularly relevant in making selections.

Note on Procedure Highlights:

• Mention single-stage procedures using known variances and more sophisticated methodologies such as two-stage methods and optimal selections for enhancing accuracy in prediction and analysis.

Summary

• Acquainted with process maps and methodologies to evaluate normal means effectively; case studies using Bechhofer's procedure will clarify usage.

Bernoulli Selection Problem

Lesson Overview

• Introduction to selecting the Bernoulli population with the largest success probability emphasizing diverse applications in simulations and trials.

Probability Requirement Contents

• Specific thresholds for probability along with generating methods will create structured evaluation paradigms to categorize failures and successes in given trials.

Summary

• Laid groundwork for Bernoulli selection methods, enhancing procedural methods to assess efficient selections in future applications.

• Next, delve into extensions which streamline such methodologies and further efficiency tricks.

Multinomial Cell Selection

Lesson Overview

• Conclude the course with discussions surrounding the multinomial selection problem focusing on practical applications and creating effective selection procedures.

Selection Procedures Overview

• Discuss methodologies developed for multinomial selections to ensure proper evaluative techniques across varying domains.

Summary

• Wrapped up comprehensive selections spanning normal distributions to multinomial problems; instilled procedural knowledge across simulation outputs enhancing overall analytical skills in the field.

Conclusion

• Final remarks and solid practical takeaways are in place, effectively concluding Module 10 on comparative systems. Goodbye, dear students!