MSDB 720 Lecture 1: Introduction to Statistical Modeling

Introduction to Data Analysis and Statistical Tasks

• Initial Steps in Data Analysis

  • Explore Data Characteristics: Understand the fundamental nature and properties of the dataset.

  • Data Screening: Actively screen for errors and perform necessary corrections.

  • Distribution Assessment: Determine if distribution patterns are normal or non-normal.

  • Data Transformation: Normalization or other transformations may be required before applying parametric methods.

  • Methodology Selection: If data is not normally distributed, non-parametric techniques may be necessary.

• Criteria for Choosing Statistical Methods

  • Analysis Objective: The specific goal or question the researcher aims to answer.

  • Observation Groups: Whether the groups are independent or dependent (paired).

  • Group Quantity: The total number of observation groups involved.

  • Data Type: The nature of the variables (e.g., continuous, categorical).

  • Data Distribution: The spread and shape of the data values.

• Primary Statistical Tasks

  • Hypothesis Testing: Using formal tests to evaluate assumptions about populations.

  • Estimation: Calculating the values of population parameters based on sample data.

  • Prediction: Estimating outcomes or future tendencies for specific subjects.

Hypothesis Testing and Estimation Frameworks

• Hypothesis Testing Methodologies

  • Parametric Tests to Regression:

    • The $2$-sample t-test and ANOVA evolve into Multiple Linear Regression.

  • Non-parametric Tests to Logistic Models:

    • Wilcoxon, Kruskal-Wallis, and Spearman tests transition into Proportional Odds Ordinal Logistic Models.

  • Survival Analysis:

    • The Log-rank test aligns with the Cox Proportional Hazards Model.

  • Model Advantages: Statistical models not only facilitate multiplicity adjustment but also allow for the shrinkage of estimates.

• Statistical Estimation

  • Model-Based Nature: Estimation is typically derived through statistical models rather than simple raw metrics.

  • Application Example: Evaluating the relative effect of increasing cholesterol from $200$ to $250\,mg/dl$ on the hazard of death while holding other risk factors constant.

  • Adjustment Significance: How other risk factors relate to the hazard determines the level of adjustment needed.

  • Partial vs. Marginal Effects: Statistical interest usually lies in adjusted (partial) effects rather than unadjusted (marginal or crude) effects.

• Integration of Testing, Estimation, and Prediction

  • Hypothesis Testing Tools: Formal tests and formal model comparisons, such as AIC (Akaike Information Criterion).

  • Estimation Tools: Point and interval estimation to determine a single predictor's effect or the outcome itself.

  • Prediction Goals: Generating an estimated outcome or tendency for a specific subject.

Predictive Modeling: Applications and Comparisons

• Diverse Fields of Application

  • Finance and Business: Financial performance, consumer purchasing behavior, and loan payback reliability.

  • Environment and Industry: Ecology and product life cycle analysis.

  • Social Justice: Studies on employment discrimination.

  • Medicine and Health:

    • Epidemiology and health services research.

    • Calculating the probability of diagnosis and the time course of a disease.

    • Validating summary indexes (e.g., BMI) or developing new summary indexes based on variable prediction of outcomes.

• Treatment and Economic Applications

  • Non-randomized Treatment Comparison: Using models to compare outcomes where random assignment was not possible.

  • Randomized Study Adjustments: If the model is non-linear, covariable adjustment is required to get correct relative effect estimates.

  • Handling Heterogeneity: Crude odds ratios are often biased toward $1.0$ in heterogeneous samples.

  • Risk Difference Estimation: Identifying the absolute treatment effect (e.g., the difference between two predicted probabilities).

  • Cost-Effectiveness Ratios: Calculated as: $\frac{\text{Incremental Cost}}{\text{Incremental Absolute Benefit}}$.

  • Critique of Averages: Most studies use average cost difference divided by average benefit, which may potentially apply to no single individual.

• Prediction vs. Classification

  • Classification Characteristics:

    • Response variable is binary.

    • Assumes the cost of misclassification is identical for every future observation.

    • Implies the ratio of the cost of a false positive to a false negative equals the analyst’s hidden ratio/rule.

    • Critiqued as sub-optimal and highly sensitive to outcome variable relative frequencies.

  • Prediction Advantages:

    • Utilizes full information rather than forcing a binary decision.

    • Predictions are made prior to and without specific knowledge of utility functions.

  • Hidden Problems in Classification:

    • Utility functions often depend on variables not collected/available until the decision point.

    • Assumes every subject and the analyst share the same utility function.

  • The Dichotomizing Motorist Example:

    • Scenario: A motorist is speeding and wonders if they will be caught.

    • Decision points: If the risk of an event P < 0.10 or P > 0.90, a decision might be clear even without utility knowledge.

    • Uncertainty: If $P = 0.6$ or the confidence interval is wide, the analyst might opt for more data collection rather than a forced classification, or make a tentative decision to be revisited later.

Planning and Limitations in Statistical Modeling

• Foundational Planning Checklist

  • Likelihood that the predictive model will actually be utilized.

  • Definition of the response and follow-up procedures.

  • Precise variable definitions.

  • Assessment of observer variability.

  • Strategies for missing data.

  • Preference for continuous variables over categorical ones.

  • Selection of subjects and study sites.

• Barriers to Appropriate Modeling

  • Failure to collect the most important predictor or response variables.

  • Datasets with ill-defined subjects or those not representative of the target population.

  • Data collection sites that do not reflect the population of sites.

  • Large volumes of missing data in key variables, especially when data is not missing at random.

  • Lack of operational definitions for variables or severe measurement errors.

  • Absence of observer variability studies.

• Potential Modeling Failures

  • Instability in the process generating the data.

  • Model mis-specification regarding interactions or nonlinearities.

  • Wrong transformation of the response variable or violation of distributional assumptions.

  • Presence of discontinuities that users can "game."

  • Unspecified correlations among subjects.

  • Overfitting the model to the specific dataset.

  • Estimation and validation not falling within the same range.

• Dimensions of Variables and Predictors

  • Key Dimensions: Age, sex, acute clinical stability, principal diagnosis, severity of diagnosis, extent of comorbidities, and socioeconomic/cultural attributes.

  • Predictor Capture Goals:

    • Baseline measurements of the response variable.

    • The subject's current status.

    • The trajectory or past levels of key variables at time zero.

    • Variables known to explain significant variation in the outcome.

    • Subtle predictors with distributions that differ significantly between levels of the primary variable of interest.

Medical Applications and Detailed Model Comparison

• Healthcare Industry Context

  • Key Areas: Public health, clinical practice, and medical research.

  • Prognosis Roles: Screening for conditions, establishing diagnosis, and determining therapy.

  • Decision-Making: The application of "prognostic models" or "prediction models" facilitates both clinical decision-making and shared decision-making between patient and provider.

• Hypothesis Testing vs. Estimation Examples

  • Estimation Scenario: Determining the risk of a patient dying within $30$ days of an acute myocardial infarction or the expected $2$-year survival rate for esophageal cancer.

  • Testing Scenario: Determining if age is a significant predictor of mortality or identifying which predictors are most important and if they are correlated with one another.

• Comparative Analysis of Regression Models