Reliability, Validity, and Hypothesis Testing in Behavioral Statistics

Fundamentals of Reliability and Validity

  • Purpose of Measurement Tools:

    • In scientific research, specialized tools measure psychological constructs such as intelligence, attention, or depression.
    • Research results are rendered inconclusive if measurement tools—including scales, tests, and questionnaires—are faulty.
    • Analogy: Using faulty measurement tools is equivalent to constructing a house using tape measures that are off by varying degrees, levels that are inaccurate, and scales that fail to function consistently.
    • Dependability and accuracy must be empirically verified for all instruments.
  • Core Definitions:

    • Reliability: The consistency of a measurement tool or instrument.
    • Validity: The accuracy of a measurement tool or instrument.

Classical Test Theory and Components of Measurement

  • Error in Measurement:

    • All empirical measures inherently contain sources of error.
    • Any Observed Score (such as performance on an I.Q. test, a reading from a bathroom scale, or an exam score) comprises exactly 22 components:
    • True Score: The theoretical component that genuinely reflects the construct of interest, such as actual intellectual capability, true physical weight, or actual statistical knowledge.
    • Error Score: The component of the score introduced by extraneous factors, such as test bias, calibration imbalance on a scale, or distracting testing environments (e.g., an air conditioning breakdown causing excessive heat).
  • Characteristics of a Reliable Measure:

    • A reliable tool minimizes the error score as much as possible, demonstrating consistent measurement across trials.
    • High-Error Demonstration in Testing: Scoring far below average on Part A of an I.Q. test and scoring in the certified genius range on Part B indicates substantial measurement error.
    • High-Error Demonstration in Physical Measurement: Stepping on a scale to find a target weight goal, then stepping on it immediately afterward to find an increase of 100 lbs100\,\text{lbs}, demonstrates that the instrument is prone to error rather than reflecting a true weight shift.

Forms and Methodologies of Reliability

  • Test–Retest Reliability:

    • Involves measuring the exact same individuals at 22 distinct points in time.
    • Example: Administering an I.Q. test on Tuesday and readministering it on Friday to the same subjects. Inconsistent scores across administrations indicate low reliability.
  • Internal Consistency:

    • Evaluates whether all items on a single test measure the same construct during a single administration.
    • Split-Half Reliability:
    • Assesses reliability by splitting a test in half and examining the relationship between the halves.
    • Can be evaluated by comparing Part A to Part B of a measure (such as an Introversion scale), or by correlating scores on even-numbered items with scores on odd-numbered items.
    • Cronbach's Alpha:
    • Mathematically correlates every item with every other item across the test.
    • Demonstrates whether all items designed to measure a specific psychological construct (such as introversion) show high inter-item correlation.
  • Interrater Reliability:

    • The degree of agreement or consistency among independent observers or raters.
    • Example: If an instructor and a teaching assistant grade the same student essay examinations but consistently disagree on scoring, interrater reliability is excessively low, rendering the grading protocol unreliable.

Forms and Methodologies of Validity

  • Core Concept:

    • Validity confirms whether an instrument measures what it is supposed to measure.
    • High reliability does not ensure validity. For instance, a scale reading 8000 lbs8000\,\text{lbs} consistently across every single trial is perfectly reliable, yet invalid. Similarly, consistently scoring 2020 on an I.Q. test on repeated trials indicates reliability, but fails to provide a valid measure of intelligence.
  • Construct Validity:

    • Assesses whether the instrument accurately captures the underlying theoretical construct.
    • Measured by correlating the instrument with other validated measures designed to assess the same construct.
    • Example: Demonstrating construct validity for the GRE by establishing correlations with the SAT and standardized Reading Comprehension tests.
  • Content Validity:

    • Evaluates whether the operationalized test content is fully representative of the specified behavioral or knowledge domain.
    • Example of Poor Content Validity: An introductory statistics examination consisting exclusively of correlation problems, omitting means, standard deviations, and ANOVAs. This exam measures correlation proficiency rather than the broader domain of statistics.
    • Subtypes Related to Construct Differentiation:
    • Convergent Validity: Two separate instruments measuring the same construct should produce highly correlated, similar scores.
      • Example: Comparing scores on the Introversion sub-scale of the Five Factor Inventory (FFI) with the Introversion sub-scale of the Myers-Briggs Type Indicator.
    • Discriminant Validity: Two separate instruments measuring conceptually distinct constructs should produce divergent, dissimilar scores.
      • Example: Because introversion is theoretically distinct from shyness, scores on the Introversion sub-scale of the Five Factor Inventory (FFI) should differ substantially from scores on the Wellesley Shyness Test.
  • Criterion Validity:

    • Evaluates how effectively test scores relate to an external benchmark or standard, either in the present or in the future.
    • Concurrent Validity:
    • Assesses whether test performance corresponds to a person's current ability.
    • Example: Determining whether current GRE scores accurately reflect concurrent academic performance in graduate school.
    • Predictive Validity:
    • Assesses whether test performance forecasts ability or outcomes at a future point in time.
    • Example: Determining whether GRE scores successfully predict graduate program completion rates and career employment outcomes.

Quantitative Measurement of Reliability and Validity

  • Statistical Indexing:

    • Both reliability and validity are typically quantified using correlation coefficients or related statistical indices (such as Cronbach's alpha), yielding coefficients between 00 and 11.
    • Values near 00 signify very low reliability or validity.
    • Values near 11 signify high reliability or validity.
  • Evaluative Scenarios and Practice Classifications:

    • Scenario 1: Blanca Evangelista administers an exam to participants and correlates scores on the even-numbered items with odd-numbered items, yielding a correlation of r=0.89r = 0.89.
    • Type: Split-Half Reliability.
    • Magnitude: High reliability.
    • Scenario 2: Electra Abundance administers a fashion quiz to models to forecast runway performance the subsequent week; the quiz correlates with show placement at r=0.14r = 0.14.
    • Type: Predictive Validity (Criterion Validity).
    • Magnitude: Low validity.
    • Scenario 3: Dr. Pray Tell calculates the rate of scoring agreement among 55 judges at a talent show, resulting in a coefficient of 0.520.52.
    • Type: Interrater Reliability.
    • Magnitude: Moderate reliability.
    • Scenario 4: Choreographer Damon Richards considers poise distinct from grace. He administers scales measuring both poise and grace to dancers, yielding a correlation of r=0.14r = 0.14.
    • Type: Discriminant Validity.
    • Magnitude: High discriminant validity (a low correlation confirms that the two constructs are distinct).

Foundations of Hypothesis Testing

  • Overview of Hypothesis Formulation:

    • Research in psychology progresses from foundational research questions to formal, testable hypotheses.
    • Hypothesis: A specific, testable statement derived from a research question, informed by formal theory and systematic observation; an educated prediction.
    • Hypotheses are developed following literature reviews of peer-reviewed articles addressing related questions.
    • Example:
    • Research Question: Do some personality types enjoy statistics more than others?
    • Hypothesis: People who score below the median on extroversion on the Five Factor Inventory (FFI) will rate the enjoyment of a statistics course higher than those who score above the median.
  • Populations Versus Samples:

    • Population: The entirety of the group about which a researcher intends to draw empirical conclusions.
    • Example: Every adult female science fiction fan in the world.
    • Sample: The specific subset of individuals observed and assessed to gather data.
    • Example: A sample of 4444 friends responding to a Facebook advertisement to complete a survey evaluating perceptions of misogyny in science fiction.
    • The numerical size of a sample is inherently smaller than the total size of the population.
  • Sampling Error:

    • The extent to which a sample fails to accurately represent its target population.
    • Can occur through systematic sample biases (e.g., all 4444 survey respondents having viewed exclusively older science fiction films, or all respondents residing in a region with atypical cultural norms regarding sexism).
    • Researchers minimize sampling error primarily through random sampling procedures.
  • Statistical Significance and Chance:

    • In a sample of n=30n = 30 individuals, extroverts might score 0.080.08 points higher than introverts on a 77-point Likert scale measuring course enjoyment.
    • Hypothesis testing and tests of statistical significance distinguish whether an observed difference is a genuine effect or merely an artifact of random sampling chance.

The Null Hypothesis Versus the Research Hypothesis

  • The Null Hypothesis (H0H_0):

    • Asserts that there is no difference or no effect among the groups under evaluation.
    • Represents the statistical baseline indicating that any observed sample difference falls within the range expected by chance alone.
    • Formulated at the theoretical level and expressed using population parameters (Greek letters, such as μ\mu).
    • Examples:
    • Individuals scoring below the median on extroversion on the FFI will rate their enjoyment of a statistics course equal to those scoring above the median.
    • A group of depressed clients administered an SSRI will demonstrate equal improvement compared to depressed clients receiving a placebo.
  • The Research Hypothesis (H1H_1 / HiH_i / Alternative Hypothesis):

    • Asserts an actual difference among groups that exceeds expectations based on chance.
    • Represents a statement of inequality.
    • Evaluated based on empirical observation and formulated using sample statistics (Roman alphabet symbols, such as Xˉ\bar{X} or XX).
    • Examples:
    • Introverts like statistics more than extroverts (\text{I} > \text{E}$).\n * Extroverts like statistics more than introverts (\text{E} > \text{I}$).
    • Extroverts will score differently than introverts (\text{I} \neq \text{E}$).\n\n* **Comparative Summary:**\n * H_0statesthatobservedrelationsdonotexceedchance;states that observed relations do not exceed chance;H_1 states that group inequalities exceed chance.\n * H_0alwaysappliestothepopulation;always applies to the population;H_1 always applies to the empirical sample.\n\n# Types of Research Hypotheses\n\n* **Nondirectional Research Hypotheses:**\n * State that a difference exists between the comparison groups, but do not predict or specify which group will score higher.\n * *Example:* Extroverts will score differently than introverts on statistics enjoyment (\text{I} eq \text{E}).\n\n* **Directional Research Hypotheses:**\n * Explicitly specify the predicted direction of the difference, defining which group is anticipated to have higher or larger scores.\n * *Examples:*\n * Introverts enjoy statistics more than extroverts (\text{I} > \text{E}).\n * Extroverts enjoy statistics more than introverts (\text{E} > \text{I}).\n\n* **Applied Derivation Exercise:**\n * *Baseline Null Hypothesis (H_0):* The group of depressed clients who take an SSRI will show equal improvement on a pain scale compared with depressed clients who take a placebo.\n * *Derived Nondirectional Hypothesis (H_1):∗ThegroupofdepressedclientswhotakeanSSRIwillshowadifferentlevelofimprovementonapainscalecomparedwithdepressedclientswhotakeaplacebo():* The group of depressed clients who take an SSRI will show a different level of improvement on a pain scale compared with depressed clients who take a placebo (\mu_{\text{SSRI}} eq \mu_{\text{Placebo}}).\n * *Derived Directional Hypothesis (H_1):∗DepressedclientstakinganSSRIwillshowsignificantlygreaterimprovementonapainscalecomparedwithdepressedclientstakingaplacebo():* Depressed clients taking an SSRI will show significantly greater improvement on a pain scale compared with depressed clients taking a placebo (\mu_{\text{SSRI}} > \mu_{\text{Placebo}}).\n\n# Statistical Notation and Formal Symbolic Statements\n\n* **Standard Statistical Symbols:**\n * \mu: Population Mean\n * \bar{X}ororX: Sample Mean\n * \bar{X}_AororX_A: Sample Mean for Group A\n * \bar{X}_BororX_B: Sample Mean for Group B\n * H_1ororH_i: Research (Alternative) Hypothesis\n * H_0ororH_o: Null Hypothesis\n * eq: Does not equal\n * >: Greater than\n *