Chapter 4: Of Tests and Testing Flashcards

Foundational Assumptions of Psychological Testing and Assessment

• Assumption 1: Psychological Traits and States Exist

  • Traits are defined as any distinguishable, relatively enduring way in which one individual varies from another (Guilford, 1959).
  • States also distinguish individuals but are relatively less enduring (Chaplin et al., 1988).
  • The attribution of a trait or state is based on observing a sample of behavior, which can be gathered through direct observation, self-report, or electronic responses.
  • There are thousands of psychological trait terms in the English language covering intelligence, cognitive style, interests, attitudes, sexual orientation, and psychopathology (Allport & Odbert, 1936).
  • Cultural evolution introduces new terms such as "gender non-binary" for individuals who do not classify themselves on a masculine-feminine continuum, or "androgynous" in sexuality literature.
  • Psychological traits are constructs: informed, scientific concepts constructed to describe or explain behavior. They are not physical entities (like brain circuits) but are inferred from overt behavior.
  • Overt behavior refers to observable actions or the products of observable actions, including test responses.
  • Traits are "relatively enduring" rather than manifested $100\%$ of the time. While personality evolves (people may become more agreeable or conscientious as they age), it remains partially stable over the lifespan (Damian et al., 2019; Lüdtke et al., 2009).
  • The manifestation of a trait is situation-dependent; for instance, a violent parolee might be subdued with a parole officer but violent with family.
  • Trait attributions are inherently relative. Describing someone as "shy" implies a comparison to the degree of shyness expected in an average person under similar circumstances. The reference group (e.g., comparing an exotic dancer to other males versus other exotic dancers) significantly influences judgments.

• Assumption 2: Psychological Traits and States Can Be Quantified and Measured

  • E.L. Thorndike (1918) asserted that "whatever exists at all exists in some amount," and thus traits can be quantified.
  • Measuring a trait requires careful operational definitions. For example, "aggressive" might mean physically harming others in one test, or socially aggressive acts like gossiping in another.
  • Test developers select items from a "universe" (or domain) of behaviors indicative of a trait.
  • Item weighting involves awarding more points for specific responses based on technical considerations, the definition of the construct, or societal values (e.g., weighing a social judgment question more heavily than a historical knowledge question on an intelligence test).
  • Cumulative Scoring: This is the most common model where the magnitude of a trait is assumed to correspond to the sum of keyed responses. For example, a correct response equals $1$ and an incorrect response equals $0$.

• Assumption 3: Test-Related Behavior Predicts Non-Test-Related Behavior

  • The purpose of testing is rarely to predict the test-taking behavior itself (e.g., blackening grids or pressing keys); instead, it provides an indication of larger behavioral patterns.
  • Samples of behavior are used to predict future performance (e.g., work performance) or to postdict past behavior (e.g., understanding a criminal defendant's state of mind during a past crime).

• Assumption 4: All Tests Have Limits and Imperfections

  • Competent users understand test development, appropriate administration, and limitations.
  • Limitations are often compensated for by integrating data from other assessment tools.

• Assumption 5: Various Sources of Error are Part of the Assessment Process

  • Error: Traditionally refers to factors other than what the test attempts to measure that influence performance.
  • Error Variance: The component of a test score attributable to sources other than the trait or ability measured.
  • Sources of error include the assessee (e.g., having the flu), the assessor (e.g., not following standardized instructions), the instrument itself, and random chance situational conditions like weather.
  • Research by Beatrice Rammstedt (2015) suggests people rate themselves as less disciplined on sunny days than on rainy days.

• Assumption 6: Unfair and Biased Assessment Procedures Can Be Identified and Reformed

  • Test developers use sophisticated psychometric procedures to identify and correct bias.
  • Fairness problems often arise from inappropriate usage—applying a test to a background for which it was not intended—rather than the test itself.

• Assumption 7: Testing and Assessment Offer Powerful Benefits to Society

  • Assessment prevents unqualified individuals from working as surgeons or pilots.
  • It reduces nepotism by favoring documented merit and provides diagnostic tools for reading difficulties, neuropsychological impairments, and military screening.

Defining a "Good Test"

• Basic Criteria: A good test includes clear instructions for administration, scoring, and interpretation. It is economically efficient in time and money and measures what it purports to measure.
• Psychometric Soundness: Evaluated through two primary pillars: reliability and validity.
  • Reliability: The consistency and precision of the measuring tool. A perfectly reliable tool measures the same way every time.
    • Scale B is reliable if it consistently weighs a certified $1\,pound$ gold bar as $1.3\,pounds$. It is inaccurate but consistent.
    • Unreliable measurement (like Scale C, which varies wildly) must be avoided.
  • Validity: The extent to which a test measures what it purports to measure for a specific purpose.
    • If a test of intelligence is based on a definition that differs from accepted norms, its validity will be scrutinized.
    • Validity involves evaluating whether items adequately sample the domain and how scores relate to other constructs (e.g., introversion scores should inversely relate to extraversion scores).

Everyday Psychometrics: Evaluating Tests

• Instrument Selection: Questions to ask include the objective of the test, the appropriateness for the age/reading level of the testtaker, the definition of the measured variable, and the existence of alternate forms.
• Published Guidelines: The American Psychological Association (APA) and other organizations provide guidelines.
  • Example: Child custody evaluations require assessing (1) parenting capacity, (2) the child's developmental needs, and (3) the "goodness of fit" between them.
• Actionable Results: A good test yields results that benefit the testtaker or society.
• Utility: This refers to the cost-effectiveness and practical value of a test. For instance, group intelligence tests in the military during world wars were higher in utility than individual Binet tests because they were faster to administer to thousands of recruits.
• Generalizability: The extent to which findings from one administration apply to other settings or populations. This is affected by norming samples, administration deviations, and cultural variables.

The Nature and Purpose of Norms

• Norm-Referenced Testing and Assessment: Evaluating an individual's score by comparing it to a group's scores to derive standing or ranking.
• Norms: Test performance data from a specific group of testtakers used as a reference for evaluation.
• Normative Sample: The specific group of people whose performance serves as the analytical reference.
• Norming: The process of deriving norms.
  • Race Norming: The practice of establishing different cutoff scores based on race/ethnicity, outlawed by the Civil Rights Act of 1991.
• User Norms (Program Norms): Descriptive statistics based on a local group of testtakers over a period, rather than formal sampling.

Standardization and Sampling Procedures

• Standardization: The process of developing replicable procedures for administration, scoring, and interpretation, typically resulting in a standardized test.
• Sampling: Selecting a portion of the universe (population) deemed representative of the whole.
  • Stratified Sampling: Including subgroups (strata) like socioeconomic status, race, or religion in proportion to their occurrence in the population to prevent sampling bias.
  • Stratified-Random Sampling: Selecting members of strata randomly.
  • Purposive Sampling: Arbitrarily selecting a sample believed to be representative (e.g., testing a product in Cleveland as a proxy for the entire United States).
  • Incidental / Convenience Sample: Utilizing a sample that is easily available, such as introductory psychology students. Findings from these samples must be generalized with extreme caution.
• Exclusionary Criteria in Standardization: Representative samples for intelligence tests often exclude individuals who:
  • Took an intelligence test in the prior $6\,months$.
  • Are non-fluent in English or nonverbal.
  • Have uncorrected visual/hearing impairments or motor disabilities.
  • Are currently hospitalized or in a psychiatric facility.
  • Are taking medications or have historical conditions (stroke, epilepsy) that might depress performance.

Types of Norms and Scoring Systems

• Percentiles: A ranking representing the percentage of people who scored below a particular raw score.
  • Drawback: In a normal distribution, they exaggerate small differences in the middle and minimize large differences at the ends.
• Age Norms (Age-Equivalent Scores): The average performance of samples at various ages.
  • "Mental Age" is often criticized for being too broad (e.g., a $6$-year-old with a mental age of $12$ does not have the social/psychological maturity of a $12$-year-old).
• Grade Norms: Indicate average performance in specific school grades (e.g., a score of $6.4$ represents performance in the fourth month of sixth grade). They do not detail specific content mastery.
• Developmental Norms: A broad term for norms developed based on traits expected to change with age or stage of life (includes age and grade norms).
• National Norms: Derived from a sample representative of the nation across geography, socioeconomic strata, and community types.
• National Anchor Norms: Equivalency tables that compare scores across different tests (e.g., Best Reading Test vs. XYZ Reading Test) using the equipercentile method.
• Subgroup Norms: Normative information segmented by criteria like handedness or regional location.
• Local Norms: Developed by test users themselves for specific local populations (e.g., a high school developing its own norms for a statewide exam).
• Fixed Reference Group Scoring System: The distribution of scores from one group serves as a permanent standard for future scores.
  • The SAT used an 11,000-person group from $1941$; in $1995$, it transitioned to a new fixed reference group of over $2\,million$ testtakers from $1990$.
  • A score of $500$ corresponds to the mean of the 1990 sample.

Criterion-Referenced Evaluation

• Definition: A method of deriving meaning from scores by evaluating them against a set standard or criterion (e.g., needing $85\%$ to pass a pilot test or obtain a driving license).
• Masters Testing: Often used in computer-assisted education where mastery of one level is required before proceeding.
• Domain- or Content-Referenced: Focuses on what the testtaker can do rather than how they compare to others.
• Comparison: Norm-referenced tests highlight brilliance and superior abilities (scores at the right end of the normal curve), whereas criterion-referenced tests focus on basic mastery.

Professional Perspectives and Culture

• Sports Psychology Case Study: Dr. Steve Julius and Dr. Howard W. Atlas used the 16PF (fifth edition) and behavioral interviewing with the Chicago Bulls in the 1990s to evaluate competencies like resilience and relationship to authority in college draft players.
• Culturally Informed Assessment:
  • Interpret data within its cultural and historical context. Margaret Mead (1978) noted that older generations grew up under skies where "no satellite had flashed."
  • Do not assume a translated test is automatically equivalent.
  • Consult members of cultural communities to ensure assessment techniques complement their worldviews.

Standard Error Summary

Self-Assessment Core Terminology

• Construct: An informed scientific concept used to describe behavior.
• Domain Sampling: Sampling behaviors or items from all possible indicators of a construct.
• Co-norming: When two tests are normed using the same sample.
• Standard Score: Often used for z-scores, while linearly transformed scores are called "standardized scores."