1.4 Validity

Validity

A test measures what it's supposed to measure in a particular context

Validity Coefficient

Correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure

Inference

A logical result or deduction

Validation

Process of gathering and evaluating evidence about validity

Local Validation Studies

Studies that are absolutely necessary when the test user plans to alter in some ways the format, instructions, language, or content of the test

Ecological Validity

Refers to a judgment regarding how well a test measures what it purports to measure at the time and place that the variable being measured is actually emitted

Internal, External, Conceptual, & Face

Types of Validity

Internal Validity

Degree of control among variables in the study (increased through random assignment)

External Validity

Generalizability of the research results (increased through random selection)

Conceptual Validity

How well a test measures what it's designed to evaluate

Face Validity

A test appears to measure what it actually measures at face-value

Content, Criterion, & Construct

Trinitarian View of Validity

Content Validity

An evaluation of the subjects, topics, or content covered by the item in the test

Criterion-Related Validity

Evaluating the relationship of scores obtained on the test to scores on other tests or measures

Construct Validity

How scores on the test can be understood within some theoretical framework for understanding the construct that the test was designed to measure

Content Validity

A fundamental psychometric property that addresses the extent to which a measurement instrument (e.g., a test, questionnaire, or observational tool) adequately and comprehensively represents all relevant facets or domains of the construct it is designed to measure

“Does the test content truly reflect the content domain it’s supposed to cover?”

Construct

Refers to a theoretical concept, trait, or attribute that cannot be directly observed or measured (e.g., intelligence, depression, anxiety, job satisfaction, mathematical ability)

Construct Underrepresentation

Failure to capture important components of a construct

Construct-Irrelevant Variance

Happens when scores are influenced by factors irrelevant to the construct

Relevance & Representativeness

Core Principles of Content Validity

Relevance

Core principle of content validity that is directly pertinent to the construct being measured

Representativeness

Core principle of content validity that adequately samples all important sub-domains or facets of the construct

Content Validity

What type of validity is a necessary precursor for other forms of validity?

Test Blueprint

A detailed plan of the content, organization, and quantity of the items that a test will contain

Subject Matter Expert (SME) Review

It is when a panel of qualified subject matter experts (SMEs) independently reviews each item of the instrument against the defined content domain

Lawshe's Content Validity Ratio (CVR) & Content Validity Index (CVI)

Quantitative Indices for Content Validity

Lawshe’s Content Validity Ratio (CVR)

For each item, SMEs typically rate its essentiality (e.g., “essential,” “useful but not essential,” “not necessary”)

Positive CVR

In CVR , this result means more than half of the SMEs consider the item essential

Zero CVR

In CVR, this result means exactly half of the experts rate the item as essential

Content Validity Index (CVI)

Can be calculated at the item level (I-CVI) or scale level (S-CVI)

I-CVI

The proportion of SMEs who rate an item as highly relevant (e.g., 3 or 4 on a 4-point relevance scale)

S-CVI

The average of the I-CVIs for all items on the scale (S-CVI/Ave) or the proportion of items that achieve universal agreement (e.g., all experts rate it as 3 or 4) (S-CVI/UA)

Pilot Testing and Cognitive Interviewing

The instrument can be pilot tested with a sample from the target population

Criterion-Related Validity

Demonstrates how well a test or measurement instrument predicts or correlates with a relevant external criterion

Criterion

A separate, independent, and often objective measure that serves as a standard against which the test scores are compared

Concurrent & Predictive

Types of Criterion-Related Validity

Concurrent Validity

Established when the test scores and the criterion measures are collected at approximately the same time

Concurrent Validity

It assesses how well the test can estimate an individual’s current standing on the criterion and is often used when a new test is developed to replace an existing, well-validated but perhaps more cumbersome, expensive, or time-consuming measure

Predictive Validity

Established when the test scores are collected at one point in time, and the criterion measures are collected at a later point in time

Predictive Validity

It assesses how well the test can predict future performance, behavior, or outcomes

Incremental Validity

The degree to which an additional predictor explains something about the criterion measure that is not explained by the predictors already in use

Construct Validity

A central concept in psychometrics and refers to the degree to which a test or measurement instrument adequately measures the theoretical construct it purports to measure

Construct

A theoretical, abstract concept or attribute that is not directly observable but is inferred from observable behaviors or indicators

Clear conceptualization, Operationalization, & Empirical Evidence

Core Principles of Construct Validity

Clear Conceptualization

A precise and well-defined theoretical understanding of the construct, including its dimensions, relationships to other constructs, and expected manifestations

Operationalization

Translating the abstract construct into concrete, measurable indicators or test items

Empirical Evidence

Gathering observable data that supports or refutes the hypothesized relationships between the test scores and other variables, consistent with the theoretical framework of the construct

Evidence of Homogeneity, Evidence of Changes with Age, Evidence of Pretest-Posttest Changes, Known-Groups Validity (Group Differences), Convergent Validity, & Discriminant Validity

Evidences of Construct Validity

Evidence of Homogeneity

Broader concept in construct validity that refers to the entire network of theoretical relationships between a construct and other constructs, as well as observable behaviors

Evidence of changes with age

Evidence observed if the construct is expected to change over time (e.g., cognitive ability in children, emotional maturity), the test scores should reflect these expected developmental patterns

Evidence of pretest-posttest changes

Evidence if the construct is expected to change in response to a specific intervention (e.g., a therapy for depression, a training)

depression, a training)

Known-Groups Validity (Group Differences)

Can differentiate between groups of individuals who are known, based on external criteria, to differ on the construct being measured

Convergent Validity

Scores on the test undergoing constructed validation tend to correlate highly in the predicted direction with scores on older, more established, and already validated test design to measure the same construct

Discriminant Validity

Validity showing a little relationship between test scores and/or other variables with which scores on the test being construct-validated should theoretically be correlated

Multi-trait Multi-method Matrix (MTMM)

It is a method for assessing the construct validity of a set of measures in a study

Monomethod-monotrait Correlations (Reliability Diagonal), Heteromethod-monotrait Correlations (Validity Diagonal), Monomethod-heterotrait Correlations (Heterotrait-Monomethod Triangles, & Heteromethod-heterotrait Correlations (Heterotrait-Heteromethod Triangles)

Analysis of 4 Types of Correlation Coefficients

Monomethod-monotrait Correlations (Reliability Diagonal)

Correlation coefficient in which the correlations of a trait with itself, measured by the same method, represent the reliability of each measure and should be the highest in the entire matrix

Heteromethod-monotrait Correlations (Validity Diagonal)

Correlation Coefficient that represents the same trait measured by different methods, significant correlations here provide evidence of convergent validity, meaning that different measures of the same trait converge or agree

Monomethod-heterotrait Correlations (Heterotrait-Monomethod Triangles)

Correlation coefficient that is between different traits measured by same methods—should be low, providing evidence of discriminant validity, meaning the measures can discriminate between different constructs

Heteromethod-heterotrait Correlations (Heterotrait-Heteromethod Triangles)

Correlation coefficient that is between different traits measured by different methods—should be the lowest in the matrix, further supporting discriminant validity

Factor Analysis

A class of mathematics procedures designed to identify factors or specific variables that are typically attributes, characteristics, or dimensions on which people may differ

Data Reduction

The most common purpose of factor analysis is to simplify complex data by reducing the number of variables, making dataset easier to work and interpret

Structure Discovery

Factor Analysis helps uncover the underlying structure or dimensions within a set of observed variables

Construct Validation

Factor Analysis is used to develop and refine measurement scales (like personality tests) to ensure that the items truly measure the intended underlying construct (e.g., intelligence, anxiety)

Latent Variables

Provides an estimate of how strongly each observed variable relates to each factor, known as factor loading

Factor Loading

Conveys information about the extent to which the factor determines the test score

Exploratory & Confirmatory

Types of Factor Analysis

Exploratory Factor Analysis

Type of Factor Analysis used to explore and discover the underlying factor structure when the researcher has no preconceived idea about how many factors exist or which variables belong to which factor. “theory-generating”

Confirmatory Factor Analysis

Type of Factor Analysis used to confirm a hypothesized factor structure. “theory-testing”

Principal Component Analysis (PCA)

Method used to reduce data dimensionality and summarize variance with fewer variables

Dimensionality Reduction

The primary goal of PCA that is reducing the number of variables (features) in the data set

Variance Maximization

The principal components of PCA are chosen to capture the maximum possible variance in the data

Orthogonality. The principal components of PCA are orthogonal (perpendicular) to each other, meaning they are uncorrelated

Scree Plot

In PCA, it is the plot of the eigenvalues vs. the number of the principal component. It helps in deciding how many components to retain by looking for an “elbow” or point where the explained variance drops sharply (point of inflection)

Explained Variance Ratio

In PCA, it is the proportion of the dataset’s variance that lies along each principal component

Kaiser Criterion (K1 Rule) & Elbow Method (Scree Test)

Two Common Traditional Heuristic Methods in EFA and PCA

Kaiser Criterion (K1 Rule)

Criterion used to retain all factors or components that have an eigenvalue greater than 1.0

Limitation of the Kaiser Criterion

Tends to overestimate the number of factors, especially when the number of variables is large. It is generally not considered the most accurate method

Elbow Method (Scree Test)

Criterion used to plot the eigenvalues on the Y-axis against the factor number on the X-axis. Retain all factors that appear before the “elbow” (the point where the slope of the curve sharply levels off)

Limitation of the Elbow Method

Highly subjective, as the “elbow” point is often ambiguous or hard to define, leading different researchers to select different numbers of factors from the same plot

Cross-Validation

Revalidation of the test to a criterion based on another group different from the original group form which the test was validated

Validity Shrinkage, Co-Validation, & Co-Norming

Concepts related to Cross-Validation

Cross-Validation

Revalidation of the test to a criterion based on another group different from the original group form which the test was validated

Validity Shrinkage

Decrease in validity after cross-validation

Co-Validation

Validation of more than one test from the same group in cross-validation

Co-Norming

In cross-validation, it is norming more than one test from the same group