Criterion-Referenced and Norm-Referenced Reliability, Validity, Correlation, and Distribution Practice Flashcards

Comprehensive Question and Answer flashcards covering Criterion-Referenced Tests, Epidemiology, Correlation, Linear Regression, Reliability, Validity, and Distribution Variability from the lecture series.

What are Criterion-Referenced Tests (CRT) and what type of data are they commonly used for?

Criterion-Referenced Tests are evaluations used to make categorical decisions using cutoff scores derived from continuous data. They are commonly used for categorical or nominal data.

What is a cutoff score in criterion-referenced testing?

A cutoff score is a established threshold value that defines identifiable groups or levels of performance.

What are the four approaches used to develop criterion-referenced standards?

1. Judgmental Approach (based on expert beliefs or experience)
2. Normative Approach (uses norm-referenced data to set standards with a theoretically accepted criterion)
3. Empirical Approach (uses an external criterion measure with cutoff scores directly based on available data)
4. Combination Method (combines experts, prior experience, empirical data, and norms).

What are the main advantages of Criterion-Referenced Tests (CRT)?

CRT standards represent specific desired performance levels linked to a criterion, establish absolute standards independent of population proportions, base achievement on reaching the standard rather than competition, allow specific diagnostic evaluations, and provide participants with explicit expectations.

What are the limitations of Criterion-Referenced Tests (CRT)?

Cutoff scores always involve subjective judgment, misclassifications can have severe consequences, and participants who reach the cutoff level may lack motivation to continue improving.

According to physical activity guidelines, what minimum level of activity is recommended for adults to obtain substantial health benefits?

Adults should perform at least $150\,\text{minutes/week}$ of moderate-intensity activity, $75\,\text{minutes/week}$ of vigorous physical activity, or an equivalent combination.

What is the Proportion of Agreement ($P$) in criterion-referenced testing, and what is its equation?

Proportion of Agreement ($P$) is the percentage of consistent performance categorizations across methods or trials, ranging from $0$ to $1$. Its formula is $P = \frac{n_1 + n_4}{n_1 + n_2 + n_3 + n_4}$. It does not account for chance agreements.

How is Kappa ($K$) defined, and why is it preferred over Proportion of Agreement ($P$)?

Kappa ($K$) is a measure of agreement or association between categorical variables adjusted for chance agreements. Its formula is $K = \frac{P - P_c}{1 - P_c}$ and it theoretically ranges from $-1$ to $+1$. Negative values imply chance agreement exceeds observed agreement.

In this 2x2 contingency table for stability, what do cells $n_1$, $n_2$, $n_3$, and $n_4$ represent?

$n_1$ represents participants who did not meet standard on Day 1 nor Day 2; $n_2$ represents those who met standard on Day 1 but not Day 2; $n_3$ represents those who did not meet standard on Day 1 but did on Day 2; $n_4$ represents those who met standard on both days.

What is epidemiology, and how do descriptive and analytical epidemiology differ?

Epidemiology is the study of the distribution and determinants of health-related states or events in populations. Descriptive epidemiology describes the frequency and distribution of mortality/morbidity by time, place, and person, while analytical epidemiology investigates the causes and prevention of mortality/morbidity.

Based on Table 7.9, how are research designs in epidemiology classified into experimental and observational types?

Experimental designs include randomized clinical trials and community trials. Observational designs include case series, cross-sectional studies, proportionate mortality or morbidity studies, case-control studies, and cohort studies.

What is the difference between incidence and prevalence in epidemiology?

Incidence is the number, proportion, rate, or percentage of NEW cases of mortality or morbidity, whereas prevalence is the number, proportion, rate, or percentage of TOTAL cases (both existing and new) in a population.

What is a correlation coefficient ($r$), and what information does it provide?

Denoted by $r$, it is an index of the linear relationship between two variables, ranging from $-1$ to $+1$. It indicates both the magnitude (strength) and direction (positive or negative) of the linear relationship and has no units.

What does a negative correlation coefficient indicate about the relationship between two variables?

A negative correlation indicates that participants scoring above the mean on one variable are more likely to score below the mean on the second variable. It indicates an inverse relationship (e.g., higher body weight related to fewer pull-ups) rather than a bad outcome.

What is the Coefficient of Determination ($r^2$)?

The Coefficient of Determination ($r^2$) represents the proportion of shared variance between two measures. For example, if $r = 0.90$, then $r^2 = 0.81$, meaning $81\%$ of the variation is shared/accounted for, and $19\%$ is unexplained error.

What are the three major limitations of correlation analysis?

1. It assumes linear relationships and is NOT the appropriate analysis for curvilinear relationships.
2. Correlation does NOT imply causation (spurious correlations can occur).
3. Restricting the variance or range of variables reduces the magnitude of $r$.

What is simple linear regression, and what are its independent ($X$) and dependent ($Y$) variables?

Simple linear regression is a statistical method used to predict a dependent outcome variable ($Y$) from a single independent predictor variable ($X$) using the equation $\hat{Y} = b X + c$. Variable $X$ is the controlled/predictor variable and $Y$ is the measured outcome variable.

How is prediction error ($E$) calculated in linear regression, and what is shown in this pull-ups prediction example?

Prediction error (residual) is $E = Y - \hat{Y}$. For Subject 3 weighing $140\,\text{lb}$, actual pull-ups $Y = 12$, predicted pull-ups $\hat{Y} = 7.97$, resulting in an error $E = 12 - 7.97 = 4.03$.

What is the Standard Error of Estimate (SEE)?

Standard Error of Estimate (SEE or SEP) reflects the average amount of error in predicting $Y$ from $X$, calculated as the standard deviation of the residual or error scores ($E = Y - \hat{Y}$).

What are the five quality criteria for selecting the best test?

1. Validity (truthfulness)
2. Reliability (consistency)
3. Objectivity (equal among observers)
4. Relevance (appropriate)
5. Feasibility (realistic).

Do reliability and validity refer to the test instrument itself or to the test results?

Reliability and validity refer strictly to the RESULTS obtained from a test under specific conditions, NOT to the test instrument itself.

How is test reliability defined?

Reliability is the degree to which repeated measurements of the same trait are reproducible under the same conditions (the consistency, repeatability, stability, or dependability of an observation).

What are the three main types of reliability?

1. Test-Retest Reliability (consistency of a measure over time, ideally with a minimum 2-week interval to avoid learning effects).
2. Inter-Rater Reliability / Objectivity (degree of agreement between two or more independent raters).
3. Intra-Rater Reliability (stability of one rater's scores across multiple assessments).

How is reliability interpreted numerically when reported as a correlation ($r$) or intraclass correlation (ICC)?

Scores closer to $1.0$ represent higher reliability:

• $0.0\text{ to }0.3$ = Low reliability
• $> 0.3\text{ and }< 0.7$ = Medium reliability
• $> 0.7$ = High reliability.

What is validity, and why is it predicated on reliability?

Validity is the degree to which a test measures what it is intended to measure (soundness or truthfulness). It is predicated on reliability because results cannot be valid if they are not reliable.

What is Content Validity, and how is it determined?

Content Validity (also called Face or Logical Validity) is evidence of truthfulness based on logical decision-making. It requires verification by domain content experts.

What is the difference between Concurrent Validity and Predictive Validity?

Concurrent Validity measures the statistical relationship between two different instruments assessing the same construct at the same time. Predictive Validity examines how well a measure taken at Time 1 predicts a future criterion score measured at Time 2.

What is Construct Validity, and what are its two subcategories?

Construct Validity is the highest form of validity, combining logic and statistical evidence to measure an unobservable theoretical construct (such as intelligence). Subcategories:

• Convergent Validity: Evidence that variables theoretically related do correlate.
• Discriminant Validity: Evidence that variables theoretically distinct do not correlate.

In a distribution, how do negatively skewed, normal, and positively skewed shapes differ as illustrated in this figure?

Negatively skewed distributions (skewness $-1$) have a long tail extending to the left; Normal distributions (skewness $0$) are symmetric; Positively skewed distributions (skewness $+1$) have a long tail extending to the right.

How does skewness affect measures of central tendency (mean, median, mode)?

Skewness pulls the Mean furthest out into the tail of the distribution, places the Median in the middle, and leaves the Mode at the peak of the distribution.

What is kurtosis, and what do mesokurtic, platykurtic, and leptokurtic mean?

Kurtosis measures the peakedness of a distribution:

• Mesokurtic: Average, normal peakedness
• Platykurtic: Flat distribution
• Leptokurtic: Highly peaked distribution.

What are range, variance, and standard deviation, and which is the most stable measure of variability?

Variability measures score dispersion:

• Range: High score minus low score (unstable).
• Variance ($s^2$): Average of squared deviations from the mean; it is the MOST STABLE measure of variability.
• Standard Deviation ($s$): Square root of the variance, accounting for every score.

Based on Table 3.2, what are the exact steps to calculate variance ($s^2$) from observed scores $X$?

1. Subtract the mean ($M = 3$) from each observed score $X$ to get deviation scores x = X - M$.\n2. Square each deviation score x^2$.
2. Sum the squared deviation scores ($\sum x^2 = 20$).
3. Divide by total observations ($N = 10$) to obtain variance ($s^2 = \frac{20}{10} = 2$).

What is a z-score, and what are its mean and standard deviation?

A z-score standardizes observations around a mean and standard deviation to allow direct comparison of different variables or units. A standard z-score distribution has a mean ($M$) of $0$ and a standard deviation ($s$) of $1$. Its formula is $z = \frac{X - M}{s}$.

Under a normal curve, what percentages of scores fall within $\pm 1\,\text{SD}$, $\pm 2\,\text{SD}$, and $\pm 3\,\text{SD}$ of the mean?

• Within $\pm 1\,\text{SD}$ ($z = -1$ to $+1$): $68.26\%$ ($34.13\%$ on each side of mean)
• Within $\pm 2\,\text{SD}$ ($z = -2$ to $+2$): $95.44\%$ ($13.59\%$ between $1\,\text{SD}$ and $2\,\text{SD}$)
• Within $\pm 3\,\text{SD}$ ($z = -3$ to $+3$): $99.74\%$ ($2.15\%$ between $2\,\text{SD}$ and $3\,\text{SD}$).