[06.09] Principles of Validity and Reliability in Measurement and Research V2.pdf

Epidemiological studies

What type of studies are conducted to know more about things like the frequency of hypertension or what causes mental disorders?

Accurate reflections of reality

How must the findings of epidemiological studies reflect reality for them to answer questions reliably?

Uncertainties

What are already introduced in findings by studying only a fraction of the population (sample)?

Mask a true association OR Introduce an association which does not exist in the first place

What are two ways factors introduced by studying a sample may affect findings?

Designing and conducting research studies OR Reviewing and critiquing studies designed and executed by others

When is knowledge of epidemiology essential?

Validity, reliability, bias, and confounding

When critically appraising epidemiological studies, which concepts may determine whether the results of a study should be trusted?

Reliability

What term refers to the degree of stability exhibited when a measurement is repeated under identical conditions?

Reliability

What term refers to the degree to which the results obtained by measurement or procedure can be replicated?

Divergences between observers or instruments of measurement OR Instability of the attribute being measured

What are two potential causes for a lack of reliability?

Measurement validity

What is defined as an expression of the degree to which a measurement measures what it purports to measure?

The thermometer displays the same temperature every time

In the example of measuring liquid temperature (Example 1), what finding indicates reliable results?

The thermometer is probably malfunctioning

In Example 1, why might a thermometer's measurements not be valid, even if conditions are controlled?

Low reliability

If several different doctors use the same symptom questionnaire but give different diagnoses for the same patient (Example 2), what does this indicate?

High validity

If a symptom questionnaire results in a reliable diagnosis when answered at different times and with different doctors, what does this indicate?

Reliability on its own

What measurement characteristic is not enough to ensure validity?

The result is 2 degrees lower than the true value

In the example of the uncalibrated thermometer, why is the reliable measurement still not valid?

Reading comprehension

If a test designed to measure working memory yields reliable results but correlates strongly with this factor, the test may be measuring this instead, indicating low validity.

Reliability

When comparing the two, which concept focuses on reproducing the same results each time measurement is done under the same, controlled conditions?

Validity

When comparing the two, which concept asks whether researchers are truly measuring what they are supposed to measure?

Checking the consistency of the results

How is reliability assessed?

Across time, across different observers, and across different parts of the test itself

What three aspects are used to check the consistency of results when assessing reliability?

Checking how well the results correspond to established theories and other measures of the same concept

How is validity assessed?

Interrater reliability

What type of reliability is assessed by checking consistency across different observers?

Intra-observer reliability

What specific term indicates how stable responses are obtained from the same respondent at different time points?

Test-retest reliability

What type of reliability measures the consistency of a measure across time?

High test-retest reliability

If participants complete a personality questionnaire days or weeks apart and give the same answers, what does this indicate?

Interrater Reliability

What type of reliability measures consistency across raters or observers?

Low interrater reliability

If 5 examiners submit substantially different results for the same student project using the same checklist, what does this assessment checklist indicate?

Internal Consistency

What type of reliability measures the consistency of the measure itself?

A strong correlation between the 2 sets of results

If you randomly split the results of a self-esteem questionnaire into two halves, what finding indicates a high degree of internal consistency?

Test-retest reliability, low reliability

What type of reliability and level of reliability is demonstrated when a group completes a personality questionnaire on two occasions with significantly different answers?

Construct validity

What is the extent to which the measurement corresponds to theoretical concepts (constructs) concerning the phenomenon under study?

Content validity

What is the extent to which the measurement incorporates the domain of the phenomenon under study?

Reflect such change

If, on theoretical grounds, a phenomenon should change with age, what should a measurement with construct validity do?

Activities of daily living such as occupational, family, social functioning, etc.

What should a measurement of functional health status embrace to ensure content validity?

Adherence of a measure to existing theory and knowledge of the concept being measured

How is construct validity defined?

Social skills and optimism

What are two examples of associated traits that might be measured alongside self-esteem to indicate construct validity?

A strong correlation between the scores

What indicates high construct validity between self-esteem and associated traits?

Listening component

If a Spanish language test lacks this component, experts would agree it lacks content validity.

Criterion validity

What is the extent to which the measurement correlates with an external criterion of the phenomenon under study?

Concurrent validity and predictive validity

What are the two aspects of criterion validity?

Concurrent Validity

Which aspect of criterion validity means the measurement and the criterion refer to the same point in time?

Predictive Validity

Which aspect of criterion validity is expressed in terms of the measurement’s ability to predict the criterion?

Academic aptitude test validated against subsequent academic performance

Give an example of predictive validity.

Measuring the true value

What does validity depend on?

How consistent the test is when used by different observers or over different periods of time

What does reliability assess?

Good validity

According to Figure 2, if the true value lies close to the mean of measurements recorded, what is achieved?

Good reliability

According to Figure 2, what condition is met if there is less dispersion or variance of the measurements, resulting in a thinner graph and a higher peak?

Low or less reliability

What is indicated by a lower graph with a longer spread (Figure 2)?

X-axis

In Figure 3, which axis represents Validity?

Y-axis

In Figure 3, which axis represents Reliability?

Bull’s eye

In Figure 3, what represents the true value?

Measurements should be around the bull’s eye

If a test has High Validity (Figure 3), where should the measurements be located?

The values should be close to one another

What is the defining characteristic of high reliability in Figure 3, regardless of whether the values are centered on the bull’s eye?

High Validity and Low Reliability

In the dartboard illustration, what scenario occurs if measurements are far from the bull’s eye, but their average is actually hitting the bull’s eye?

Accuracy

What is an alternative term for Validity?

Precision

What is an alternative term for Reliability?

Points are close together and close to the true value (bull’s eye)

What characteristics define High Accuracy/Validity and High Precision/Reliability?

Internal validity

What type of study validity assumes a true association exists among the group of patients being studied (e.g., in Case #1)?

Extended or generalized to the source population

When critiquing a study, what question must be considered regarding the sample data findings?

Internal validity

What must first be followed before inferences can be applied to individuals outside the source population (external validity)?

Internal validity

What is a prerequisite for external validity?

Study validity

What refers to the degree to which the inference drawn from a study is warranted when account is taken of the study method, sample representativeness, and population nature?

Internal Validity

Which type of study validity ensures that observed differences between comparison groups are attributable only to the hypothesized effect under investigation, apart from sampling error?

Generalizability

What is External Validity also referred to as?

External Validity

Which type of study validity produces unbiased inferences regarding a target population beyond the subjects in the study?

Subject-matter-judgment

What is evaluation of generalizability often based on more than internal validity?

Systematic errors

What type of error threatens validity and is associated with a flaw in the design of a study or an instrument used for measurement?

Confounding, Selection bias and generalizability, Information bias

What are three threats to validity, other than systematic errors?

Random errors

What type of error threatens reliability?

Factors which vary from one measurement to another

What causes random errors in measurement?

Increasing the sample size

What action will minimize random errors?

The error will still be there

What happens to systematic error even if the sample size is increased?

When present, the graphs still hover over or are close to the true value of the mean

What visual characteristic is true of graphs affected by random errors (Figure 7)?

A wider and shorter graph

What visual effect do random errors lead to due to more variance?

Increase number of readings and repeat measurements

What two actions minimize random errors, leading to a narrower and taller curve?

The graphs shift from where the true value of the mean is found

When systematic errors are present, what happens to the graphs?

Values to be skewed

What is the result of systematic errors shifting the graph?

Graph shifts to the right AND Leads to a wider and shorter graph

What is the combined visual effect of random and systematic errors?

Confounding

What is the distortion of the estimated effect of an exposure on an outcome caused by an extraneous factor associated both with the exposure and the outcome?

A risk factor for the outcome among non-exposed persons AND associated with the exposure of interest

What two conditions define a confounding variable regarding risk and association?

Not an intermediate step in the causal pathway between exposure and outcome

What must a confounding variable not be?

May mask an actual association OR May falsely demonstrate an apparent association when no real association exists

What are two ways confounding may affect the relationship between treatment and outcome?

PCF is predictive of the outcome even in the absence of exposure

According to Figure 8, what characteristic must a Potential Confounding Factor (PCF) have regarding the outcome?

Associated with the exposure being studied but is not a proxy or surrogate for the exposure

What must a PCF be regarding the exposure?

Exposure cannot have an effect on the confounder

What relationship between exposure and confounder must hold true for confounding to occur, according to Figure 9?

Vertebroplasty

In Case #2, this procedure involves injecting cement into a fractured vertebra to relieve pain.

2.0 (a two-fold risk)

In the unadjusted analysis of vertebroplasty (Case #2), what was the relative risk of subsequent vertebral fractures compared to conservative care?

Smoking status

In Case #2, which potential confounding factor was found to be more common in the Vertebroplasty group (55% vs. 8%)?

Similar between treatment groups in each stratum (close to 1.1 and 1.2)

After stratifying the results by smoking status (Table 5), how did the risk of subsequent fractures compare?

Smoking

What was concluded to be a confounding factor in the vertebroplasty study because it distorted the true relationship?

The design and implementation of the study

When should the potential for confounding always be considered?

Measure and report all potential confounders

What must researchers do regarding potential confounders to address them?

Adjust for them in the analyses

What should be done routinely with confounding factors during data analysis?

Randomization, stratification, or statistical adjustment

What three methods can be used to control confounders?

True value lies close to the mean of measurements; Wide spread with lower peak

Based on Figure 2, what combination describes good validity and poor reliability?

Systematic

What type of error causes the graph to shift away from the true value of the mean, leading to skewed values that may or may not include the true mean?

Bias

What is an unavoidable problem in clinical and epidemiological research, and one of the most common sources of error?

Selection bias

What occurs when the subjects studied are not representative of the target population from which conclusions are to be drawn?

Researchers get a bias estimate of the association between the outcome and the exposure

What happens if an individual with characteristics related to both exposure and outcome is included in the study sample?