STATS lecture 3

Reliability

indicates the consistency of the measures it obtains of an attribute, concept or situation in a study or clinical practice (dependability, consistency, stability, reproducibility and comparability)

Reliability testing

examined the amount of measurement error in an instrument that is used in the study

Stability reliability

concerned with the consistency oof repeated measures of the same variable or attribute with the same scale or measurement method over time.

Test-retest reliability

repeated measurement of a variable over time, examines the amount of random error in a measurement technique. A strong tee-retest reliability is desired

Equivalence Reliability

compares two versions of the same scale or instrument or two observers measuring the same event.

Interrater reliability

comparison of two observers

Alternate-forms reliability/parallel reliability

comparison of two scales (alternate forms of instruments are often used in the development of normative knowledge testing i.e. graduate record exams)

Internal consistency/ homogeneity reliability testing

examines the extent to whom all the items in a multiple-item instrument or scale consistently measures a variable

• measured with cronbach alpha (liker and differential scale) and Kuder-Richardson formula (KR-20)(dichotomous (yes/no, 2 opts.) or nominal data (boxes))

• multi-item scales usually include muscles and internal consistency is determined for these sub scales and the total scale

Cronbach’s Alpha

score of 1.00 reflect perfect reliability (never usually reported, all instruments have error)

• 0.00 = no scale reliability

Strong Internal Reliability

≥0.80

Moderate Internal Reliability

0.60-0.80

Low Internal Reliability

<0.60

Cronbach Alpha measurement of error and percentage of error equation

1.00-(conbach’ alpha)²= measurement of error ex: 1.00-(0.80)² = 1.00-0.64=0.36

100% x error = percentage of error ex: 100% x 0.36 = 36% error

Precision

Degree of consistency or reproductivity of measurements made with physiological measure of the same variable or object under specified conditions

• determined by manufacturers and is partly controlled by agency using the device

• physiological equipment should be recalibrated as indicated by the manufacturer

• higher levels of precision (0.90-.99) important when monitoring critical physiological functions

Validity

determines whether the measurement method accurately reflects the concept it was developed

• focus on appropriateness, meaningfulness and usefulness of the measurement method

• 3 categories: ** 1) content related validity 20 construct validity 3) criterion related validity

1) Content Related Validity

examines the extent to which the measurement method contains all the major elements relevant to the concept being measured

determined by:

• review of literature

• construction of the scale based on literature and researcher expertise

• review your experts (completeness, conciseness, clarity and readability)

• readability is essential component of both validity and reliability (easy reading level)

Construct Validity

focus of determining whether the instrument actually measures the theoretical construct that it is expected to measure

• examine the fit between the conceptual and operational definition of a variable

Convergent validity (part of construct)

examined by comparing a newer instrument with an existing instrument that measures the same concept or construct

• validity of other instruments in strengthened when the values obtained a=have a moderate to strong positive relationship

Divergent Validity (part of construct)

examined when the scores from an existing instrument are correlated with the scores from an instrument measuring the opposite concept

• two scales measuring opposite things could be examined for divergent validity (i.e. hope vs hopelessness)

Factor Analysis

examines if an instrument includes the elements of the concept being measured

-researchers use Exploratory and or Confirmatory factor analysis to determine if the items in an instrument are related and cluster those together

• if factor analysis results identify the essential elements of the concept to e measured by an instrument then the validity is strengthened

Validity from Contrasting Groups

Identifying groups that are expected or known to have contrasting scores on an instrument and then asking the group to complete the instrument

• if scores contrast then the validity of the instrument is strengthened

• (i.e scores of people with diabetes compared to people without diabetes on an insulin competency scale)

Successive verification validity

achieved when an instrument is used in additional studies with a variety of subjects

Criterion-Related Validity

strengthened when a study participants score on an instrument can be used to infer his or her performance on another variable or criterion

Validity from the prediction of future events (Criterion-related)

achieved when the scores on an instrument can be used to predict future behaviors, attitude, and events (i.e. fall risk scale, ACT, SAT)

Validity from prediction of concurrent events (Criterion-related)

can be tested by examining the ability to predict the concurrent value of an instrument on the basis of the value obtained on an instrument to measure another concept

• i.e. you might use the results of a self-esteem scale to predict the results of a confidence scale

Accuracy

accuracy of physiological and biochemical measures is similar to the validity of scales used in research

• involves determining the closeness of the agreement between the measured value and the true value of the physiological variable being measured

• - new measurement devices are being compared to old

• - there should be a very strong positive correlation