NSE 212 Week 6: Quantitative data quality & levels of measurement

Test-retest(stability)(reliability)

• Does the instrument produce the same results as the repeated testing?

• Stable instrument scares when administered more than once to same participants under similar conditions

Internal consistency (homogeneity)(reliability)

• Do the items on survey belong together and measure the same concept/characteristics 

• With which the items in a scale reflect measure the same concept 

• Items in scale are correlated with each other and the subparts measure the same characteristics

Equivalence (interability)(reliability)

• Does the tool produce the same results when equivalent procedures are used 

•  is either consistency or agreement among observers who use the same measurement tool or agreement between alternative forms of a tool

• Used with structured  instruments that used to observe behaviours that are systematically recorded

• Refers to the consistency of observations between 2 or more observers with the same measurement tool or 1 observer could observe the same behaviour on several occasions 

• Reliability of observer not instrument 

Reliability

extent to which an instrument yields the same results on repeated measures 

• Concerned with consistency, accuracy, precision

• Ex: thermometer that gives same correct reading over and over

Validity

refers to whether an instrument measures what it intends to measure

Content validity *

Degree to which control of the measure represents the universe of content, domain of given construct 

Concern (content validity)

whether the measurement tool & items are representative of the universe of content to be measured

Face validity (content subtype)

• Instrument appears to be measuring the appropriate concept 

• Judgement based 

• Not considered an alternative to other types of validity

Criterion- related validity

• The degree of relationship between the participants on the measurement tool and the participants actual behaviours 

• Assesses same concept being studied 

• Two types 

  • Concurrent validity 

  • Predictive validity

Concurrent validity (criterion related)

• Degree of correlation of the measures of the same construct administered at the same time 

  • Higher concentration = agreement between measures 

Predictive validity (criterion related)

• Degree of correlation between the measure of the concept and a future measure of the same concept 

• Passage of time mean correlation coefficients are likely to be lower

Construct validity

• Extent to which a test measures a theoretical construct or trait 

• Validity is established when researcher validates body of theory underlying the measurement

Random or chance errors (measurement error)

• Are unpredictable and difficult to control and cannot be corrected 

• Eg. respondents anxiety at time of testing

Systematic error or constant error

• Measurement error that is given to stable characteristics of study population that may bias their behaviour, cause incorrect instrument calibration ot both 

• Eg a scale that gives weight 1 kg less than actual consistently. It is reliable with invalid results

Measure can be reliable but not valid

E.g researcher measuring anxiety by measuring temp; temp is reliable but relation to anxiety is not valid 

A valid instrument is reliable

An instrument that is erratic, inconsistent and inaccurate cannot validity measure the attribute 

Critiquing data quality criteria

• Appropriate method was used to assess the reliability and validity of the tool

• The appropriate method used to test the reliability and validity of the tool

Measurement

Assigning numeric values to qualities (people or objects) to designate the quantity of the attribute

Rigour

determined by measurement instruments that reflect the concepts being tested in a study

Psychometric assessments

• obtain evidence of the quality of instruments and include 2 properties related to the quality of a meadier 

  • Assessment of the reliability and validity of measurement instruments

Reliability coefficient

• Designated as r: ranges from 0-1 

• The closer to 1 the higher the reliability of the measure 

• If over 0.70 or higher is considered acceptable

Cronbach’s alpha test

• Is most common test for reliability 

• Low-reliability core means items on the instrument are not well correlated and, therefore the instrument is unreliable ( which jeopardizes the study)

Reliable not valid

•  not measuring what we want to  measure

• -consistently and systematically  measuring the wrong value for all  respondents

Valid not reliable

measuring what we want to  measure but not all of the time

Not reliable not valid

not measuring what we  want to measure

Reliable and valid

i.e.  measuring what we want to  measure

Levels of measurement

• Help determine the type of statistical analysis to use when analyzing data 

• Higher the level of measurement, the more flexible in choosing statistical procedures and use the highest level measurement obtained from the data 

• Four levels

  • Nominal level

  • Ordinal level

  • Interval level

  • Ratio level

Nominal level (levels of measurement)

• discrete/ categorical exclusive and exhaustive

• #s assigned used as labels -do not have quantitative meaning

• no order to categories

• MCT: mode, bar graph/chart 

• Dichotomous(nominal level): (two true values) 

• Categorical (nominal level): (more than two true values) 

• Mutually exclusive categories used to classify data 

• Example: gender (male & female), ethnicity (caucasian or african american) 

• Researcher assigns meaningless numbers to the data

Ordinal level (levels of measurement)

• Discrete /categorical, mutually exclusive categories- variables are given a number 

• #s assigned not arbitrary 

• categories can be placed in a meaningful numerical order  (rank ordering)     i.e. lowest to highest

• distances between categories are not known or exact

  • example: Olympic medals; Stages of illness

• MCT: mode, median, bar graph/chart 

• Examples: candidates for a job (ranked 1, 2, 3 place) 

Interval level (levels of management)

• Variables are continuous and order ranked, infinite values 

• can be placed in a meaningful numerical order (rank ordering)

• distances between values are numerically equal

• zero is arbitrary  (does not have a true zero)( zero on temperature doesn’t mean absence of temperature)

• therefore, not possible to provide absolute amount of attribute

• often psychosocial / psychological variables

  • example: temperature; test scores; QOL

• MCT: mode, median, mean histogram

• Example: 10 words correct on a test is 5 more than getting 5 words correct 

Ratio level (levels of measurement)

• Variables are continuous i.e infinite # of values possible

• highest level of measurement  

• an be placed in a meaningful numerical order (rank ordering)

• distances between values are numerically equal

• zero is absolute  (true zero) i.e.  indicates absence of the attribute being measured

• Values cannot go below zero

• therefore, possible to provide absolute amount of attribute

• often biophysiologic and physical measures 

  • example: height; weight; BP; pulse; age 

• MCT: mode, median, mean, histogram