(14) TEST ADMINISTRATION, SCORING, INTERPRETATION, & USAGE

Issues in Test Administration - Before the Test

• Selecting tools that are appropriate to use

• Secure tools (must not be known by the test taker in advance)

Protocol

the form or sheet or booklet which a test taker’s responses are entered

Issues in Test Administration - During the Test Administration

Building up rapport

Rapport

working relationship between examiner and examinee

Issues in Test Administration - After the Test Administration

• Safeguarding the protocol

• Writing report

• Scoring and interpretation of the test result

Issues in Test Administration - Examiner and the Subject

Different assessment procedures require different levels of training

Issues in Test Administration - Examiner and the Subject

Examiners should be aware that their rapport with test takers can influence the result

Issues in Test Administration - Examiner and the Subject

The test should be given in the language that the test taker feel is their best

Issues in Test Administration - Examiner and the Subject

The race should also be considered while administering the test

Accommodation

the adaptation of test, procedure, or situation or the substitution of one test for another, to make the assessment more suitable for an assessee with exceptional needs

Alternate Assessment

an evaluative or diagnostic procedure or process that varies from the usual, customary, or standardized way of measurement

Issues in Intelligence Testing - Flynn Effect

is the progressive rise in the intelligence test score that is expected to occur on a normed intelligence test from the date when the test was first normed

Issues in Intelligence Testing - Frog Pond Effect

theory that individuals evaluate themselves as worse when in a group of high-performing individuals

Issues in Intelligence Testing - Bias in Testing

presence of systematic error in the measurement of certain factors

Issues in Intelligence Testing - Culture Free

attempt to eliminate culture so nature can be isolated. without cultural influences and biases

Issues in Intelligence Testing - Culture Fair

minimize the influence of culture with regard to various aspects of the evaluation procedures

Issues in Intelligence Testing - Culture Loading

the extent to which test incorporates the vocabulary, concepts, traditions, knowledge, etc. of particular time

Test Equivalence - Linguistic

wording and content

Test Equivalence - Linguistic

Whether the test has been translated accurately

Test Equivalence - Back Translation

once translated to a certain language, it is translated back to the original language

Test Equivalence - Conceptual

construct has the same meaning

Test Equivalence - Conceptual

requires constructs to have the same meaning in various cultures

Test Equivalence - Metric

same psychometric feature across different groups/culture

Classical Test Theory

a score on an ability is presumed to reflect not only the test taker’s true score on the ability being measured but also the error

Classical Test Theory

Also referred to as true score (or classical) model of measurement

Classical Test Theory

Everyone has a “true score” on a test

Classical Test Theory - True Score

a value that, according to CTT, genuinely reflects an individual’s ability (or trait) level as measured by a particular test

Classical Test Theory - True Score

Its assumptions allow for its application in most situations

Classical Test Theory - Error

the component of observed test score that does not have to do with the test taker’s ability

Classical Test Theory - Error

Errors of measurement are random

True

TRUE OR FALSE. The greater the number of items. The higher the reliability

Classical Test Theory - Error

Factors that contribute to inconsistency: characteristics of the individual, test, or situation, which have nothing to do with the attribute being measured, but still affect the scores

Classical Test Theory - Error Variance

variance from irrelevant random sources

Measurement Error

all of the factors associated with the process of measuring some variables, other than variables being measured

Random Error

source of error in measuring a targeted variable caused by unpredictable fluctuations and inconsistencies of other variables in the measurement process.

Random Error

Type of error that is unavoidable

Systematic Error

Source of error in measuring a variable that is typically constant or proportionate to what is presumed to be the true value of the variable being measured

Systematic Error

Type of error that is avoidable if corrected

Sources of Error - Item Sampling/Content Sampling

variation among items within a test as well as to variation among items between tests

Sources of Error - Item Sampling/Content Sampling

variation among items within a test as well as to variation among items between tests

Sources of Error - Test Environment

room temperature, level of lighting, and the amount of ventilation and noise

Sources of Error - Test taker variable

emotional problems, physical discomfort, lack of sleep, effects of drug, formal learning experiences, casual life experiences, therapy, illness, and changes in mood or mental state

Sources of Error - Examiner-related variables

examiner’s physical appearance and demeanor, nonverbal gestures, and professionalism

Reliability Estimate Error - Time Sampling

he longer the time passes, the greater likelihood that the reliability coefficient would be insignificant

Reliability Estimate Error - Carryover Effects

happened when the test-retest interval is short, wherein the second test is influenced by the first test because they remembered or practiced the previous test

Reliability Estimate Error - Practice Effect

scores on the second session are higher due to their experiences of the first session of testing

Test-retest

_____ with longer interval might be affected of other extreme factors, thus, resulting to low correlation — At least 2 weeks, maximum of 6 months

Parallel Forms/Alternate Forms Reliability

Item sampling (immediate), item sampling changes over time (delayed)

Counterbalancing

technique to avoid carryover effects for parallel forms, by using different sequence for groups

Counterbalancing

Most rigorous and burdensome, since test developers create two forms of the test

Counterbalancing

Main problem: difference between the two test

Counterbalancing

Test scores may be affected by motivation, fatigue, or intervening events

Inter-item Consistency

Item sampling homogeneity

Split-half Reliability

Item sample: nature of split

Inter-rater Reliability

Scorer differences

Standard Error of Measurement

provide an estimate of the amount of error inherent in an observed score of measurement

Standard Error of Measurement

Index of the amount of inconsistency or the amount of the expected error in an individual’s score (true score)

Standard Error of Measurement

Allows to quantify the extent to which a test provide accurate score

FALSE. higher reliability = lower SEM

TRUE OR FALSE. higher reliability = higher SEM

Standard Error of Score

index of the extent to which one individual’s score vary over tests presumed to be parallel

Confidence Interval

a range or or band of the test scores that is likely to contain the true score

Standard Error of the Difference

statistical measure that can aid a test user in determining a large difference should be before it is considered statistically significant

Standard Error of Estimate

refers to the standard error of the difference between predicted and observed values

Hit and Miss Rate - Hit Rate

the proportion of people a test accurately identifies as possessing or exhibiting a particular trait, behavior, characteristics, or attribute

Hit and Miss Rate - Miss Rate

the proportion of people the test fails to identify as having, or not having a particular trait, behavior, characteristics, or attribute

True Positives (Sensitivity)

predict success that does occur

True Negatives (Specificity)

predict failure that does occur

False Positive (Type 1)

success that does not occur

False Negative (Type 2)

predicted failure but succeed

Reactivity

when evaluated, the behaviors increase

Reactivity

Hawthorne Effect

Drift

moving away from what one has learned going to idiosyncratic definitions of behavior

Drift

Subjects should be retained in a point of time

Contrast Effect

cognitive bias that distorts our perception of something when we compare it to something else, by enhancing the difference, decrease one’s performance

Expectancies

tendency for results to be influenced by what test administration expect to find

Rosenthal/Pygmalion Effect

test administrator’s expected results influences the result of the test

Golem Effect

negative expectations decreases one’s performance

Galatea Effect

self-expectations result in higher level of performance

Rating Errors

intentional or unintentional misuse of scale regardless of the performance of the test taker

Leniency Error

the rater is lenient in scoring (generosity error)

Severity Error

rater is too strict in scoring

Central Tendency Error

rater’s rating would tend to cluster in the middle of the rating scale

Halo Effect

tendency to give high score due to failure to discriminate among conceptually distinct and potentially independent aspects of a ratee’s behavior

Horn Effect

opposite of halo effect

True

TRUE OR FALSE. To overcome the rating error is to use rankings

Fundamental Attribution Error

tendency to explain someone’s behavior based on internal factors such personality or disposition, and too underestimate the influence the external factors have on another person’s behavior, blaming it on the situation

Barnum Effect

people tend to accept vague personality descriptions as accurate descriptions of themselves (Aunt Fanny Effect)

Bias

factor inherent in a test that systematically prevents accurate, impartial measurement

Bias

Prejudice, preferential treatment

Estimated True Score Transformation

Prevention of bias during test development

Social Desirability

tendency to say good things about yourself or to mark items that you believe will be approved by the examiner

Social Desirability

Impression Management and Faking Good/Bad

Levels of Interpretation - 1

there is a minimal amount of any sort of interpretation

Levels of Interpretation - 1

• Minimal concern with intervening processes

• Data are primarily treated in a sampling or correlate way

Levels of Interpretation - 1

• No concern in large-scale selection testing

• For psychometric approaches

Levels of Interpretation - 2

Descriptive Generalization

Levels of Interpretation - 2 (Hypothetical Construct)

the assumption of an inner state which goes logically beyond the description of visible behavior

Levels of Interpretation - 3

the effort to develop a coherent and inclusive theory of the individual life or a “working image” of the patient

Levels of Interpretation - 3

The clinician attempts full scale exploration of the individual personality, psychosocial situation, and developmental history

Extra Test Behavior

Observations made by an examiner regarding what the examinee does and how the examinee reacts during the course testing that are indirectly related to the test’s specific content but of possible significance to the interpretation