Intro to Evaluation Final

Audience of Evaluation Reports

Audience of Evaluation Reports

Insurance

Family

Teacher

Physician

Evaluation Process

1. Screen the client using data from referral, observations, and knowledge of potential issues

2. Begin interview process and occupational profile

3. Interview to assist in choosing FOR

4. Assessment choices and data collection

5. Evaluation report/summary with goals

Bias

Preference given to certain areas related to assessments or certain groups

Examiner/personal bias

Personal beliefs of the evaluator that are conscious or unconscious

Contextual factors

Physical, social, cultural, and temporal that impact/influence clients performance

Choosing a FOR (2 stages)

Screening Stage- Gather data about client

Assessment Stage- Choose assessments consistent with FOR

Top down approach

Look at persons occupations first

Botton up approach

Look at components of a task interfering with performance first

An occupational therapist is working with a client with low endurance and she is concerned that he may not be able to complete the long version of an assessment.  A short version of the test is available.  Which type of reliability information would be most important to validate the decision of the therapist to administer the shorter version?

Split-half reliability (Think split into a shorter piece)

This type of validity refers to the degree to which the assessment items represent the construct being measured.

Content (fits the content of the construct)

_____ is the ability of a test to identify or detect those who do not have a condition while _____ is the ability of the test to correctly identify people with the condition.

Specificity, Sensitivity

Which of the following is provided by a standardized test?

Communication

All of these are provided by a standardized test

Objectivity

Quantification

All of these are provided by a standardized test

Researchers were using a polygraph machine to screen for dishonest responses on a questionnaire.  After many administrations of the polygraph test, they begin to worry that the machine is actually measuring anxiety and NOT dishonest responses.  If this is true, it would be an example of:

Low construct-related evidence of validity (not valid because constructed poorly)

A positive correlation between two measures suggests that:

If the client scores well on one test, he is likely to score well on another

Which of the following is not characteristic of standardized assessments?

Specific scoring protocol

Typically have scaled scores ranging from 0 to 100, with a mean of 50

Uniform procedure for administration

Supporting reliability and validity studies

Typically have scaled scores ranging from 0 to 100, with a mean of 50

Which of the following is a characteristic of non-standardized tests?

Likely to take place in a more controlled setting

Specific administration procedures to be adhered to

Likely to take place in a naturalistic setting

Produce primarily quantitative data about the client's performance

Likely to take place in a naturalistic setting

This type of reliability is concerned with the degree of agreement between items on an assessment that measure similar concepts or skills.

Internal Consistency (consistent among concepts or skills)

Which of the following is NOT an indicator of a test’s validity?

Ability of the test to discriminate between typical and clinical populations

Concurrence of results with tests evaluating similar constructs

Agreement of results over multiple test administrations

Accuracy of the test in predicting performance in other tasks requiring similar skills/abilities

Agreement of results over multiple test administrations

____________________________ refers to the extent to which there is agreement between scores on one measure with those on the existing ‘gold standard’ measure

Concurrent validity

An occupational therapist is working with a client who is progressing rapidly and achieving many of her goals based on the initial evaluation and subsequent treatment plan. The OT now wants to re-evaluate the client’s cognitive skills but is concerned the client will remember the assessment items since it was first administered only 4 weeks ago. Which type of assessment reliability information would be most important for the therapist to attend to at this time?

Parallel forms reliability (parallel assessment items)

There are random, unpredictable influences on test scores, resulting in some amount of measurement error. When performance on an assessment is measured at initial evaluation and again, after the client has been participating in treatment for some time, it is important to ascertain change due to the random error vs. that due to the therapeutic intervention. Measurement error aside, the ______ is the change that the assessment can detect while the _____ is the change that is functionally meaningful for the client.

Minimal detectable change, Minimal clinically important difference (key words change and important)

You have just been assigned a new client in the hand therapy clinic. She is a seamstress in her late 40s who recently had surgery on her dominant hand for carpal tunnel syndrome (compression of the median nerve that causes numbness, tingling, and weakness in the hand). You would like to identify an assessment that will help you determine what tasks she will likely be able to complete when she returns to work. What type of validity would you be most interested in when selecting an appropriate assessment?

Predictive validity (predict what will happen when she returns to work)

Which of the following is an incorrect pairing between assessment type and reference/comparator?

Normative assessment – specific sample of people

Criterion-referenced assessment – established scores/levels of mastery

None of these are incorrect (that is, they are all correct pairings)

Ipsative assessment – no reference (this is strictly descriptive)

Ipsative assessment- no reference (this is strictly descriptive)

In your first OT job, you use a self-reported depression inventory assessment for one of your clients who exhibits low motivation following a stroke. You learned about the assessment during you level II fieldwork, where it was used by many of the therapists there. While you are documenting your evaluation notes and preparing your treatment plan, you realize (through some literature searches) that the tool has low psychometric integrity. What is the best way to proceed?

Report the results, noting the limitations

During your fieldwork at a hand therapy clinic, you are asked to assist your supervisor with a research project. He has data from several patients who completed both the DASH assessment (Disabilities of the Arms, Shoulder, and Hand) and the SLUMS (Saint Louis University Mental Status) exam. You notice that the scores between the two measures have a low correlation. What do you attribute this to?

Divergent Validity

In the context of psychometrics, responsiveness refers to…

Ability of scores on an instrument to detect change due to intervention (respond to change)

This is the scoring tool utilized for the DASH a rating scale you will complete in the upcoming weeks. For this assessment, clients are asked to rate their ability to perform several activities using the following scale:

No Difficult- 1

Mild Difficulty- 2

Moderate Difficulty- 3

Severe Difficulty- 4

Unable- 5

Ordinal

Which of the following is a measure of reliability of an obtained test score and estimates the possible deviation of a test taker's observed score from his true score?

Split-half reliability

Internal consistency

Test-retest reliability

Standard error of measurement

Standard error of measurement

Swelling Measurements

Volumetric Displacement

Circumferential Measurement

Classification of Edema

Observation of Skin (shiny, taut)

Palpation (pitting edema, brawny edema)

Carpal tunnel tests

Tinel’s test

Phalen’s test

Reverse Phalen’s Test

Cubital tunnel test

Froment’s sign

Wartenberg’s sign

Elbow flexion test

Radial tunnel tests

Resisted long finger extension

Resisted supination

Resisted wrist extension

Rule of 9

Sensation Tests

2-Point Discrimination

Semmes-Weinstein Monofilaments

Strength Tests

Dynamometer

Pinchmeter

Standardized Assessments

Nine Hole Peg Test

Jebsen-Taylor Hand Function Test

Minnesota Rate of Manipulation Test

Purdue Pegboard Test

Box and Blocks

How does a norm-referenced test differ from a criterion-referenced test?

Comparison of scores with a population or representative sample

Which of the following is a measure of central tendency?

Reliability Coefficient

Variance

Standard Deviation

Mean

Mean

A measurement scale that has a standard unit of measure that allows for estimating the magnitude of difference between responses is

An interval scale (difference=intervals or like levels)

Likert scales and semantic differential scales result in which type of data?

Ordinal

This measure of central tendency evenly divides the population in half, with 50% above and 50% below

Median

When using standardized assessments, testing bias may result from

Person, environment, and item

In scoring a client’s standardized test, you calculate his Z-score as -1.4. This tells you that his score is…

Below the mean

A well-constructed assessment item results in everyone scoring the same on that item.

False

When reporting scores on a standardized assessment, it is better to report the age or grade equivalent over a percentile or rank measure.

False

Which of the following is true about the DASH?

Items may be read aloud to the patient

It takes approximately 30 minutes to administer

All items must be completed in order to score it

If the patient does not understand an item, provide an explanation to ensure it is interpreted correctly

Lower scores indicate more disability

Items may be read aloud to the patient

Nominal Scale

identifies a person as having or not having a particular characteristic; category, name, label (e.g., male/female, pain/no pain)

Ordinal Scale

represents a rank ordering and provides information about relative position; rank, greater-lesser, relative (e.g., tallest to smallest; gold, sliver, bronze medals)

Interval Scale

has a standard unit of measure that allows for estimating the magnitude of difference between responses; the distance between the points/units is equal (e.g., body temp scale in F or C; dynamometer grip)

Ratio Scale

includes a true zero, which represents an absence of something (e.g., height, weight, range of motion)