Principles of Assessment Midterm Review

Screening

Brief snapshot of a client that determines if there is a need for a full evaluation

Types of Screening

brief observation, chart review, standardized screening tools, short interview

Evaluation

Comprehensive overview of a client that includes obtaining data to identify strengths/limitations, set goals, and plan intervention

Re-evaluation

Comprehensive overview of a client as some point after initial evaluation to track client progress

2 main components of evaluation

1. Occupational profile

2. Analysis of occupational performance

Assessment

Specific tool, interaction, observation, or systematic protocol that gives us data

4 types of assessment

1. Interviewing

2. Questionnaires

3. Skilled Observation

4. Performance-based

Re-assessment

Review of client performance any time other than the initial evaluation

Professional Reasoning

A whole body process that includes our senses, metacognition, ways of thinking, and everything that goes into making care decisions.

Characteristics of Professional Reasoning

Always in use, holistic, critical, context-dependent, evolving, fast, client-centered, complex

3 things that are considered in professional reasoning

1. Client preferences

2. Theory & Evidence

3. Personal & Professional Experiences

Scientific

Reasoning that involves the use of theory, evidence, and an understanding of the client’s condition.

Diagnostic

Reasoning that occurs when we attempt to detect the cause of occupational performance challenges and test our hypotheses about a client

Procedural

Reasoning that involves following steps, protocols, processes, guidelines, and forming habits.

Narrative

Reasoning that includes getting a client’s story to understand their occupational history and profile. Often used when interviewing

Pragmatic

Reasoning that considers logistics, contexts, and limitations such as time, cost, resources

Ethical

Reasoning that seeks to identify what should be done and considers moral dilemmas

Interactive

Reasoning that is the hallmark of face-to-face client time, involving rapport/trust, relationship building, and empathy.

Referral

written order for a specific service that often triggers the start of OT

Other names for a referral

script, consultation, order

3 places referral requirements/guidelines come from

1. State

2. Insurance

3. Setting

Direct Access

Allows OTPs to evaluate and treat patients without a physician’s prescription

Purpose of screening

Saves time, money, and resources and increases efficiency by ensuring that the right clients get evaluations. Also helps inform your evaluation

Characteristics of screening

Short (<15 mins), brief, fast, broad/general, not billable, non comprehensive, high sensitivity, low specificity

A client should be evaluated when:

A screening reveals challenges in occupational performance (not just based off a diagnosis)

Top-Down Approach

Assessing occupational performance, identity, and meaning first and then considering performance skills/deficits.

Bottom-up Approach

Considering performance skills/deficits first, and then moving to look at occupational performance.

Advantages of a Top-Down Approach

Client-centered, occupation-based, central and unique to OT’s professional philosophy, more modern, more holistic, increased client motivation and satisfaction.

Advantages of a Bottom-Up Approach

Very valid and reliable, more medical/historical, easily understood by others, can be more collaborative, helps to find the root causes of dysfunction and see small improvement

Disadvantages of a Top-Down Approach

More difficult to measure and show progress, more subjective, assessments typically require more training

Disadvantages of a Bottom-Up Approach

Increased performance skills do not equal improved occupational performance, miss potential deficits, too reductionist, often contrived/controlled environment

Advantages of Theory use in the evaluation process

Guides professional reasoning, provides direction, helps you explain the OT process, keeps us client-centered and evidence-based, help motivate clients

Occupational Behavior focus of concern

Preventing and reducing disruptions/incapacities in occupational behavior that result from injury/illness. Health and wellbeing is represented by a balance of occupational behavior in self-care, work, and play/leisure.

Model of Human Occupation focus of concern

Focus on the PERSON, especially their sources of motivation, patterns of behavior, roles, habits, routines, and performance capacities

Occupational adaptation focus of concern

framework that describes a normal human phenomenon called adaptation and its impact on the interactive process between a person and their environment.

Ecology of Human Performance focus of concern

the role of CONTEXT in task performance within the areas of ADLs, work and productivity, education, leisure/play, and social participation

Person-Environment-Occupation-Performanace

the interdependent relationship among person, OCCUPATIONS (valued roles, tasks, and activities) and performance. These occupations in turn influence one’s life roles

Construct

ability assessed by a test

Test-retest reliability

Same score across tests at different points in time

Validity

Accuracy with which the test assesses what it is supposed to

Reliability

Consistency/reproducibility of test scores (same every time)

Inter-rater reliability

same score across different test givers when administering the same assessment

Intra-rater reliability

Same scores when the same person does the test multiple times

.90-.99 on the reliability scale

High and preferred, not common

.80-.89 on the reliability scale

satisfactory/adequate

.70-.79 on the reliability scale

weak or minimally accepted

Below .70 on the reliability scale

inadequate/unacceptable, caution with using

What can challenge reliability?

Not adhering to the methods/going off script, human error, context, rater bias, client bias such as practice effects, growth/aging

Unified view of validity

All forms of validity fall under the term construct validity

Face Validity

The extent to which an assessment appears to measure the construct

Issue with face validity

It cannot be empirically tested and is subjective, so it is not a significant form of validity

Content Validity

The extent to which the domain of interest is comprehensively sampled by the items on the assessment (includes all relevant components)

Criterion Validity

Extent to which the assessment produces similar scores to an agreed upon gold standard assessment

Predictive Validity

Extent to which scores predict future abilities or outcomes

Construct validity

Extent to which an assessment reflects the underlying theory or relationships of a construct?

Fairness in Testing

All test takers should have unobstructed opportunity to demonstrate their understanding/performance of the construct being measured (made possible by reasonable accomodations)

Challenges to fairness in testing

assumption of previous knowledge, cultural differences, time constraints, environmental constraints or distractions, language barriers, motor unfairness

Responsiveness

The ability of the instrument to detect change over time

Minimal Detectable Change (MDC)

The smallest amount of change that is real change, and not random error/noise

Minimally Clinically Important Difference

The smallest change in scores that results in substantial, meaningful gains.

Ceiling Effect

When an assessment fails to measure differences in performance at the top of the scale (higher performers all look the same)

Floor effect

When an assessment fails to measure differences in performance at the bottom of the scale (poor performers all look the same)

Specificity

Ability of the instrument to correctly identify those that DO NOT have the condition or impairment

Sensitivity

Ability of the instrument to correctly identify people WITH the condition or impairment with high accuracy

Lots of true negatives

High specificity

Lots of true positives

High sensitivity

Clinical utility

How feasible an assessment is to use in real practice (pragmatics, client factors, context)

Standardized Assessments

Instruments with standard instructions/administration procedures, scoring, and established psychometric properties

Characteristics of Standardized Assessments

structured, formal, prescribed, objective, evidence-based, quantitative, time consuming, expensive, sometimes required, compared to norm/criterion data

Test Booklet

Contains information about a standardized assessment’s psychometric properties, administration procedures, scoring, etc

2 categories of standardized assessments

1. Performance-based

2. Questionnaire

2 categories of Non-Standardized assessments

1. Skilled observation

2. Interview

Advantages of standardized assessments

Backed by evidence, credibility, shows progress clearly, objective, useful in research, sometimes required, rigorous testing, reliable/valid, quantifiable, straight-forward, easy to interpret

Disadvantages of standardized assessments

strict guidelines/protocols, limited environment, less client-centered, can only be used with specific populations, costly, time-intensive, may require training, more bottom-up

Advantages of non-standardized assessments

Use OT expertise, more client-centered, adaptable, low/no cost, no training needed, intrapersonal validity, can be done in natural environment, highly sensitive, more top-down, can assess many aspects at once

Disadvantages of non-standardized assessments

subjective, difficult to show clear progress, not backed by research, unknown reliability/validity, requires more professional reasoning,

Performance-Based assessments

standardized tool that enables observation and assessment, clients are asked to perform a series of set tasks

Norm-referenced assessments

Compares data to a specific sample of people with similar characteristics, often scoring uses a bell curve

Criterion-referenced assessments

Compares client performance to a standard of performance/set crtieria, reflects mastery of a specific task

Ipsative-referenced assessments

Instruments that have standardized procedures BUT are individualized so that people can be compared back to themselves over time (mix)

Non-standardized assessments

Tools that have no standard administration procedures or study of measurement properties

Characteristics of non-standardized assessments

adaptable/flexible, broad, client-centered, highly sensitive, qualitative, subjective, requires less resources but more professional reasoning, often done in natural environment

intrapersonal

within person comparison

Skilled observation

directly perceiving the actions of a client through systematic examination

4 Data collection methods for skilled observation

1. Event

2. Duration

3. Rate recording

4. Time sampling

Event recording

how many times an event occurs (frequency)

Duration recording

How long something lasts or how much time is required for a task

Rate recording

How many instances of an event in a given time period

Time sampling

Periods of observation and recording, dispersed in periods of no recording

Times you would use a standardized over a non-standardized assessment

when required by insurance/setting, when you want to compare the client to the normal population, when you want to document clear progress or make predictions, when you want to inform big decisions like discharge, or when a client is motivated by seeing their data/scores

Times you would use a non-standardized over a standardized assessment

When you want to assess something personalized (such as morning routine), when you want to observe someone in their natural environment, when standardized assessment don’t fit your client, when we need more flexibility and don’t know what to expect, and when the client cannot follow specific directions or has lines/leads/medical equipment that would get in the way.

Purpose of evaluation

Establishing baseline data, facilitating development of goals, fostering rapport, and driving intervention planning/recommendations

Occupational Profile

Summary of a client’s occupational history and experiences, patterns, interests, values, needs, and contexts

Analysis of occupational performance

Client’s assets and limitations or potential problems are more specifically determined through assessment tools designed to analyze, measure, and inquire about factors that support or hinder occupational performance.

Interview

Process of inquiry during which one person asks another one or more questions, prompts, or probes

Proxy

someone who can answer on a person’s behalf

Advantages of an interview

Stay client-centered, holistic, builds rapport, flexible, can clarify

Disadvantages of an interview

relies on client-report, client may not want to answer questions, requires professional reasoning

Challenges to obtaining the occupational profile

Cognitive deficits, aphasia, lack of rapport, lack of medical knowledge, time, age, heath illiteracy, lack of a proxy/permission, language barriers, medication side effects, family influences

Motivational Interviewing

Collaborative conversation style for strengthening a person’s own motivation and commitment to change