Research Evaluation SPA4800 quiz 1

What is the best available evidence?

Systematic Reviews or Meta-analyses. To assist in clinical decision-making but do NOT make recommendations for clinical practice

If systematic reviews are not available, use Individual studies but you need to carefully evaluate studies for their quality

What is Science?

Science is an organized system of methods used to produce knowledge about certain phenomena (e.g., natural, social, economic)

Science is organized around falsifying claims

Claim: All apples are red.

The existence of red apples does not prove all apples are red

The existence of a green apple disproves all apples are red

If a claim cannot be falsified, it is generally not a scientific question

What is the scientific method?

A problem-solving approach

An iterative process

Systematic of inquiry

What are the steps to the scientific method?

Identify/define a problem

Develop hypothesis or questions

Plan a set of procedures

Collect data

Analyze the data

Report research findings

Why is the scientific method important?

Clinicians and scientists

Medical and education settings

Programs and services quality

Guide clinical practice (i.e.EBP)

What are the steps of evidence based practice?

Framing the Clinical/Foreground Question (e.g., the PICO question)

Finding the Evidence

Assessing the Evidence

Making the Decision

What is PICO?

Population, Intervention, Comparison, Outcome

To assist in clinical decision-making for a specific situation

To direct the literature search for evidence about a specific issue

What is evidence based practice?

Using the clinical approach that has the best evidence showing it is effective

Decisions and practices grounded in scientific research and data

Critical Appraisal: The evidence is carefully assessed for its quality, relevance, and applicability to specific situations.

Example: when you order an item from Amazon, what do you trust more, a product with 4.3 stars and 925 reviews or a product with 4.3 stars and 7 reviews? This is critical appraisal of evidence

What is ABP? Authority based practice

Decisions and practices are based on statements, opinions, or beliefs not necessarily backed by empirical evidence or systematic research.

Lack of scientific evidence, reliance on beliefs or authority, risk of bias

Example: Practice of bloodletting

How to assess the best evidence?

1.Systematic Review study (qualitative analyses)

-Designed to provide clinical and research guidelines

2.Meta-analyses study (quantitative analysis)

-A statistical method to estimate effect sizes of an individual factor

3.Well-designed randomized controlled study

4.Well-designed controlled study without randomization

5.Well-designed quasi-experimental study

6.Well-designed non-experimental studies, i.e., correlational and case studies

7.Expert committee report, consensus conference, clinical experience of respected authorities

How to decide on the best evidence?

Is the available evidence relevant to your client?

Ex. Many studies, assessments, and protocols may be based on data from monolingual individuals, but your client is bilingual.

Are the guidelines for treatment truly evidence-based?

What is critical thinking?

From making good arguments and avoiding faulty reasoning to navigating sources of information (and mis/disinformation) and discerning science from pseudoscience

What are the characteristics of a critical thinker?

Critical thinker ensures that they have GOOD reasons to support their beliefs

Critical thinker is neither dogmatic nor gullible

Critical thinker avoids truthiness

Critical thinker is skeptical (ordinary, philosophical, and scientific skepticism

Why is critical thinking important?

From making good arguments and avoiding faulty reasoning to navigating sources of information (and mis/disinformation) and discerning science from pseudoscience

What are BAD arguments?

Fallacies are flawed, deceptive, or false arguments that can be proven wrong with reasoning

Types of fallacies:

Strawman, either/or (sometimes called false dichotomy/dilemma, or black or white), hasty generalization, circular reasoning, slippery slope, ad hominem, burden of proof, red herring

What is confirmation bias?

The tendency to search for info that confirms one's hypothesis, rather than looking for info that will disconfirm it

What are the five sources of knowledge?

Authority - People tend to accept ideas that come from authority figures. -> unbiased and have competence in the area??

Tenacity - Information that has been repeated over and over and over again, is often believed to be true (the Illusory Effect)

Intuition - knowledge based on "gut feeling" not based on inference or reasoning

Experience (also referred to as Empiricism) - can be a compelling source of knowledge -> subjective and hard to control; Our perceptions can be influenced by bias, namely Confirmation Bias

Science - a systematic method of reducing bias and increasing the objectivity of the results.

Straw man

•Creates easier target to attack or defend

•"We should spend less on the military."

"Soyou want to leave the country defenseless?"

False Dichotomy

•Assumes there are only two choices

•No middle ground

•Labeling everyone an introvert or extrovert misses the people who are a blend

Hasty Generalization

•Overly broad statements based on an unrepresentative sample

•"My grandfather smoked and he lived until 90 so smoking can't be that bad for you."

Circular Reasoning

•Assumes what the very thing it is trying to prove is true

•"It's time to go to bed. Why? Because it is bedtime."

Slippery SLope

•A claim is rejected because of the belief it will lead to a chain reaction resulting in an extreme and undesirable end.

Ad Hominen

•An attempt to discredit a claim based on personal traits of the person making the claim

Burden of Proof

Shifts the burden of proof from the person making the claim to the person refuting the claim

A claim is assumed correct because it hasn't been shown to be false

red Herring

•A claim that misleads or distracts from the real claim

•Used in mystery stories

•A literary device that might have come from using herring to distract dogs from chasing rabbits

What is psuedoscience?

A questionable alternative approach that has no compelling scientific basis

includes questionable ideas, products, and services

What are characteristcs of psuedoscience?

Authority - Persuasive power of "experts"

Tenacity - long-held beliefs (not facts)

IIntuition and Experience - "just feels right" and is based on "personal experiences"

Do not report evidence-based research findings

What are the criteria for distinguishing science from pseudoscience?

●A healthy debate about the product or therapy

●Quantifiable Data are used

●Valid data are disseminated

●Information is peer-reviewed

●Independent confirmation of findings

●Scientific method is followed

●Results have a theoretical explanation

●Use of historical data

●Results are "Too Good to be True"

●Both Misses and Hits are counted

7 criteria

1.Disconfirming evidence is ignored and practice continues even though the evidence is clear

2.The approach is disconnected from well-established models, theories or paradigms

3.New terms are invented or meanings are redefined in non-standard ways.

4.The only evidence is anecdotal, supported with statements from personal experience

5.Inadequate evidence is accepted - support from case studies, or unscientifically sound approaches

6.The printed materials are not peer reviewed

7.Grandiose outcomes are proclaimed

What motivates people to promote pseudoscience?

1.Financial Incentives

2.Influence/Recognition/Ego

3."Maverick" Identity

4.In-Group Belonging

5.Emotional Protection and Hope

ASHA principles of ethics

1.Researchers hold the welfare of participants to be of paramount importance

2.ASHA members must not permit misrepresentation of professional competency

3.Rules for accurate and honest information about an individual's contributions to research activities

three basic principles: What is participants' right?

informed consent, Participation is voluntary

the nature & purpose of the study, Confidentiality and privacy protection, Right to discontinue participation, Coercion violates the autonomy of participants

What is participants' benefit?

The principle of beneficence, A direct benefit to the participants, Contribution to the good of society, "Maximize possible benefits and minimize possible harms" (National Commission, 1979)

Who is invited to participate in research?

the principle of justice, Inclusion or exclusion of particular groups on scientific basis, No discrimination in the delivery of professional services or in conduct of the research (race, ethnicity, sex, etc)

Institutional Review Boards (IRB)

Evaluate proposed studies before data collection

Includes information about research design, participants, procedures, recruitment, and informed consent

Conduct risk/benefit analysis of the proposed research

Research integrity (e.g., data management practices, mentorship, authorship, intellectual credit, confidentiality and privacy, peer review, etc.)

Confidentiality and Privacy

Providing access to your data after publishing it

Confidentiality of patient information

Disclosure/release of information

Ownership of client records

HIPAA (Health Insurance Portability and Accountability Act)

Establish standards and requirements for electronic transfer of health information

Respect privacy and confidentiality

Prevent denial of health care coverage because of pre-existing conditions

CITI Online Training Course

Required since 2000 for submission of grants or IRB proposals

Passing score is 80% correct; individual quizzes may be retaken

Printable certificate, good for three or four years, is awarded to those who pass

Includes an introduction, history, codes and regulations

Reviews material in the Belmont Report

Basic Research

Also called "fundamental" or "pure" research.

Focused on increasing the overall knowledge base and understanding of fundamental principles and phenomena.

Not driven by the immediate goal of developing a product, treatment, or clinical application.

Seeks to answer the "how" and "why" behind natural processes.

translational research

Often described as "bench-to-bedside"

The process of turning laboratory discoveries and theoretical findings into practical applications

clinical/applied research

Determines the safety and effectiveness of medications, devices, diagnostic products, and therapies intended for human use.

Involve human participants and specific measures of health outcomes

What is qualitative research?

No set hypothesis beforehand & no statistical analysis

Intended to answer the questions of "how" and "why"

Design is naturalistic and emergent

Data=words

What is quantitative research?

data=numbers

Set a hypothesis and try to predict something

Intended to answer the questions of "how many" and "how much"

Design is experimental/nonexperimental

Variables carefully operationalized, larger sample sizes, and the ability to be replicated

What is non-experimental research?

Naturally occurring changes (case studies like Alzheimer's Dementia)

Use objective measures

Cannot lead to cause-effect statements (causal inference)

what is experimental research?

Manipulations to create changes (controlled lab observations)

Manipulate at least one variable(s) to create two or more conditions

Systematic, controlled designs

Intended to examine causality

what makes a study an experiment?

different conditions or experiences

subjects assigned to different groups/conditions

what is true research?

Experimental manipulation & random assignment. Yes/no therapy, participants are randomly assigned to the 2 groups

what is quasi research?

no full experiemnt control,one treatment & one comparison group, but participants are not randomly assigned to the 2 groups

What is non-empirical research?

Using existing information/data

What is research? empirical

gathering new data

Categorial Variables

Nominal

(unordered)

e.g., gender, disease status

Ordinal

(ordered)

Education level (less than high school, high school, college, graduate degree, etc.)

Numerical variables

Discrete

(can be counted)

e.g., # of correct

age in years

Continuous

(decimals, fractions)

e.g., weight=53.2

mean

average

the mode

nominal variables, the most common value)

the median

○the 50th percentile of a data set)

range

minimum to maximum of difference

IQR

IQR; 25th and 75th percentiles or difference

What is not a reason people promote pseudoscience?

skepticism of claims

what is the prinicple of benefits?

maximize possible benefits and minimize possible harms

what is the main issue with the turksegee study?

all of the above, lack of information consent for particpants, lack of beneficnce for particpants, lack of justice in selection of particpants

What is a logical fallacy?

A.Flawed, deceptive, or false arguments used to support a claim that may sound convincing.

What is an example of an ad hominem fallacy?

A."You can't trust anything Senator Smith says about the economy; he's a known liar."

What is confirmation bias?

seek information that confirms our claims.

What is evidence-based practice?

Basing decisions and practices on scientific research and data

Using the clinical approach that has the best evidence showing it is effective

What is an example of authority-based practice?

A.A hospital administrator changes the intake process for stroke patients based on new guidelines showing increased effectiveness of early intervention.

What are the components used to frame a clinical question?

A.Population, Intervention, Comparison, Outcome

Which type of study provides the highest level of evidence in the evidence-based practice framework?Which type of study provides the lowest level of evidence in the evidence-based practice framework?

A.Systematic review; expert committee report

What is missing in the example question?

Does Constraint-induced Aphasia Therapy (intervention) improve aphasia (outcome) in stroke patients (population)?

B. Comparison

What is missing in the example question?

In kindergartners, is individual pull-out treatment more effective than group pull-out treatment for improving articulation disorders?

nothing

What is missing in the example question?

In kindergartners, is individual pull-out treatment more effective than group pull-out treatment for improving articulation disorders?

A. Population

B. Intervention

C. Comparison

D. Outcome

E. Nothing missing

nothing

What is missing in the example question? Does Constraint-induced Aphasia Therapy (intervention) improve aphasia (outcome) in stroke patients (population)?

comparison

Still learning (9)

You've started learning these terms. Keep it up!