Research Design

Why Should We Care About Research?

• Foundation of ________

• Informs ________

• Our field is based on ________, compared to those with MBAs

• Relevant to everyday life (we don’t fall for fads)

• Foundation of knowledge

• Informs knowledge

• Our field is based on evidence/science, compared to those with MBAs

• Relevant to everyday life (we don’t fall for fads)

Why Should We Care About Research?

• We are research ________ (creates information) and research ________ (understands research - practioners)

• We are research producers (creates information) and research consumers (understands research - practioners)

What Is The Purpose Of Research?

• _______ between variables (positive, negative, and none)

• Relationships between variables (positive, negative, and no relationship)

What Is The Purpose Of Research?

• Effects: see the _______ of an intervention and causality

• effectiveness

What Is The Purpose Of Research?

• _________: determine if research findings can be applied to numerous settings

• Generalizability

List the Research Claims:

• Frequency claims

• Causal claims

• Association claims

Research Claims:

• Frequency claims:

• Frequency claims: how often things occur (Mode, mean, range, standard deviation)

  • rate at which events occur

Research Claims:

• Causal claims:

• Causal claims: what causes/makes an impact on what (Correlation, ANOVA, t-test, chi-square, regression)

Research Claims:

• Association claims:

• Association claims: relationship between variables (ANOVA, correlation, t-test, chi-square, regression)

Finding & Evaluating Information

• Research vs. Experience

  • _________ feels convincing, but _________ is usually more reliable because it systematically collects evidence.

• Personal experience

• research → Leads to more confidence

Finding & Evaluating Information

• _______: the way our minds trick us into making incorrect decisions

• Cognitive biases

Finding & Evaluating Information

• Cognitive biases: the way our minds trick us into making incorrect decisions

  • List them-

• Good story

• The present/present bias

• Pop up principle/Availability heuristic

• Cherry picking

• Biased questions

• Being overconfident

Finding & Evaluating Information

• Cognitive biases: the way our minds trick us into making incorrect decisions

  • Good story:

• experiences make something appear correct even if it is not true

  • Makes sense that releasing aggression would help

Finding & Evaluating Information

• Cognitive biases: the way our minds trick us into making incorrect decisions

  • The present/present bias:

•  when you don’t remember times when something happens when it doesn’t fit your preconceived notions

  •  Don’t remember the times you didn’t punch a punching bag

Finding & Evaluating Information

• Cognitive biases: the way our minds trick us into making incorrect decisions

  • Pop up principle/Availability heuristic:

• remembering things if they can easily come to mind

  • what easily comes to mind

Finding & Evaluating Information

• Cognitive biases: the way our minds trick us into making incorrect decisions

  • Cherry picking:

• only pay attention to information to information that you already believe

  • seeing what we want to see

Finding & Evaluating Information

• Cognitive biases: the way our minds trick us into making incorrect decisions

  • Biased questions:

• questions affect context of the answer

Finding & Evaluating Information

• Cognitive biases: the way our minds trick us into making incorrect decisions

  • Being overconfident:

• does not mean accuracy

Theory-data cycle:

Scientists collect data to test, change, or update their theories

Theory-data cycle: Scientists collect data to test, change, or update their theories

• _______: theory-data cycle of collecting data of using it to develop, support, or challenge a theory

• Empiricism

Theory-data cycle: Scientists collect data to test, change, or update their theories

• Theory:

• statement(s) that describe general principles about how variables relate to each other

• Theory: statement(s) that describe general principles about how variables relate to each other

  • What makes a good theory?

• Falsifiable

• Supported by data

• Parsimonious (all things are equal)

Theory-data cycle: Scientists collect data to test, change, or update their theories

• __________: research study to test the theory

• Experiment

Theory-data cycle: Scientists collect data to test, change, or update their theories

• Hypothesis:

• prediction about what you’ll find in the experiment

• Basic-Applied Research Cycle

  • Basic-research:

• the goal is to enhance knowledge

• Basic-Applied Research Cycle

  • Applied research:

• The goal is to determine how to solve a problem

• Peer-review cycle:

• Submission -> review -> feedback -> revisions -> review

• Journal to journalism cycle:

• popular press; possible accuracy issues

Observational Research:

watching & systematically recording what they are doing

Observational Research: watching & systematically recording what they are doing

• 2 types:

• Direct

• Indirect

Observational Research: watching & systematically recording what they are doing

• 2 types: Direct-

• observing behavior as it occurs

  • Can be with or without intervention

Observational Research: watching & systematically recording what they are doing

• 2 types: Indirect-

• observing the artifacts of behavior

Observational Research: watching & systematically recording what they are doing

• 2 types: Indirect- observing the artifacts of behavior

  • methods→

• Physical traces: remnants, fragments, and products of past behavior

• Archival records

Observational Research: watching & systematically recording what they are doing

• People: Observing a sample of ______ that represent a population of interest

• people

  • who we want to observe

Observational Research: watching & systematically recording what they are doing

• Behaviors: Observing a representative sample of the population of _______

• behavior

  •  what behavior you want to make sure you see

Observational Research: watching & systematically recording what they are doing

• Time: Observing behavior in a random or systematic sampling of ________

• time intervals

  •  what times you want to sample

Observational Research: watching & systematically recording what they are doing

• Situation: observing behavior in a variety of ______

• situations

  • Different types of environments/activities (i.e. online vs. in-person)

Observational Research: watching & systematically recording what they are doing

• Observation without intervention: Relationships between variables are observed as they _____ occur

  • Helps establish _____ validity

• naturally

  • Helps establish external validity

Observational Research: watching & systematically recording what they are doing

• Observation with intervention: Observer intentionally _______ or introduces variables into setting to measure responses

• manipulates

Observational Research: watching & systematically recording what they are doing

• Recording Behavior: 

  • _______ Records: includes all records→ Faithful reproduction of behavior as it originally occurred (written transcripts, audio & video recordings)

• Comprehensive

Observational Research: watching & systematically recording what they are doing

• Recording Behavior: 

  • ________ Records: only a portion is included

• Selected

Observational Research: watching & systematically recording what they are doing

• Recording Behavior: 

  • ___________: when non-random records are systematically more likely to be created 

• Selective deposit

Observational Research: watching & systematically recording what they are doing

• Recording Behavior: 

  • ________: Some records are more likely to be kept while others are deleted.

• Selective survival

Observational Research: watching & systematically recording what they are doing

• Recording Behavior: 

  • How to helps reduce bias while picking records?

• Randomization

Things To Consider:

• Ethics

• Feasibility

Ethics

List the 2 important ethics concepts:

• The Belmont Report

• 5 General ethical principles:

Ethics

• The Belmont Report

• Respect for persons

• Beneficence

• Justice

Ethics

• The Belmont Report

  • Respect for persons:

• Respect for persons: Informed consent; protection for vulnerable populations

Ethics

• The Belmont Report

  • Beneficence:

• Beneficence: cost-benefit analysis for participants and society (low risk, high benefit is best)

Ethics

• The Belmont Report

  • Justice:

• Justice: How are participants selected

Ethics

• 5 General ethical principles:

1. Beneficence & nonmaleficence

2. Fidelity & responsibility

3. Integrity

4. Justice

5. Respect for people’s rights and dignity

Ethics

• Tuskegee Syphilis Study: ethic violations include ->

• Harmed, not treated respectfully, targeted a disadvantaged group

Ethics

• Milgram’s obedience study: _______ to participants, lasting effects of study

• Stressful

• Mediator: _______ the effect of the IV to the DV

• Transmits

• THROUGH

  • X → M → Y

• Moderator: _______ the relationship between the IV and the DV

• Changes

• DEPENDS

  • X → M → Y

Survey Research

• Survey and poll are ________

• interchangeable

Survey Research

• Survey research is a ______ method of collecting data to gather ______ reported information.

• systematic ( same questions for everyone)

• self-

Survey Research

• Used to measure people’s →

• attitudes, opinions, characteristics, behaviors 

Survey Research

• Do not use it when you need _____ information (people can be ______ & memories can be inaccurate)

• accurate

  • (people can be biased & memories can be inaccurate)

Survey Research → Question Types

• List them:

• Closed-ended format/questions

• Open-ended format/questions

Survey Research → Question Types

• Closed-ended format/questions-

• respondent picks from options

• respondent is given a set of response options to choose from.

Survey Research → Question Types

• Closed-ended format/questions- respondent picks from options

  • _____-style format (can be any range of response numbers and anchors i.e. not often-often)

• Likert

Survey Research → Question Types

• Closed-ended format/questions- respondent picks from options

  • _______- Participants rate between opposite traits. (i.e. neuroticism-emotional stability)

• Semantic Differential

Survey Research → Question Types

• Open-ended format/questions- you can obtain _____ information, but it’s ____ (time, bias) to analyze

• richer/ more

• harder

Survey Research → Survey Validity

List the ways-

• Optimizing

• Satisficing

• Conversational conventions

• Context matters

• Primacy effect

• Recency effect

• Adjacent questions

• Social desirability

• Acquiescence

• Leading questions

• Double-barreled questions

• Double negatives

• Yeah-saying

• Nay-saying

• Fence-sitting

• Faking good or bad

Survey Research → Survey Validity

• Optimizing:

• interpret question and deduce it’s intent; searching memory for relevant information and integrate info into single judgement

• Participant carefully thinks through answers.

Survey Research → Survey Validity

• Satisficing:

• Participant gives easier, less thoughtful answers.

Survey Research → Survey Validity

• Conversational conventions:

• People interpret survey questions based on normal conversation rules.

Survey Research → Survey Validity

• Context matters:

• can provide info into what researcher is interested in

Survey Research → Survey Validity

• Primacy Effects:

• First information has stronger influence.

  • Order effects: changes in a participant’s performance caused by the order in which conditions or tasks are presented.

Survey Research → Survey Validity

• Recency effects:

• Most recent information is remembered best.

  • Order effects: changes in a participant’s performance caused by the order in which conditions or tasks are presented.

Survey Research → Survey Validity

• Adjacent questions:

• Earlier questions shape interpretation of later ones.

Survey Research → Survey Validity

• Social desirability:

• People answer in ways that make them look good.

Survey Research → Survey Validity

• Acquiescence:

• Tendency to agree with statements regardless of content.

Survey Research → Survey Validity

• Leading questions:

• may prime or bias the respondent

Survey Research → Survey Validity

• Double-barreled questions:

• asking two questions in one

  • Ex: “Do you enjoy your professor and textbook?”

  • ~Question order can prime and provide meaning. 

  • Split into 2 questions

Survey Research → Survey Validity

• Double negatives:

• using two or more negative words in a sentence

  • Confusing

Survey Research → Survey Validity

• Yeah-saying (Leniency):

• Agreeing with everything.

  • Response set: when people respond to questions consistently in a certain way

Survey Research → Survey Validity

• Nay-saying (Severity):

• Disagreeing with everything.

  • Response set: when people respond to questions consistently in a certain way

Survey Research → Survey Validity

• Fence-sitting:

• always answering in the middle

  • Response set: when people respond to questions consistently in a certain way

Survey Research → Survey Validity

• Faking good or bad:

• Trying to appear unusually positive or negative.

Survey Research → Survey Validity

• _______ Validity: do the questions you ask accurately measure the construct/variable you’re trying to measure

• Construct Validity

Survey Research → Survey Validity

• Construct Validity: do the questions you ask accurately measure the construct/variable you’re trying to measure

  • Ex: Leader satisfaction: How supportive is your leader? - ____ construct variable vs. Does your leader give good vibes? - ____ construct variable

• good

• poor

Survey Research → Survey Validity

• Construct Validity: do the questions you ask accurately measure the construct/variable you’re trying to measure

  • ______ validity: how well the measure of one variable correlates/relates to a measure of another variable that theoretically it should be related to

• Convergent

Survey Research → Survey Validity

• How does convergent validity tell us if there is construct validity?

  • If your exam grade correlates with something that it should theoretically correlate with then there is probably ______ validity.

    • Ex: Exam grade accurately measures class knowledge

• construct

Survey Research → Survey Validity

• Construct Validity: do the questions you ask accurately measure the construct/variable you’re trying to measure

  • _________ validity: the measure of one variable does not correlate to a measure of another variable 

• Divergent/discriminan

Survey Research → Survey Validity

• _______ Validity: the degree to which the results of a study can generalize (how well they can apply & would be the same in other groups of people, situations, time, etc.)

• External Validity

Survey Research → Survey Validity

• External Validity: the degree to which the results of a study can generalize (how well they can apply & would be the same in other groups of people, situations, time, etc.)

  • Ask how do we know if there is external validity -> there is no survey that has _____ external validity, but some surveys would generalize better than others

• complete

Survey Research → Survey Validity

• External Validity: the degree to which the results of a study can generalize (how well they can apply & would be the same in other groups of people, situations, time, etc.)

  • The makeup of your sample - your results would generalize to ____ demographics

• similar

Survey Research → Survey Validity

• Why are construct & external validity forms of accuracy?

1. Construct: are you accurately measuring what you intend to measure?

2. External: are the results of your study accurate in other settings?

Survey Research → Survey Reliability

• ______ Reliability: the extent to which items within a scale all measure the same thing

• Internal Reliability

Survey Research → Survey Reliability

Why are interrater & internal reliability forms of consistency?

• Interrater: are both raters consistent in their ratings?

• Internal: are the items in your scale consistently measuring the same thing?

Survey Research → Best Practices: To Do

• _________ questions

• Clear, concise, and specific

Survey Research → Best Practices: To Do

• Clear, concise, and specific questions

• _______ periods: give the time period you want respondents to answer about (i.e. “In the last 6 months…” or “How do you feel today?” )

• Reference

Survey Research → Best Practices: To Do

• Clear, concise, and specific questions

• Number of _____ points: number of points does not matter too much, but too little means less detail and too many is overwhelming

• scales

Survey Research → Best Practices: To Do

• Clear, concise, and specific questions

• ______ of items: how many questions to measure one scale? (too few= inaccurate data and too many = overwhelming)

  • Soft recommendation: ______ items

• Number

  • 5-10

Survey Research → Best Practices: To Do

• Clear, concise, and specific questions

  • Survey ______ (i.e. pulse survey)

• length

Survey Research → Best Practices: To Do

• Clear, concise, and specific questions

• _______ checks (i.e. repeat question or mark an answer here)

• Attention

Survey Research → Best Practices: Watch For

• _________ items: Asking about the same content from the opposite direction (i.e. “I always support colleagues” and “I never help at work”)

Reverse-scored

Qualitative Data Analysis

• Data ______: Categorizing information into themes/patterns.

• coding