Research Analysis and Design

0.0(0)
Studied by 3 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/94

encourage image

There's no tags or description

Looks like no tags are added yet.

95 Terms

1
New cards

Produce answers

If you do research (as a scientist) or part of your career

2
New cards

Consume answers

Sometimes very important answers that are very real consequences for decisions you make

3
New cards

Descriptive questions

  • You use empirical methods to describe and explain

  • Definition: claim about a state of the world


4
New cards

Theory

  • You use empirical methods to describe and explain

  • Definition: claim about how and why things are related


5
New cards

Basic (reason for asking)

asking for the sake of knowing, for the sake of generating knowledge

  • What are the basic patters of human behavior, thought and feeling (personality) over time?

  • What are the causes of psychopathology?


6
New cards

Applied

asking for the sake of solving a practical problem

  • How can people improve their well being?

  • How can we improve student learning?

  • How can we treat anxiety disorder?


7
New cards

Basic vs Applied research

Fuzzy boundary btw the two

  • basic often ends up informing application

  • Applied research on one particular problem can lead to questions about how things work more generally, which leads to basic questions and basic knowledge


8
New cards

Quantitative Methods

turn empirical observations into numbers

9
New cards

Qualitative Methods

Create rich descriptions

10
New cards

Empiricism

Evidence acquired from systematic, objective unbiased observation, independently verifiable by other observers or scientists (practice of gathering information through direct observation)

  • the belief that all knowledge, behavior, and mental processes come from sensory experience and environmental interaction rather than from innate ideas

  • Empiricist (scientist)= show me the data


11
New cards

Non-scientific evidence

  • Intuition or common sense

  • own personal experience

  • anecdotes or testimonials (experiences of others)

  • Authority or convention (base advice on intuition, convention)


12
New cards

4 problems with non-scientific answers

  • Bias (intentional or unintentional)

  • Lack of valid comparison

  • Problems with casual interference

  • Lack of generalizability


13
New cards

Bias (problem w/ non scientific answers)

  • Bias of others, trying to sell of persuade (intentional)

  • Bias in your own memory and interpretation of your own experience (unintentional)


14
New cards

Availability heuristic

relying on information that comes to mind quickly

15
New cards

Confirmation bias

Tendency to search for, interpret, and recall information that confirms preexisting beliefs

  • Safety of vaccinations

  • Evidence of human cased climate change


16
New cards

Lack of valid comparison (4 problems with non-scientific answers)

  • (non-scientific answer compared to what?)

  • No control group


17
New cards

Problems with causal inference (4 problems with non-scientific answers)

Other things can correlate with this happening

  • Ex: I learned more when I got coffee before class

  • Confounds: less stress, more sleep, may correlate with with time to coffee (less stressed if have time to get coffee?)

  • Placebo effect: expected that coffee will improve learning

  • Bad at figuring an actual outcome


18
New cards

Frequency

How often does X happen?

  • Focuses on one variable

    • How many people smoke?

    • How many people text while driving?

    • How common is voter fraud?


19
New cards

Association

Is X related to Y?

  • Is course performance related to attendance?

  • Is time in day care related to behavioral problems?

  • NOT CASUAL question/claim, just whether two or more variables related


20
New cards

Causality

  • Does X cause Y (Descriptive)

    • Does caffeine improve memory?

  • Why/how does X cause Y (explanatory)

    • Why does caffeine improve memory?


21
New cards

Variables

A factor or dimension that can take more than one value (on which people can vary)

  • any characteristic, quality, or condition that can change or take on different values across people, situations, or time

  • In the questions and claims we talked about, X and Y are placeholders for variables

    • How does X happen? (how many students will study)

    • Is X relative to Y? (gender and fear of horror of films)

    • Does X cause Y? (personality and aggression)

    • How does X cause Y? (how many time of day affect mood)


22
New cards

Conceptual (variables)

The abstract concept

  • It explains the meaning of a variable/term that you can’t measure

  • ex: Height, intelligence, anxiety


23
New cards

Operational definition

  • How the concepts gets cashed out in terms of something concrete that can be observed

  • turn an abstract, general idea into a clear, measurable, and concrete action or value so it can be tested in a research study

    • Easy sometimes (height=centimeters)

    • Sometimes tricky

      • anxiety (I cant see, hear, or feel from your anxiety… What can I see)

      • learning (I cant see your learning or mine.. what can I seee)


24
New cards

Measured variable

Naturally occurring

  • the data or outcome that a researcher records, observes, or tests in an experiment

    • Study time

    • sadness


25
New cards

Manipulated variable

Researcher sets up condition to produce it

  • Study Time

  • Sadness


26
New cards

Discrete value

countable values

  • Gender (many studies compare men to women)

  • Smoking (smoker, non-smoker)

  • Yes or no, this or that


27
New cards

Continuous

views human growth and change as a gradual, cumulative, and smooth process rather than a series of abrupt, distinct stages

  • Response time variable: 1.07 seconds, 2.53 secs, 5.25 secs

  • Smoking variable: 1,5, 10 cigarettes per day

  • range


28
New cards

Constant

A factor or dimension that could vary but doesn’t in a particular study

  • Participants are all the same age or gender

  • All participants are given the same amount of time for a task

  • Sample of participant all diagnosed with ADHD


29
New cards

Self-report

A measure in which participants provide responses to questions about themselves

30
New cards

Informant-report

Reporting beliefs about what someone else knows or does

31
New cards

Observational/behavioral

A measure in which the researcher records observable behaviors, including behavioral indicators of internal states

  • Observe what people do


32
New cards

Physiological measures

measure of a biological state

33
New cards

Converging evidence

When different measures provide outcomes pointing to the same conclusion

34
New cards

Reliability

The consistency of the measure

  • bathroom scale (can we trust it?)


35
New cards

Three types of reliability

test retest

interrater

internal

36
New cards

Test retest reliability

similar scores when the measure is repeated

  • height of an adult measured by tape measure

  • Use scatterplot to evaluate relaibility


37
New cards

Interrater reliability

different observers record similar result

  • short answer test responses scored by same two people

  • number of people on cell phones


38
New cards

Internal reliability

different questions/ parts of a measure produce similar results

  • People who are more likely to agree with any one item are more likely to agree with all items


39
New cards

Why isn’t reliability perfect?

  • Observed score (we can never know true score, just measured score)

  • Measurement error


40
New cards

Observed score

Observed score= signal+ measurement error

41
New cards

Measurement error

any factor that distorts the observed score for reasons unrelated to the construct

ex: construct=learning

ex: operationalization = exam score

42
New cards

How to improve test-retest reliability

  • Standardize the administration procedure

  • Control situational factors (lighting, temp)

  • minimize fluctuation in participants' transient states (mood, fatigue)

    • ex: administer at same time


43
New cards

How to improve internal relaibility

  • Carefully examine items

    • ex: depression (sad, feel sad, cry, irritable, worry)

  • Give clear instructions to participants

  • Make task length manageable

  • Minimize distraction


44
New cards

Improve interrater reliability

  • Develop good scoring key

  • Train the raters: teach evaluators how to assess performance, behavior, or data accurately while reducing personal bias and subjective errors

  • Check for coding errors


45
New cards

Construct validity

  • Does the assessment measure what it’s supposed to measure?

  • Does it predict what it is supposed to predict?


46
New cards

Face validity

Does the measure resemble what it’s supposed to measure?

  • judged based on own theory of construct

  • Note: this is subjective criteria

    • Ex: How adventurous are you?

    • Construct (abstract concept used to explain human emotions)= Extraversion (high energy, wanna interact with external world)

Sometimes, valid items do not have face validity

  • Ex: Do you prefer showers or baths? (low face validity, used to measure empathy, able to use measure cause people who seem to pick showers are more empathetic)


47
New cards

Concurrent Evidence

What should the measure relate to right now?

48
New cards

Predictive Evidence

What should the measure predict later?

49
New cards

Convergent Evidence

What similar measures agree with it?

50
New cards

Discriminant Evidence

Is it distinguishable from different constructs?

51
New cards

Concurrent validity

measure related to an outcome or behavior that it should be related to, that is measured at the same time

  • shows how well the results of a new test or measurement agree with an established, trusted benchmark (a "gold standard") when both are administered at the same time

  • ex: blood pressure measure by doctor, while blood pressure ring also measuring blood

  • ex: motivation measured NOW —> current GPA


52
New cards

Predictive Validity

Is a measure related to an outcome or behavior that it should be related to, that is measured in the future?

  • ex: motivation measure NOW —> GPA at end of Year


53
New cards

Similarities between concurrent validity vs. predictive validity

Both collect measures of X NOW

  • collect measure of outcome either NOW or in FUTURE

    • timing of outcome based on when you measure it, NOT when it happens

Both established via

  • Correlations (across participants)

  • Differences in means via known groups

  • differences in means via manipulated varaible


54
New cards

Convergent validity

Is the measure strongly related to other measures of the same (or similar) constructs?

  • similar measures


55
New cards

How is convergent validity different than “outcome” in concurrent/predictive validity?

  • Another measure vs. the outcome of a measure

  • Sometimes fuzzy boundaries between measures and outcomes


56
New cards

Discriminant validity

Is the measure weakly related (or not at all) with measures of a different construct?

  • ex: Extraversion -> Eye color (not compelling)


57
New cards

Internal validity

  • Can this study support a causal conclusion?

    • Does X and only X produce the observed change?

  • the degree of confidence that a study's results truly show a cause-and-effect relationship between the independent variable and the dependent variable, rather than being the result of other unintended factors

  • Experiments have strong internal validity


58
New cards

External validity

  • The degree to which the results generalize to people and contexts besides those in the study (generalizability)

  • Does the claim generalize across the dimensions of who, what, where, and

    when?

  • E.g., Claim: “Caffeine improves memory”

    • Measure: cued recall test (good construct validity for memory)

    • Study: College students, word pairs, 20-minute delay

    Would it generalize to...

    • older adults?

    • children?

    • face recognition?

    • eyewitness memory?

    • exams?

    • two weeks later?


59
New cards

How to create good self reported measure? (survey methods

  • Good = reliable and valid: influenced by how you create and how you administer


60
New cards

How do you ask the question (survey methods)?

  • Understandable: construct validity compromised if people don’t understand the question

  • Simple questions (why?)

    • misunderstanding lowers construct validity

    • technical wording increases error

    • literacy levels vary

  • Overall: Clear wording is not “dumbing down.” It is better measurement.

  • Use unbiased questions


61
New cards

Framing effect

  • The frame in which objective information is

    communicated can change how people respond


62
New cards

Common kinds of frames

Gain/loss (promotion/prevention)

  • What is the likelihood you’ll remember? What is the likelihood you’ll forget?

  • E.g., health messages, flossing, etc.

Anchor points

  • People rely to heavily on the first info they are receiving

    • Out of 1,000 people, how many experience spiritual

      struggles?


63
New cards

What is problem with order of questions (Framing)

Order of questions can bias subsequent responses

  • Earlier questions can change later answers

  • ex: Forced choice items followed by open-ended items

    • What was your most significant spiritual struggle: Moral or Ultimate Meaning?

    • Please describe your most significant spiritual struggle:

  • Order of response options can bias responses

    • Each year, do more people join religion or leave religion?


64
New cards

Solution for ordering and framing effect

Frame or order questions/options more than one way and see if the outcomes are the same

65
New cards

Pros and cons: open ended questions

  • Pros: more information, richer information, not constrained by researcher response options

  • Cons: require coding, length/detail desired may not be clear to participant, recall memory is worse than recognition memory

    • List your favorite books (requires recall) vs. choose your favorite books from this list (recognition)


66
New cards

Pros and con: close ended

Closed-ended:

  • Pros: Clearer response options, quicker to complete and score

  • Cons: May not include all options participants would report, may be influenced by framing effects


67
New cards

Observation focuses on

Systematic recording of observable behavior

  • what people do — not only what they say they do

  • performance in tasks or tests — often speed or accuracy

  • behavioral traces or outcomes — products of behavior

    • ex: Helping, texting, blushing, arguing, hand-washing, littering, free throws,

      reaction time


68
New cards

How to Collect Observational Measures

Three design questions

  • What behavior can be observed?

  • Where will it be observed?

  • How might observation influence behavior or recording?

Bottom line: Observation is not automatically objective. It has to be designed

69
New cards

Naturalistic Observation (what is being observed)

Observe and record behavior in its natural environment without intrusion.

  • Examples

    • How do children form cliques on a playground?

    • Is cell-phone use more common on highways or city streets?

    • Who washes hands more after using the restroom?

  • Advantage: behavior observed in an environment it occurs in


70
New cards

Laboratory Observation (what is being observed)

Observe and record behavior in a controlled environment.

  • The researcher controls more of the situation:

    • who participates

    • what task they complete

    • what is recorded

    • when and how often behavior occurs

  • Key tradeoff

    • More control, but potentially less natural behavior.


71
New cards

Lab and naturalistic observation combined

Field experiments create a contrived situation in the real world.

  • Examples

    • Helping behavior after staged subway accidents

    • Hospital hand-washing reminders tested in real settings

    • Public behavior after signs, prompts, or environmental changes

  • Key idea:

    • Field studies combine ecological realism with some experimental control.


72
New cards

Pros and cons of Naturalistic observation

  • Behavior in the wild

  • Less artificial

  • Less control over who, what, and when


73
New cards

Pros and cons of Laboratory observation

  • More control over task, timing, and recording

  • More options for cameras, physiology, reaction time, and coding

  • May alter real behavior


74
New cards

How observation influence data

  • Observer bias

  • observer effects


75
New cards

Observer bias (how observation influence data)

  • Expectations influence what observers notice, record, or interpret.


76
New cards

Observer effects (how observation influence data)

  • Participants may behave differently because they are being watched.


77
New cards

Solutions to reducing Observer Bias

Possible solutions

  • Use blind observers or coders.

  • Develop clear coding rubrics.

  • Use more than one observer.

  • Check interrater reliability.

  • Video record behavior and code later.

Takeaway

  • Objectivity is designed into the method; it does not happen automatically.


78
New cards

Reactivity

Reactivity occurs when participants change behavior because they know they are

being observed.

79
New cards

Solutions to reducing reactivity

Unobtrusive observation

  • One-way mirrors or hidden recording when ethical

  • Habituation: wait until people adjust

  • Measure behavioral traces instead of observing directly

  • ex: Litter in public spaces, beer bottles in trash, parenting in lab settings


80
New cards

Bivariate correlation studies

Examine the relationship between two measured variables in the same participants.

  • measured not manipulated


81
New cards

Multivariate analysis

several variables

82
New cards

Methods for studying association questions

  • Bivariate correlation: two variables

  • Multivariate analysis: several variables


83
New cards

Basic design for bivariate coorelation

  • One group of participants

  • Measure X

  • Measure Y

  • Estimate their association


84
New cards

Two parts of correlation

Strength and direction (pos, neg and no association)

85
New cards

Positive association (Direction of an Association)

  • Higher values of one variable tend to occur with higher values of the other.

  • Effective study strategies ↔ learning

  • Social support ↔ life satisfaction


Important: direction and strength are independent

• -1 is as strong as 1 (sign = direction; absolute value = strength)

• 0 means no linear relationship (a curved association may remain)

86
New cards

Negative association (direction of an association)

  • Higher values of one variable tend to occur with lower values of the other.

  • Absenteeism ↔ grades

  • Stress ↔ sleep quality


Important: direction and strength are independent

• -1 is as strong as 1 (sign = direction; absolute value = strength)

• 0 means no linear relationship (a curved association may remain)


87
New cards

No association (direction of an association)

Knowing one variable does not help predict the other.


Important: direction and strength are independent

• -1 is as strong as 1 (sign = direction; absolute value = strength)

• 0 means no linear relationship (a curved association may remain)

88
New cards

Strength of a correlation

A statistic indicating the strength of the linear relationship between variables

• Ranges from -1.00 to 1.00

• Cohen’s (1992) guidelines for strength:

• .10/-.10 (weak), -.30/.30 (moderate), .50/-.50 (strong)

• Newer guidelines from Gignac & Szodorai (2016):

• .10/-.10 (weak), -.20/.20 (moderate), .30/-.30 (strong)


Important: direction and strength are independent

• -1 is as strong as 1 (sign = direction; absolute value = strength)

• 0 means no linear relationship (a curved association may remain)

89
New cards

Dichotomous (measured value categorical)

exactly 2 categories

90
New cards

Ordered Categorical (measured value categorical)

categories have a meaningful order

91
New cards

Multivariate studies include:

  • one outcome variable

  • two or more measured predictors

  • no manipulated variables

  • no random assignment

    • Allows for Moderation and Multiple Regression analyses

  • research designs that involve measuring and analyzing three or more variables at the same time to understand complex relationships


92
New cards

Moderating variable

  • a third variable that changes the strength, direction, or presence of the relationship between an independent (predictor) variable and a dependent (outcome) variable

    • A moderator changes the X–Y slope or pattern; it is not simply another

      predictor of Y.


93
New cards

Regression variable

  • a data factor or element used in a statistical model to analyze or predict an outcome

  • what remains after measured adjustment?

Regression estimates the association between X and Y (moderator variable) conditional on included measured Z. It does not assign X at random or identify every plausible Z.


94
New cards

Common ways of reporting multiple regression

  • “when researchers controlled for…”

  • “even after accounting for…”

  • “above and beyond…”


95
New cards

Common ways of reporting moderation

  • “relationship was weaker for…”

  • “the association was the same regardless of…”