Research methods exam 2

observational research

involves direct observation recording the behavior of others

naturalistic observation

observe behavior in real-world setting

1.Participant observation—researchers may engage in same activities as participant
◦PROBLEM?
◦May influence participants' responses

2.Passive observers—researchers simply observe behavior without engaging

contrived observation

setting arranged for observing and recording behavior
- often in lab setting
- can occur in "real" world

undisguised observation

Participants know they are being observed

PROBLEM?
◦Reactivity—people behave differently when they know they are being observed

disguised observation

Observing participants without their knowledge
- observe public behavior
PROBLEM?
none, as long as behaviors occur in pubic
◦If secretly recording someone (e.g., two-way mirror) without first receiving consent, participants must have the right to withdraw data

Oberver bias

◦observers record what they expect to see rather than accurately representing what is actually occurring

observer effects (expectancy effects)

participants believe that they are being observed, they may alter their behavior to match what they think the observer is expecting

3 techniques to minimize observer bias/effects

\> use coding manuals
\> employ multiple raters
\> masked research design

Coding Manual

means of providing a precise statement of the operational definitions

Employ multiple raters

used to calculate inter-rater reliability

◦A high inter-rater reliability value does not mean that the construct was observed without bias

masked research design (blind design)

observers are unaware of the conditions to which participants have been assigned and/or unaware of that the study is about

◦E.g., Clever Hans-the mathematical horse

survey research

the most common type of research in the behavioral sciences
◦Easy/cheap; allows access to internal states

Open-ended survey

allow participants to respond in any way they choose to the survey question

advantages and disadvantages of open-ended survey

Advantages:
◦Allows for unexpected/novel responses

◦Disadvantages:
◦Must develop & apply coding scheme
◦Answers may not be relevant to hypotheses
◦hard to find a link between 2 different responses

closed-ended survey

require that participant responses conform to a pre-defined set of options

advantages and disadvantages of closed-ended survey

Advantages:
◦Easy to code
◦Covers relevant responses

◦Disadvantages:
◦May not reflect participants' true responses

3 types of closed-ended surveys

* forced choice
* likert-type scale
* semantic differential

forced-choice format

picking the best of two or more options

How often do you buy cheese?
1. I never buy cheese
2. I frequently purchase cheese

Likert Scale

Options anchored by the level of agreement

I take satisfaction in purchasing cheese
1 strongly disagree- 7 strongly agree

semantic differential

Options anchored by opposing adjectives

When i consider buying cheese I feel
Disgusted 1234567 Delighted

Why is it important for researchers to consider the construct validity of their survey questions

It is important to consider the unique issues with validity presented by using survey methods

Question wording can influence responses

unclear questions

Question wording can influence responses

Are you a good student?
is it because you get good grades or because you study and go to class?

leading questions

questions are framed in a way that suggests the researcher expects a particular answer
◦Participants are more likely to conform to expectations

double-barreled questions

simultaneously ask two things at once
◦It is impossible/difficult for participants to answer them differently, so you cannot tell which question drives their responses

Ex: How satisfied are you with your pay and work conditions?

negatively worded questions

occur when two (or more) types of negation are used in the same question
◦Participants who are not paying attention to detail can very easily answer the opposite of the intended question

Ex: No one actually thinks that people who don't watch TV are hipsters.
- No TV\= hipster
- No TV\= not a hipster

can also create problems with just one negative word
Ex: chocolate should never be restricted.

Why is it important for researchers to consider the construct validity of their survey responses?

Insensitive scales can fail to pick up on meaningful differences in participant's responses

response sets

occur when subjects respond according to a pattern instead of individual items
◦Two types:
◦1. Acquiescence: 'Yea-saying'/'nay-saying'
2. Fence sitting

Problems with response sets

Response sets can be avoided (or detected) by including items that are: * reverse-scored
* attention/quality check items
* infrequency scale
* removing midpoint to avoid fence-sitting

issue: socially desirable responding, where participants pick the popular/good choice

question order

earlier questions can change the way respondents answer later questions

Ex:
1. How happy are you with your grade on exam 1?
2. How satisfied are you with life overall

sampling

the process of selecting a sample of participants from a population of interest

Population

* entire set of people that is of interest
◦Population characteristics are called parameters
◦Parameters are denoted with Greek letters
◦The population mean is m

sample

subset of people in population
◦Sample characteristics are called statistics
◦Statistics are denoted by Roman letters
◦The sample average is M

Biased (non-representative) samples

systematically different from the target population in some way

unbiased/representative sample

accurately reflect the target population

probability sampling (random sampling)

every person has equal chance of being sampled

simple random sampling

Step 1: Create sampling frame \= population list
Step 2: Randomly select participants from population list
◦Random number table
◦Random digit phone dialing
◦Random number generator (e.g., randomizer.org)

problems with simple random sampling

◦Can be difficult and time consuming
◦Its randomness may result in non-representative sample, purely by chance

cluster sampling

technique in which clusters of participants within a population are randomly selected, and then all individuals in each selected cluster are used
◦Example: 12 bars in Towson and want to sample participants from 4 of them
◦Step 1: Randomly select from the 12 bars in Towson
◦Step 2: Collect data from all members from the 4 selected bars

multistage sampling

◦Two random samples: (1) Random sample of clusters; (2) random sample of people in clusters

◦Example: 12 bars in Towson and want to sample 16 people from 4 clusters
◦Step 1: Randomly select from the 12 bars in Towson
◦Step 2: Randomly sample 4 people from each of the 4 clusters

stratified random sampling

technique in which one picks categories of participants and randomly samples from each category (strata)
◦Strata are meaningful categories, not random categories like cluster and multistage sampling
◦Example: assessing opinions of gender equality within STEM fields and worried gender of participant may influence DV, so want proportion of gender in sample equal to that in population
◦Step 1: Create strata based on gender
◦Step 2: Randomly sample from women until reach 38% of sample, then randomly sample from men to get rest of sample

Oversampling

a variation of stratified sampling in which a researcher intentionally over represents one or more groups

convenience sampling

◦Method in which only those who are easiest to recruit are used in the sample

Reasons why researchers rely on convenience samples:

◦Easily accessible
◦Often less expensive
◦Can be quicker to obtain
◦May not have access to other forms of sample
◦Location
◦IRB
Rareness of population

When is it okay to use convenience sampling?

if studying basic psychological process

sampling error

difference between sample and population
◦Difference causes results from sample to differ from results if entire population was used

margin of error

◦degree to which the data obtained from sample is estimated to deviate from population

reduce it with large samples

correlation coefficient

a statistical measure of the extent to which two factors vary together, and thus of how well either factor predicts the other

What is the numerical range of the correlation coefficient?

-1 to +1

What two pieces of information are communicated in the correlation coefficient?

effect size and statistical significance

positive correlation

Both variables move in the same direction.

negative correlation

as one variable increases, the other decreases

coefficient of determination (r^2)

Proportion of variability in one variable that can be determined from the relationship with the other variable

statistical significance (p-value)

◦the probability of obtaining results that are at least as extreme as those observed, assuming the null hypothesis is true

ex: game, the best out of 3, have to beat them 2 times to show sufficient evidence of being a better player

the larger the p value,

the least significant
correlations with p-values < .05 are generally considered statistically significant

effect size

magnitude/strength of a correlation

benchmarks of effect size

0.10, -0.10 : small
0.30, -0.30 : medium
0.50, -0.50: large

restricted range

If the data does not accurately reflect the population, the correlations it produces may not either

Outliers

data points that do not fit with the general pattern of the rest of the data
◦Typically extreme scores on one of the variables

PROBLEM?
they can 'pull' correlations towards them
◦This is especially problematic in small samples

EXCLUDE THEM FROM DATA

non-linear data

◦Some strong relationships may be curvilinear in nature
◦If we examine bivariate correlations on curvilinear data, the resulting correlation will not accurately reflect the pattern
◦More complex model-fitting statistics are required (beyond the scope of this class)

Subgroups

groups formed using the third variable in which we can calculate separate correlations

moderator

◦If we see different patterns within our subgroups, it suggests the third variable may be a moderator
* third variable that affects the correlation between the variables of interest
*. it is necessary to interpret the correlation separately for each group

3 types of multivariate designs

longitudinal, multiple regression, mediation

3 criteria for causal claims and which tests address each of the three?

1.Covariance - variables are related
◦As A changes, B changes
2.Temporal Precedence - one variable comes before the other
◦A comes first in time, before B
3.Internal Validity - eliminate alternative explanations
Third variable(s) do not account for observed relationship(s

-bivariate correlational

longitudinal design

◦How do relationships change over time?

Types of results
◦Cross-sectional
◦Autocorrelations
Cross-lag correlations

cross-sectional study

designs examine the relationship between two or more variables at a single time point

Limitations of cross-sectional studies

cohort effect
◦Different environmental influences
◦Always confounded
◦E.g., different gun laws affecting access to firearms, different rearing practices with firearm exposure (hunting)

Autocorrelation

examine the relationship of one variable with itself across at least two time points

follow the same group of people over time and examine any changes

problem with longitudinal studies

selective attrition (mortality)
◦Individuals may drop out of a study over time in a non-random fashion (healthier, better-educated stay longer)
◦
This may be a major concern for suicide-related studies

cross-lag correlations

◦Assess the degree to which the earlier measure of one variable is related to the later measure of another variable
◦Addresses directionality (i.e., temporal precedence)
◦Cross-lag
◦Cross \= across variables
◦Lag \= "lag" in time

multiple regression

a statistical technique that computes the relationship between a predictor variable and a criterion variable, controlling for other predictor variables

◦Helps increase internal validity by ruling out third variables

predictor variable (independent variable)

measured variables that are used to predict something

the greater the number of predictor variables, the more effectively the model rules out confounds

Criterion Variable (dependent variable)

measured variable that is predicted

regression coefficient

◦Value that indicates how well it predicts the criterion variable controlling for all other predictors

what does it mean to control for a predictor in regression?

measuring extraneous variables and accounting for them statistically to remove their effects on other variables

Mediation Design

Any variable that explains the relationship between two other variables

5 steps of mediation

1.Establish correlation c (pattern of interest)
2.Establish correlation a
3.Establish correlation b
4.Test whether c' is still significant with the mediator present
5. rarely used step is to experimentally manipulate the mediator
◦Requires that predictor variable is measured/manipulated first, then the mediator is manipulated, then the outcome is measured
◦Known as causal mediation

independent variable

variable that is manipulated

conditions

levels of the independent variable

dependent variable

variable that is measured, outcome variable

control group

the level of the IV that receives no treatment

experimental(treatment) group

◦The level(s) of the IV that receives the "treatment" or manipulation

When manipulating an independent variable, why is it important to vary only one thing at a time?

to establish the one variable(IV) that causes another variable (DV)

comparison groups

◦allow us to see how scores on a dependent variable vary across conditions
◦Gives us a better understanding of how the independent variable affects the dependent variable

design confound

is present whenever the researcher introduces a factor that varies with the IV
◦This confounding variable always serves as a possible alternative explanation for any result

, the variable must be systematically associated with the IV

systematic variability

the levels of a variable coinciding in some predictable way with experimental group membership, creating a potential confound

unsystematic variability

◦These add 'noise' to the data, but do not interfere with internal validity
◦Unavoidable
◦E.g., participant characteristics, features of the testing area, environmental characteristics

effective way to avoid design confounds

choose the control group wisely
◦The more similar the control group is to the experimental group, the smaller the risk of design confounds

selection effects

◦The participants in one group systematically differ from those in the other
◦
Selection effects are often observed when participants are allowed to choose which group they join
However, they can also occur when researchers determine what group participants join

matched groups

helps avoid selection effects
◦participants are matched with other participants on a characteristic that is directly relevant to the focal DV
◦Randomly assign one person from each cluster to each condition in the experiment

random assignment

process by which participants have an equal chance of being assigned to each condition

random sampling

process by which participants have same chance of being selected

between-subjects (independent or between groups)

an experimental design in which participants are assigned to only one level of the independent variable

•Participants have less knowledge about the purpose of the study
•Avoids issues with participants seeing DVs multiple times

within-subject (within-groups)

an experimental design in which participants go through all the levels of the independent variable

•Individual differences of participants between conditions are controlled
•
•Fewer participants needed

post-test only design

the DV is measured only after exposure to the IV

◦Example: New anti-depressant medication
◦Take medication or placebo \> measure depression

pretest-posttest design

the DV is measured both before and after exposure to the IV

◦Example: New anti-depressant medication
◦Measure baseline depression \> take medication or placebo \> measure depression

order effects

◦Possibility that earlier versions of the IV will have a lasting effect on the DV
Introduces a confound to all subsequent DVs

practice effects

◦participants may get better or worse at a task over time

carryover effects

some aspect of a prior manipulation is carried over to a subsequent manipulation

Counterbalancing

eliminates order effects
◦Randomly assign participants to one of the potential order combinations in the experiment

Ex:
participants 1,4,5 drink coffee than rate attention, then drink tea, then rate attention

participants 2,3,6 drink tea, then rate attention, then drink coffee, then rate attention

full counterbalancing

subjects are randomly assigned to every possible combination of conditions