Applied Research Methods: Development

what type of research can be conducted using observations?

finding phenomena

what type of research can be conducted using correlations and quasi-experiments?

finding relationships

what type of research can be conducted using experiments?

finding causal relationships

what is meant by the precision of a theory?

accuracy of classification of which new data can be explained by the theory

e.g. “A will score higher than B in X, but lower than C, and B will score higher than A in Y".”

principle of parsimony

choosing the most straightforward theory from among theories fitting the data equally well

why are falsifiability and testability important?

to distinguish science from pseudoscience and compare studies based on their degree of falsifiability

internal validity

extent to which the observed results represent the truth in the population we are studying

→ observed mean difference

external validity

extent to which the results of the study are generalizable to other situations, populations, …

construct validity

the extent to which your test or measure accurately assesses what it's supposed to

statistical validity

the extent to which drawn conclusions of a research study can be considered accurate and reliable from a statistical test

how can correlations be interpreted?

direction and size

how can regression be interpreted

prediction

\
advantages of within-subject designs

* requires fewer participants
* increases chance of discovering a true difference among conditions
* more statistical power (individual variation is reduced)

advantages of between-subject designs

* minimizes learning effects across conditions
* shorter sessions
* easier to set up and analyze

what is the difference between a quasi-experiment and a true experiment?

no randomization in a quasi-experiment

alpha error

type 1 error → false positive

beta error

type 2 error → false negative

effect size

how large the difference/correlation/relationship is

true effect

effect size in the population, cannot be observed only estimated

(should be made before the study)

observed effect

calculated after the study, also an estimation of the true effect

statistical power

probability that the effect is statistically significant and correctly rejects the null hypothesis

what does low/high statistical power mean?

low power → small chance of detecting a true effect, results likely to be distorted

high power → large chance of detecting a true effect

cohen’s d formula

d = (m1-m2) / SD

what are the conventional values for small/medium/large effect sizes?

>0,8 → large

which factors affect the statistical power of a study

* effect size
* alpha
* sample size

how can the power of a study be increased?

* increasing sample size
* increasing the measured effect size
* increasing the alpha error (because more results are accepted as significant)

disadvantages of small sample studies?

more fluctuation and inflated effect size

→ publication bias

which results should you not trust?

results of studies with small samples cannot be trusted

cross-sectional design

all measures are collected in a single assessment

longitudinal design

measures collected in repeated assessments

advantages of experimental research

manipulation isolates the effect of interest, so alternative explanations are minimized

disadvantages of experimental research

* difficult to conduct (time & money)
* biases (volunteer & selection bias)
* can be ethically problematic
* limited generalizability

advantages of observational research

* more generalizability
* easier to obtain larger samples

disadvantages of observational research

* longitudinal studies are expensive (time & money)
* greater risk of biases and confounds
* data more likely to be incomplete and of poorer quality

cohort studies

assess prospective changes → looking forward in time

case-control studies

assess retrospective predictors → looking backwards in time

selection biases

biases introduced in the selection process, so that proper randomization is not achieved

\
e.g. sampling bias, allocation bias, non-response bias, publication bias, volunteer bias

information biases

biases introduced by systematic differences in the collection and handling of information in a study

\
e.g. misclassification bias, observer bias, interviewer bias, social desirability bias, recall bias, performance bias, detection bias

moderator

a variable that alters the strength of the linear relationship between a predictor (X) and an outcome (Y)

which statistical analyses are commonly used to test moderation

if X and Z are categorical and Y is continuous → ANOVA

if X, Z, and Y are continuous OR if X and/or Z are categorical and Y is continuous → Multiple linear regression

if X and Z are continuous and Y is categorical OR if X is continuous and Z and Y are categorical → Logistic regression

How to interpret a moderation effect from a linear regression?

1. Center X and Z scores (→ individual score-M)
2. Calculate interaction with centered scores (X\*Z)
3. Perform analysis with centered scores and new interaction effect as predictors
4. If significant → plot simple slopes

When should a moderator variable be measured?

at the same time as predictors

mediator

The IV influences the mediator variable, which in turn influences the DV

total effect (c)

effect X has on Y including the effect of the mediator

direct effect (c’)

effect X has on Y without taking the mediator into account

indirect effect

total effect - direct effect

absolute mediation

indirect effect explains the complete effect

How is mediation commonly tested?

using regression…


1. Show that X predicts Y
2. Show that X predicts the mediator
3. Show that the mediator predicts Y
4. Show the mediator produces an effect of X on Y

bootstrapping

testing of mediation effects using a resampling technique to adjust the standard errors of the coefficitents

3 things to consider before conducting a mediation analysis

1. have a directionality assumption (avoid reversal causal effect)
2. consider when to measure the mediator
3. choose reliable measurements

When should the mediator be assessed?

after the predictor and before the outcome

What is the difference between mediation and moderation?

Mediation describes indirect effects, moderation describes conditional effects

signal detection theory (SDT)

measures the ability to differentiate between stimuli (information-bearing patterns) and noise (random patterns)

→ measures how humans make decisions under circumstances of uncertainty

\
e.g. witness tries to identify a criminal, trying to remember whether you know someone, looking for spelling mistakes, discovering a spider on your wall

According to SDT, what are the 2 factors affecting human discrimination decisions?

1. sensitivity
2. decision/response criterion

sensitivity (in SDT)

* strength of the signal
* ability of the observer

decision/response criterion (in SDT)

* consequences of decisions (pay-off matrix)
* frequency of signal

According to SDT what are the possible responses (matrix)

theoretical assumptions that explain differences in sensitivity

in reality, dichotomous events (there is a signal or not) but subjective experience varies and is usually distributed around the mean

what does it mean when someone responds liberally/neutrally/conservatively?

liberal → lot of false alarms but few/no misses

neutral → responds with yes and no equally

conservative → many misses but no few/no false alarms

d’ (d prime in SDT)

standardized difference between the means of the Signal Present and Signal Absent distributions (strength of the signal relative to noise)

→ d’ = z(Hits) - z(False Alarms)

β (in SDT)

 observers ability to correctly identify a stimulus (willingness to give Yes responses)

→ β = y(Hits) / y(False Alarms)

(β = 1 means neutral responses)

what does a low/high d’ mean?

low = 0 (personal cannot discriminate at all, guessing)

high = 4.66 (almost perfect at discriminating, 99% accuracy)

what problems are caused by missing data?

* Response rate bias
* low statistical power
* invalid conclusions

missingness mechanisms

reasons why data is missing

ignorable missingness mechanisms

MCAR (missing completely at random) → independent of observed or missing values

MAR (missing at random) → partly depends on observed values but not missing ones

non-ignorable missingness mechanisms

MNAR (missing not at random) → depends on missing values themselves

proactive strategies for minimizing missingness

* advanced warnings
* personalized surveys
* follow-up reminders
* monetary incentives

listwise deletion

deleting all cases with any missing values

(violates a fundamental principle of missing data analysis)

pairwise deletion

still including cases with missing values into the analysis

(attempts to minimize the loss that occurs in listwise deletion)

imputation

replacing missing data with substituted values

(SPSS: → Analyze → Multiple Imputation → Analyze Patterns)

mean item imputation

mean of the observed values for each variable is computed and the missing values for that variable are imputed by this mean

(can lead to severely biased estimates)

advantage of using multiple imputation over single imputation

does not provide a deterministic idea of what the missing value should be, but allows it to have a range of different scores

what is the best way of handling missing values?

* Item(s) missing → mean item imputation
* Scale(s) missing → multiple imputation (if not MCAR)

disadvantage of comparing an experimental treatment with a waiting-list/treatment-as-usual group

easy to show that experimental condition is effective but overall less power

formula: percentage of improvement

(pre-post)/pre\*100

Lasagna’s Law

overestimation (with a factor ten to one) of the number of patients available for inclusion into your study in a certain period

when do you use Bonferroni correction?

→ if you test multiple times (to reduce inflated error probability)

ways to reduce unnecessary within-group variance in treatment outcome measurements

* specific hypotheses
* specific instruments
* inclusion- and exclusion criteria
* treatment manual, trained therapists, trained assessors
* inspection for outliers

treatment dropout/´refusal in psychoptherapy

\~25%

What are ERPs and what brain activity do they reflect?

= voltage fluctuations in the ongoing EEG that are time-locked to an event (e.g. stimulus onset or response execution)

\
→ reflect the sensory, cognitive, affective, and motor processes elicited by the event; usually labeled by their polarity (N/P)

Which properties make ERPs useful?

* covert monitoring of processing when overt behavior is difficult to measure
* can measure processes not evident in behavior

→ e.g. in infants, animals, coma patients, …

major challenge of the ERP technique and how to deal with it

many different processes happen in the brain at the same time

→ using a zero measurement (starting point assessment)

cognitive artifact

something physical or digital that has aided a mental process

→  leftover remnants indicative of the efforts it takes to unravel mental processes

challenges of recording/studying ERPs in clinical populations and their solutions

* The effect of medication on cognitive processing → compare to unmedicated patients
* opposing effects of comorbid disorders → investigate individuals of the same disorder with and without comorbidity
* individuals with disorder show more artifacts → adjust recording procedures (less conditions, less electrodes, rest breaks, etc.)

how are ERP components usually quantified?

latency = amplitude and time between stimulus and peak

conclusions that can be drawn from ERP studies

presence, size, or timing of a specific mental operation, and the effect of manipulations or individual differences on these factors

Big Hypothesis (Rossignol et al., 2012)

people with high levels of social anxiety have a greater P1 (encoding of faces) and P2 (attentional resources) effect than people with low levels of social anxiety.

why did they use ERPs? (Rossignol et al., 2012)

attentional bias can already be detected 100ms after the first stimulus is presented

Conclusions (Rossignol et al., 2012)

perceptual processing of social cues is extra strong in people with social anxiety (P1 component), but linking attention is generic for all anxious states

what differences do you look at when comparing ERPs across conditions/individuals?

* amplitude differences
* latency differences

How can ERPs be used to study disturbances in clinical populations?

* between-group comparisons (diagnosed/no diagnosis)
* correlational approach (relate ERPs to symptoms/traits)
* longitudinal studies (assess risk for psychopathology)

examples for direct vs. indirect measures

* direct measures: questionnaires (directly infer attitudes)
* indirect measures: reaction time (infer attitudes from results)

advantages of indirect measures

measure implicit attitudes, which predict automatic behavior

projective tests and their disadvantages

tests that commonly use ambiguous stimuli

e.g. Rorschach, TAT

→ low reliability and validity

advantages of modern indirect measures

* more objective than projective tests
* reliability and validity can be determined

how does the Implicit Association Test (IAT) measure associations?

measures mean reaction time for compatible and incompatible blocks of stimuli → short RTs indicate stronger associations

which 4 types of stimuli are needed for IAT?

2 target stimuli and 2 attribute categories

formula IAT effect

mean RT(incomp. block)-mean RT(comp. block)

problems with the interpretation of IAT effects

two types of targets → effect can be caused by either target being more associated with one stimulus or both (usually both)

problems with most indirect measures

* lack of convergence
* reliability
* validity
* general vs individual stimuli