Empirical research studies

A in the ABC

Maximum potential for generalizability over actors. Findings hold across a large population of actors. Sample studies and formal theory come closest

B in ABC

Maximum potential for precision of measurement of behavior. You get tight control and precise measurement of what actors do. Laboratory experiments come closest

C in ABC

Maximum potential for realism of context. You study behavior in its natural setting. Field studies come closest

The ABC order

laboratory experiment

experiment simulation

field experiment

field study

computational simulation

formal theory

sample study

judgement study

Retrospective case study

studies a case after it happened, using historical data and people’s recollections. It’s fast, cheap and the outcome is already known but you are limited to whatever was recorded and peoples memories are biased

Longitudinal case study

follows a case over time as it unfolds. You see changes as they happen and have control over what data gets collected but it takes a lot of time and resources. The case may change or end and participants may drop out

Triangulation

Studying the sample phenomenon from several angles. It makes the findings more valid and credible and it makes up for the weaknesses of any single source or method

Data triangulation

use several data sources or collect the same data at different times

Probabilistic sampling + example

every member of the population has a known, non zero chance of being selected and selection is random. This allows statistical generalization. Ex simple random sampling

Non-probabilistic sampling

the researchers choice or peoples availability decide who is selected. The selection probabilities are unknown, so the sample can be biased. Ex convenience sampling

Conclusion validity

the statistical relationship may be wrong, ex fishing for results

Internal validity

something other than the treatment may cause the effect, ex choose experts may represent experience rather than a tool

Construct validity

the treatment or the measure may not represent the intended construct, ex measure outcome in only one way

External validity

the result may not generalize to other people or settings, ex subjects dont represent target population

The four type of threats to validity

conclusion, internal, construct, external

The three cycles of design science

Relevance: make sure the research is relevant

Design: produce a working, evaluated artifact

Rigor: make sure the research is rigorous and contributes to something new

Complete pooling

ignores the clusters and uses one shared intercept for everyone. It underfits since it assumes all clusters are identical and hides real differences

No pooling

gives each cluster its own independent intercept with a fixed prior so no information is shared. It overfits since each estimate relies only on that clusters data

Partial pooling

gives each cluster its own intercept drawn from a shared population distribution whose mean and spread are learned. Best overall

Design science study

a solution seeking research that aims to design and evaluate an artifact that solves relevant real world problems and to generate knowledge while doing so

Design science guidelines

1. Design as an artifact

2. Problem relevance

3. Design evaluation

4. Research contributions

5. Research rigor

6. Design as a search process

7. Communication of research

Conditions for using incomplete survey questions

When the dropout is random

The questions needed for a specific analysis were answered

The respondent consented to their partial data being used

ZIP and ZAP

handle more zeros than a Poisson distribution predicts, the difference is where the zeros come from

ZIP: a mixture two part model and zeros come from both parts of the mixture

ZAP: a hurdle two part model and the zeros only come from the binary part

Compulsory steps in BDA

1. Model your causal assumptions in a discrete acyclic graph

2. Identify the adjustment set using d-separation

3. Select a distribution family with maximum entropy

4. Build a regression formula with an appropriate link function

5. Select and assess prior probability distributions

6. Generate posteriors with MCMC algorithms

7. Assess the model fit via training diagnostics

8. Plot marginal effects to draw conclusions from bayesian models

Link function

maps the parameter’s constrained range onto the unbounded scale of the linear model, so it can’t predict impossible values. Its inverse maps the linear predictor back to the parameter’s scale

Parameter space

the scale on which the model’s parameters and the linear predictor live. It’s unbounded and effects are additive and linear

Outcome scale

the scale of the quantity you care about and observe. You get it through the inverse link function

Pilot study

a small scale trial run of the survey before the real data collection, to find problems when you still can fix them

Five essential aspect for a case study

1. An empirical enquiry or investigation

2. A contemporary phenomenon

3. In its real life context

4. Using multiple sources of evidence

5. The boundary between phenomenon and context is unclear

Observer triangulation

use more than one researcher to collect and analyze data to reduce individual bias

the 3 degrees of data sources

First degree: direct methods

Second degree: indirect methods

Third degree: independent analysis

First degree of data sources

The researcher is in direct contact with the subjects and collect data in real time. Gives rich data but is expensive and intrudes on the subjects

Second degree of data sources

The researcher collects data directly but without interacting with the subjects. It’s less intrusive but the data is harder to interpret without context

Third degree of data sources

The researcher analyses work artifacts that already exists or that other compiled. It’s cheap and unobtrusive but the data wasn’t collected for your question so its quality and completeness aren’t under your control

Exp design: between subjects design, completely randomized +sc

each subject is randomly assigned to exactly one treatment and the groups are compared

sc: you need many subjects since each gives data for only one treatment. Individual differences between subjects also add a lot of variance which lowers statistical power

Exp design, repeated measures design + sc

every subject receives all treatments one after another so each subject act as their own control

sc: learning, fatigue or carry over effects

Exp design, factorial design +sc

two or more factors are varied at the same time and subjects are assigned to combinations of their levels, this can estimate interaction effects between the factors

sc: the number of treatment combinations grow multiplicatively with the number of factor and levels, so you need many subjects

x <- z -> y

confounder, condition on z

x -> z -> y

mediator, condition on z if you want direct effect

x -> z <- y

collider, never condition on z

five steps action research cycle

Diagnosing: identify and define a problem
Action planning: consider alternative courses of action
Action taking: implement the selected intervention

Evaluating: study the consequences of the action

Learning: identify general findings from the process

Three important aspects of action research

action / intervention: we do something at the company

observation / measurement: we measure the effect of our action

context / host: we need to act at the company’s premises and intervene with their processes

Three conditions for Q to verify as an instrumental variable for X ← 𝑈 → 𝑌

Q is independent of U
Q is not independent of X

Q only influence Y through X

What are instrumental variables used for

estimating an unbiased causal effect even when an unmeasured confounder makes the backdoor criterion impossible to satisfy

What explicitly do modeling covariance between coefficients improves over regular cross-classified models

Modeling the covariance means giving the intercepts and slopes of each cluster a joint multivariate normal prior with a correlation matrix. This improves pooling across parameters, not just across clusters, makes more accurate estimates and more realistic predictions for new clusters

name two criterias by which you can assess the quality of a scientific publication

peer review and venue ranking and citation count

Peer review and venue ranking

a paper published in a peer review venue has been checked by independent experts before release. They can be misleading since different reviewers judge differently and positive results are more likely to get published

Citation count

the number of times the paper has been cited

Error types in hypothesis tests

false positive: you reject H_0 even when its true
false negative: you fail to reject H_0 even though its false

Obtrusiveness

how much a researcher intrudes on or manipulates the research setting to collect data

Generalizability

how far the findings extend beyond the specific setting studied

Golem of science

a statistical model is like a golem: powerful but mindless, it does exactly what it’s told without understanding why. The scientist must provide the intent and supervise it, otherwise it can produce confident but meaningless or harmful results

Three problems with significance tests

1. Hypotheses are not models

2. Observation errors

3. Most hypotheses are continuous, not all or nothing

Explain “hypothesis are not models”

a scientific hypothesis doesn’t translate directly into a single statistical model. One hypothesis can match several models and one model can match several hypotheses. So rejecting the null model tells you little about which substantive hypothesis is true

Explain observation error

real observation contain measurement error so you can’t be sure a falsifying observation is real

Explain “most hypothesis are continuous, not all or nothing”

“all swans are white” can be falsified by one black swan but most scientific hypotheses are claims about degree and a single observation can’t falsify them. The real question is how large the effect is and how uncertain we are

Prior probability

describes how plausible each possible value of a parameter is before you see the data

Strategies to ensure ethical research

informed consent and confidentiality and anonymity

Four pitfalls for ensuring ethical research

1. Fabrication: making up data or results that was never collected or lying about procedure

2. Falsification: manipulating real data or materials to get a specific result

3. Plagiarism: presenting someone else’s work as your own

4. Omission and commission: not saying something that should be said or say something false

Backdoor path

any path between the exposure x and the outcome y that starts with an arrow pointing into x. It creates a non causal association between x and y

Underfitting

the model is too simple or too rigid to capture the regular features of the data. It learns too little from the data and give poor predictions on both the training data and new data

Overfitting

the model is too flexible or complex and learns the noise of a particular sample. It learns too much from the sample so it’s a very good fit for the training data but poorly predicts new data because noise doesn’t repeat

Strategies to avoid overfitting

regularizing priors: make the model skeptical of extreme parameters

cross validation or information criteria: estimate out of sample performance so you can predict overfitting

General purpose of information criteria

estimate a models out of sample predictive accuracy, there is AIC, DIC and WAIC

3 different experiment designs

between subjects design, within subjects design, factorial design

Interaction effect

occur when the effect of one variable on the outcome depends on the value of another variable

Two diagnostics to measure MCMC success

trace plots: should look like a hairy caterpillar
trunk plot: the chains rank histograms are flat and overlapping with no chain consistently above the others

Shrinkage

each clusters estimate is pulled toward the population mean. Less data and further from the mean leads to more shrinkage

What makes a cross classified model differ from other multilevel models

cross classified models has more than one type of cluster at the same time and the cluster types cross each other instead of being nested

What research questions do cross classified model answers

where the variation come from and what the effect of a predictor is once all sources of clustering are accounted for