research methods final

research design

a plan that shows how one intends to study empirical questions

what are the aspects of a research design?

• theory or hypothesis

• unit of analysis

• necessary observable data

• data collection procedures

• analytical procedures

correlation

a statement that two things are systematically related

what are the necessary aspects of causal research design?

• covariation

• time order (cause precedes effects)

• all joint causes are eliminated

spurious relationship

two things are both affected by a third factor that appears to be related

classical randomized experiment

a type of experiment that includes a control group and a treatment group that have randomized membership for each. There is controlled administration of a stimulus to the treatment group and a pre and post test measures the impact of the stimulus. This experiment controls the environment.

stimulus/test factor

a condition applied to participants in an experiment in a way which the researcher can measure some sort of effect

experimental effect

difference in responses between group recorded by pre and post tests resulting from the administration of a stimulus

internal validity

integrity of the experiment or study and the methods in which it was conducted

what are the factors of internal validity

• maturation

• test-subject interaction

• selection bias

• experimental mortality

• demand characteristics

maturation

a factor of internal validity in which there is a change in subjects over time that might produce differences between experimental and control groups

test subject interaction

a factor of internal validity the process of measuring the dependent variable before the experimental stimulus, which may itself affect the posttreatment scores of subjects

selection bias

a factor of internal validity where subjects were not chosen independently

experimental mortality

a factor of internal validity where there is a loss of subjects from treatment to control groups

demand characteristics

a factor of internal validity where the aspects of the research situation cause participants to guess at the investigators goals and adjust their behavior or opinions accordingly

external validity

extent to which the results of a study can be generalized across populations, times, and settings

Causes of effects

A way to ask a causal question that considers studies that are qualitative, use case studies (small N), and start with a potential cause and measure its impact on the outcome.

effects of causes

a way to ask a causal question that considers studies that are quantitative with a large N, starts with an outcome and works backwards to causes, and relies on statistics.

types of case studies

• idiographic

• hypothesis generating

• hypothesis testing

• plausibility probes

Idiographic

a type of case study that aims to focus on a singular historical episode with no intention of being generalized. It may be inductive (focuses on all aspects of a case) or theory-guided (explicitly structured with a conceptual framework that focus on some aspects but not others)

Hypothesis-Generating

a type of case study that examines one or more case studies to develop more theoretical general propositions

Hypothesis-testing

a type of case study testing hypothesized empirical relationships

Plausibility Probes

a type of case study that sharpens hypothesis, theory, operationalization, measurement, or to preview a case study to test theory

method of difference

a strategy of case selection in which the researcher selects cases whos outcomes differ, compares one dissimilar factor, and concludes that it is the cause or effect

method of agreement

strategy of case selection in which the researcher selects similar cases with the same outcome and then identifies the factor that is shared

method of strength

a strategy of case selection in which the research measures strengths of conclusions tested by most likely to least likely. Sometimes they pick a deviant case study that does not conform to theory

counterfactual understanding of causation

difference making where the outcome is determined if cause and outcome are absent, If a = b, a and b won’t occur

mechanistic understanding of causation

causal process within individual cases, focus on the mechanism that means a = b

process tracing

qualitative method that involves deductive reasoning (trace a process leading to an outcome) and it only involves one case and testing evidence

4 tests of process tracing

• hoop test

• smoking gun test

• doubly decisive test

• straw in the wind test

Hoop test

a test of process tracing that involves evidence that is certain but not unique

Smoking Gun test

a test of process tracing that involves unique evidence that is not certain

Doubly Decisive test

a test of process tracing that involves both certain and unique evidence

Straw in the wind test

a test of process tracing that involves evidence that is neither unique or certain

necessary cause

a condition that must be present in order for the outcome to occur

sufficient cause

a condition in which the outcome is always found

Casually Homogenous Population

a generalization that expects the conclusion to have the same outcome or causal relationship across the population

Causally Heterogenous Population

A generalization that a cause may have different effects across different cases or the same cause and same outcome through different causal mechanisms

Bayesian Logic

a generalization that involves probability calculations to evaluate evidence from tests and update beliefs about competing explanations

Drawbacks of case studies

they requires long and arduous efforts and may result in researcher bias and subjectivity

What is the choice of data collection method determined by?

• validity of measurements

• effect of data collection on measurement

• population

• resources and cost of a method

• public availability of data

• ethical implications

• reactivity

Primary data

data is recorded and used by the researcher making the observation

Secondary data

data is used by the researcher who did not personally collect the data

types of empirical observations

• firsthand information

• document analysis

Firsthand information

a type of empirical observation in which information is collection in the field or in a lab study. It can be through a direct observation, indirect observation, accretion measures, erosion material. The latter two are more likely to be biased.

document analysis

a type of empirical observation that relies on record keeping like written records, running records, episodic records, or interview data.

Advantages of using a running record

• cost

• accessibility

• covers a more extensive record

Disadvantages of using the running record

• at mercy of record keeping organizations

• not willing to share raw data

• difficult to find out processes

advantages of the written record

• access to more difficult to reach subjects

• raw data is non-reactive

• records available for analysis over time

• can use a larger sample size than with interview or direct observation

• less expensive because costs are borne by record keepers

disadvantages of the written record

• selective survival

• large gaps in archives

• may be biased, incomplete, inaccurate, falsified

• lack a standard format

Structured observations

looks for and systematically records the incidence of specific behaviors

unstructured observations

all behavior is considered important and is recorded

4 methodological observations

• direct or indirect

• participant or nonparticipant

• overt or covert

• structured or unstructured

reactivity

effect of data collection itself on phenomena measured

DA-RT

data accessibility and research transparency principles

ethnography

observation beyond description of events or actions to reveal the cultural constructions of everyday life

field studies

open ended and wide-ranging observations in a natural setting. It observes behavior in an ordinary setting and with an accuracy of detail. Most likely observation in political science.

what are the levels of review?

• exempt

• expedited review

• full board review

what are the three ethical principles?

• informed consent

• beneficence

• justice

what are the potentials for harm in a study?

• negative repercussions for reputation

• invasion of privacy

• stress during research

• disclosure of behavior or information resulting in harm after the study

content analysis

when the research can take verbal, nonquantitative documents and transform it into quantitative data

Steps of content analysis

• select sampling frame from set of materials

• coding units

• sampling the actual material from that sampling frame

• choose recording unit

• define the categories of content going to be measured

• decide on numeric values that will be used to code each category in each category in each recording unit

post-test design

a test in which there are two groups, two variables, but no pretest. It requires the random assignment of subjects

what are the pros and cons of post-test design?

pros

• doesn’t require a pre test because it assumes the sample is truly random and sufficiently large that one can assume that the control and experimental groups are equivalent

cons

• assumes extraneous factors have been controlled for

repeated measurement design

an experiment that may contain several pretreatment and posttreatment measures and can be more accurate because of multiple measures of the dependent variable

what are the pros of repeated measurement design?

pros

• more statistical power

• more accurate because of repeated results that are consistent

• small N that makes it easier to control for extraneous factors

cons

• consuming of time and resources

• exposed to order effects like salience, redundancy, consistency, and fatigue

multiple group design

more than one experimental or control group created so that different level of experimental variable can be compared and it needs to randomly assigned.

what are the pros and cons of multiple group design?

pros

• improves internal validity

cons

• increases the effect of participant variable like qualities of participants. this could be a reason to not do the test at all.

• it is incredibly difficult to control for spurious relationships but it is very hard to do

• if only doing a posttest, it reduces internal validity because you assume extraneous factors have been controlled for

field experiment

an experiment in a natural setting in which the investigator does have control over group membership but does not have control over one or more independent variables. In other words, the independent variable is manipulated and the dependent variable is measured but the experiment is carried out in a setting that is natural to the participant.

what are the pros and cons of a field experiment?

pros

• higher external validity

• direct observation

• allows for more social nuance

• very detailed

cons

• cant control for anything which is not great for internal validity

• subjective methods matter (alice goffman)

natural experiment

factors not under the investigator’s control, assign individuals, to control or treatment groups. in other words, the researcher is unable to manipulate the independent variable as it preexists but the dependent variable is still measured based on a naturally occurring independent variable.

quasi-experimental design

It has a treatment and control groups, but not randomly assigned, usually preexisting groups

what are the pros and cons of the quasi-experimental group?

pro

• able to weigh groups

con

• bad for validity

what are the pros and cons of a natural experiment?

pros

• tends to have more internal validity if its truly random and external validity

• can get around ethical bounds by being in a natural setting

cons

• must be localized and cannot be generalized

• more difficult to measure and is more expensive

qualities of nonexperimental design

• presence of a single group

• lack of control over the assignment of subjects to groups

• lack of control over the application of the independent variable

• inability to measure the dependent variable before and after the exposure to the independent variable occurs

types of nonexperimental design

• small-N designs

• focus groups

• cross section designs

• longitudinal designs

small n designs

a type of nonexperimental design that uses case studies for exploratory, descriptive, and explanatory purposes

focus groups

a type of nonexperimental design that can be used to create hypotheses for testing through other research designs

cross section designs

a type of nonexperimental design characterized by measurements of the independent and dependent variables at approximately the same time. It utilizes surveys and polls and thus needs data analysis for causal inferences. It is advantageous because it was highly doable, cheap, time efficient, and can be very accurate based on the method of polling. Although, it is subject to subject reported information and there is no control group for comparison.

longitudinal design

a time series modeled and measured across time, examined time order of causal relationship, and estimate age, cohort, and period affects.

what are the pros and cons of longitudinal designs"?

pros

• can be clear to visualize

• descriptive and detailed

cons

• likely to have holes in your data

• super time consuming

observational study

an experimental design in which the researcher neither manipulates experimental variables nor randomly assigns the subjects to treatment but instead merely observes causal sequences at one point in time

what are the pros and cons of an observational study?

pro

• more control of the process of measurement

con

• no control over the environment

types of surveys

• in person

• mail

• telephone

• email

• internet

• group surveys

what are the influencing factors of surveys or polls?

• cost

• completion rates

• sample-population congruence

• questionnaire length

• data processing issues

what does the cost of surveys depend on?

• questionnaire design

• length of the questionnaire

• geographic dispersion

• callback procedures

• respondent selection

• availability of trained staff

• method of specific costs like long distance rates or postage

what are the pros and cons of closed ended questions?

pros

• easy to code and analyze

• allows for self-selection into predetermined categories

• respondents are more likely to answer these questions with respect to sensitive topics

cons

• may force respondents to choose pre-coded answers that do not match their own position

• may include inappropriate choices leading to a choice of another

what are the pros and cons of an open ended question?

pros

• allow respondents to fully explain their answers

• allow researchers to find answers that they otherwise wouldn’t have come across

cons

• time consuming

• may be too complex, unclear, too long or short, too much diversity in the answers

question-order effect

an effect in which the order of questions results in salience, redundancy, consistency, and fatigue

sample-population congruence

how well individuals in a sample represent the population from which they are drawn

response rate

the proportion of persons initially contacted who actually participated

how do you manage data?

• sorting data by variable

• creating a new variable based on existing data

• deleting cases or variables

• selecting cases or variables for analysis

data matrix

an array of columns and rows that store observed values of variables. the rows show the data for each case or unit of analysis. the columns show the values of a single variable for each case have a heading with variable names

nominal variables

a variables that is indicated only by a difference between categories

ordinal

categories may be ranked in order in addition to indicating a difference between categories

interval

includes all of the information of the preceding levels and adds meaningful intervals between values of the variable but does not use a meaningful zero

ratio

includes all of the information of the preceding levels and adds meaningful intervals between values of the variable and uses a meaningful zero

frequency distribution

a table that shows the number of observations associated with each value of a variable

relative frequency

what share the frequency makes up in relation to other frequencies

cumulative percentages

sum of relative frequencies as cases progress down the rows