Research Methods Exam 1

Research Methods

A process of steps, rooted in the scientific method to systematically collect and analyze information with the goal of producing new knowledge

Ways of Collecting Knowledge

Trust a source of authority, observation/experience, listen to intuition, logic (such as solving a math problem)

Intuition

A way of collecting knowledge; Relying on guts, emotion, and/or instincts to guide us. Involves believing what feels true. Can be wrong, driven by cognitive and motivational biases, not logical reasoning or scientific evidence.

Authority

A way of collecting knowledge; Accepting new ideas because some authority figure states that they are true. May be wrong, uses intuition of who to trust.

Rationalism

A way of collecting knowledge; Using logic and reasoning to acquire new knowledge. Premises are stated and logical rules are followed to arrive at sound conclusions. But if premises are wrong or there is an error in the logic then the conclusions are not valid.

Metaphysical Explanations

Attributing causal power to non-physical entities (deities and the supernatural). Includes animism, mythology, religion, and astrology.

Animism

Type of Metaphysical Explanation; Non human entities are alive and act on the world (ex. trees are alive)

Mythology

Type of Metaphysical Explanation; non corporeal deities act on the world

Astrology

Type of Metaphysical Explanation; Behavior is determined by celestial bodies

Philosophy

Using observation and the rules of logic to collect and analyze info (400BC) challenge to metaphysics

Physiology

Using experimental methods to study the relationship between the body and brain

Psychology

The scientific study of human behavior (experimental philosophy)

Scientific Cannons

Determinism, Empiricism, Testability, Parsimony

Determinism

Scientific cannon; All events have meaningful and systematic natural causes

Empiricism

Scientific cannon; Basing knowledge of causal relationships on observation. We can learn about systemic causes by observing the world around us. Learn about the world systematically by carefully planning, making, recording, and analyzing observations of the natural world.

Testability

Scientific cannon; Theories should be testable and falsifiable

Falsifiable

Scientific claim must be expressed in such a way that there are observations that would (if they were made) count as evidence against the claim.

Parsimony

Scientific cannon; When explanations are equally good at predicting behavior, we should prefer the more simple one

Law

A universal statement of the nature of things that allows reliable predictions of future events (always occurs under certain circumstances). There are none of these in psychology.

Theory

A general statement about the relationship between 2 variables. Coherent explanation or interpretation of one or more phenomena. It can be untested or extensively tested and supported.

Hypothesis

Testable predictions about specific events derived from one or more theories. A specific prediction about a new phenomenon that should be observed if a particular theory is accurate.

Scientific Method

In psychology this allows us to gain knowledge about human nature that we don’t (humans outside our sample) or can’t (invisible psychological processes) observe. Process of systematically collecting and evaluating evidence to test ideas and answer questions. Most likely to produce valid knowledge. BUT not always feasible, cannot be used to answer all questions just empirical questions.

Operational Defintion

Definitions of theoretical constructs (hunger, fear, etc.) that are stated in terms of observable and testable procedures. We can look at bodily reactions, physiological reactions (hormones, heart rate), neural activity (fMRI), facial expressions, action tendency.

Equifinality

Same phenomenon often has multiple different causes, why we support a hypothesis but do not find it to be true.

Induction

Reasoning from the specific (hypotheses) to the general (theories). (e.g. experiment shows that people prefer symmetrical to asymmetrical faces, therefore I theorize that symmetry increases physical attraction). Problem: how many times must we confirm our theory? Always possible that next hypothesis test contradicts theory

Deduction

Reasoning from the general (theories) to the specific (hypotheses). (e.g. i theorize that symmetry increases perceived attractiveness, therefore I hypothesize that morphing faces to be more symmetrical will increase perceived attractiveness).

Cognitive Dissonance

The unpleasant state of arousal that occurs when our thoughts, behaviors, and feelings are inconsistent. Why peoples beliefs will strengthen in the face of contradictory information.

Nuremberg Code

A set of 10 principles written in 1947 in conjunction with the trials of Nazi physicians accused of cruel research on concentration camp prisoners during WW2. Importance of weighing risks against benefits and informed consent.

1. Voluntary consent of the human subject is absolutely essential

2. The experiment must yield generalizable knowledge that could not be obtained in any other way and is not random and unnecessary in nature

3. Animal experimentation should precede human experimentation

4. All unnecessary physical and mental suffering and injury should be avoided

5. No experiment should be conducted if there is reason to believe that death or disabling injury will occur

6. The degree of risk to subjects should never exceed the humanitarian importance of the problem

7. Risks to the subjects should be minimized through proper preparations

8. Experiments should only be conducted by scientifically qualified investigators

9. Subjects should always be at liberty to withdraw from experiments

10. Investigators must be ready to end the experiment at any stage if there is cause to believe that continuing the experiment is likely to result in injury, disability or death to the subject

Belmont Report

1978 set of federal guidelines, explicitly recognized the principle of seeking justice: risks and benefits distributed fairly across groups at the societal level. Acknowledges the importance of respect for person’s. Recognizes the principle of beneficence: maximizing the benefits of research while minimizing harms to participants and society. Main Ideas: Justice, Respect, Beneficence

Justice

Main idea from the Belmont report: risks and benefits distributed fairly across groups at the societal level

Beneficence

Main idea from the Belmont report: maximizing the benefits of research while minimizing harms to participants and society

Peter Neubauer Twin Studies

8 sets of twins, 1 set of triplets, put up for adoption at Louise Wise agency in NYC - Twins separated out to families, not told of existence of siblings - Researchers studied development of twins over the years, conducting interviews, measuring outcomes - No consent received - Triplets randomly found each other - Subjects demanded release of data, researchers refused and entered into a contract with Yale to seal the data until 2065. Caused psychological harm, one committed suicide.

Milgram’s Obedience Studies

(1961) Milgram wanted to study conformity, will people conform to aggressive behavior. The study was supposed to be a baseline for a future conformity study, but the baseline research ended up defying expectations. If the researcher says they must continue the “teacher” will continue. Caused psychological harm to the victims.

Stanford Prison Experiment

(1972) separated undergrads into prisoners and guards, shut down after a few days, students suffered psychological and physical harm. Zimbardo did not shut it down fast enough.

National Research Act

(1974) act that required formation of IRB’s

Institutional Review Board (IRB)

A committee that is responsible for reviewing research protocols for potential ethical problems; board responsible for conducting risk-benefit analysis for all studies using human subjects

Risk Benefit Analysis

benefits: gaining knowledge, experience with research; risks: potential physical/ psychological harm

How to Minimize Risks

anonymity/ confidentiality, informed consent, freedom from coercion, debriefing

Anonymity

Participants names and other personally identifiable information is not collected at all

Confidentiality

Agreement not to disclose participants personal info without their consent

Informed Consent

Researchers obtain and document peoples agreement to participate in a study after having informed them of everything that might reasonable be expected to affect their decision

Debriefing

Informing participants as soon as possible of the purpose of the study, revealing any deception, and correcting any other misconceptions they might have

Reliability

The degree to which a measurement, or set of research findings, produce consistent results. The consistency of a measure; psychologists consider 3 types of consistency: over time (test retest), across items (internal consistency), and across different researchers (inter rater)

Validity

How accurately your measures and design test what they are meant to test. The extent to which the scores from a measure represent the variable they are intended to.

Construct

Idea that cannot be observed directly, often they represent tendencies to think, feel, or act in certain ways. They often involve internal processes.

Conceptual Definition

Definition that describes the internal processes that make up a construct, along with how it relates to other variables.

Operational Definition

Defintion of a variable in terms of precisely how it is to be measured.

Internal Consistency, Test-retest Reliability, Interrater Reliability

Types of Reliability

Internal Consistency

Type of Reliability; (for scales and surveys) The degree to which participants respond in the same way to multiple items in a single scale intended to measure one construct (ex. answers “agree” to most questions, if none are reverse coded). To figure this out you calculate Cronbach’s Alpha. The consistency of peoples responses across the items on a multiple-item measure; can only be assessed by collecting and analyzing data.

Cronbach’s Alpha

Used to figure out internal consistency; a measure similar to the correlation between items on a scale. High this= high internal consistency (usually greater than 0.6 is acceptable) scale is 0-1

Test Retest Reliability

Type of Reliability; (for scales and surveys) degree to which participants respond in the same way to a measure at different times. Although sometimes it is not reasonable to expect peoples responses to remain the same over time, is there reason for us to expect that things have changed? Extent to which a measure is consistent across time; test produces consistent results when given to the same people on separate occasions.

Interrater Reliability

Type of Reliability; (when you have people observing/rating) the degree to which different judges or observers agree on a measurement or judgement. The extent to which different observers are consistent in their judgements.

More Observations

How to increase the reliability of a measure

Replication

How to increase the reliability of experimental findings

Replication

The degree to which identical results of a study can be found by other (ideally independent) researchers. Helps increase the reliability of experimental findings

P-Value

(alpha) the acceptable level of probability of falsely rejecting the null hypothesis (committing a Type 1 error). Probability of type 1 error = alpha (typically p=0.05)

File Drawer Problem

Many studies that are conducted are never reported, and these studies might differ, on average, from studies which are reported. Studies that find nothing, usually do not get published. So we only see research that supports the hypothesis. Published research shows distorted effects

Type 1 Error

False positive; when a researcher concludes their results are statistically significant when in reality there is no real effect in the population and the results are due to chance.

Type 2 Error

Missed opportunity; when a researcher concludes that their results are not statistically significant when in reality there is a real effect in the population.

Funnel Plot

Plots sample size of studies against effect size of the study. The bigger the sample the less variability there should be in effect sizes. If effect sizes don’t follow this pattern, then some effects are being left out of the literature (i.e. file drawer problem)

Publication Bias

Tendency for reviewers/journals to prefer positive findings compared to null results; reason for the file drawer problem

Construct Validity

The degree to which your measured variables reflect the hypothetical variables or “constructs” of interest (instead of another variable). Example: Testing whether a questionnaire actually measures "depression" rather than general anxiety or fatigue

Convergent Validity

Type of construct validity; Degree to which different measures of the same construct correspond. Ex. 3 surveys get the same results.

Discriminant Validity

Type of construct validity; measures of constructs that are not theoretically related should not correspond. The extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct (ex. self esteem and mood) proves they are two distinct variables.

Criterion Validity

(aka Concurrent and Predictive Validity) measures of constructs should correspond with (concurrent) or predict (predictive) measures of theoretically related constructs (ex. happiness and prosocial behavior). Extent to which people’s scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with.

Concurrent Validity

Type of criterion validity; How well the test correlates with a criterion measured at the same time (e.g., a short anxiety scale matching a clinician's current diagnosis). When the criterion is measured at the same time as the construct.

Predictive Validity

Type of criterion validity; How well the test forecasts future performance or behaviors (e.g., SAT scores predicting college GPA). When the criterion is measured at some point in the future (after the construct has been measured).

Content Validity

The degree to which a measure represents all dimensions of a given construct. Practically impossible (to find all dimensions). Extent to which a measure “covers” all the aspects of a variable.

Face Validity

The degree to which a measure appears to be relevant to the construct that it intends to measure. Ex. exam questions that appear related to material covered. The extent to which a measurement method appears “on its face” to measure the construct of interest; relies on intuition.

Levels of Measurement

4 levels that correspond to 4 types of information that can be communicated by a set of scores, and the statistical procedures that can be used with the information

Nominal Scale

Categorical scales with non-numerical names; no order or ranking. (ex. choose fav ice: cream, vanilla, chocolate, or cookie). Used for categorical variables and involves assigning scores that are category labels, does not imply any ordering

Ordinal Scale

categorical scales that involve order or ranking, but no information about relative size or degree of difference between categories. Cannot calculate an average. (ex. how spicy: hot, hotter, hottest). Involves assigning scores so that they represent the rank order if the individuals. Does not necessarily mean equal intervals.

Interval Scale

Use numbers to reflect relative difference in magnitude. Have order and ranking but also give information about relative size and degree of difference between units. Can calculate an average. Involves assigning scores using numerical scales in which untervals have the same interpretation throughout. (ex. IQ scores)

Ratio Scale

Like interval scales but with a true and meaningful zero point. (ex. weight, height). Involves assigning scores in such a way that there is a true zero point that represents the complete absence of the quantity.

Internal and External Validity

Types of validity in an experiment

Internal Validity

The extent to which a set of research findings provides compelling evidence about causality. High this means you can distinguish correlation from causation. Said to be high if the way an empirical study was conducted supports the conclusion that the independent variable caused any observable differences in the dependent variable. Degree to which the study proves the IV caused changes in the DV.

Non Experimental Design

Type of experimental design; Observational, archival, correlational, no comparing across groups. Low Internal Validity. High External Validity.

Quasi Experimental Design

Type of experimental design; Comparing groups but no random assignment. (e.g. gender) Medium internal validity. Medium external validity.

Experimental Design

Type of experimental design; Comparing groups with random assignment. High internal validity. Low external validity.

Covariation, Temporal Sequence, Eliminating Confounds

3 things you need to demonstrate to establish causality (i.e. to have internal validity)

Covariation

One of the things you need to demonstrate to establish causality; Changes in one variable correspond with changes in another.

Temporal Sequence

One of the things you need to demonstrate to establish causality; Changes in causal variable must precede changes in the outcome variable.

Eliminating Confounds

One of the things you need to demonstrate to establish causality; Ruling out all “third variables”.

Manipulation

Systematically altering the levels of variables (the IV) to see if it causes changes in some outcome variable (the DV). This helps avoid confounds (procedural and operational). For an independent variable this means to change its level systematically so that different groups of participants are exposed to different levels of that variable, or the same group of participants is exposed to different levels at different times.

Operational Confounds

When manipulation of data does not adequately change the construct of interest. (Ex. disgust is not best manipulated via “eating rum raisin ice cream”)

Procedural Confounds

Experimenter unknowingly manipulates more than one thing (Ex. disgust and humor are both manipulated)

Manipulation Check

Survey items which are meant to ensure that the IV has indeed been changed across comparison conditions. Separate measure of the construct the researcher is trying to manipulate. Purpose is to confirm that the IV was successfully manipulated.

Noise

Extraneous variables that are evenly distributed across conditions (so they don’t systematically bias results) but that influence the DV and make it more difficult to detect and effect of the IV. (Ex. testing performance but there is construction nearby so the participants have a harder time concentrating, so they perform more poorly). You can minimize this by controlling the setting.

External Validity

Degree to which research findings provide an accurate description of what happens in the real world (generalizability). Said to be high if the way an empirical study was conducted supports generalizing the results to people and situations beyond those actually studied.

Applied Science

Applying findings from experiments to the “real” world. External validity is very important here. Conducted primarily to address some practical problem (ex. prompted by safety concerns, led to the enactment of law)

Basic Science

Using findings to understand processes and develop/support theory. External validity might not be as important here. Conducted primarily for the sake of achieving a more detailed and accurate understanding of human behavior, without necessarily trying to address any particular problem.

Confounds

Threat to internal validity (3rd variable problem); a variable that systematically changes along with the independent variable (IV) and dependent variable (DV), and therefore could account for the relationship between IV and DV. Present the biggest risk to non-experiments

Person Confounds

Type of confound of experimental design; 3rd variables associated with individual differences that account for relationships between IV and DV.

Environmental Confounds

Type of confound of experimental design; 3rd variables associated with the context/environment that accounts for relationships between IV and DV.

Pseudoexperiment

Exposing people to a variable of interest (IV) and then measuring the outcome variable of interest (DV). (Ex. introduce a drug and them measure IQ)

Pre-Post Test

A design that test participants (DV) before and after a treatment (IV). (Ex. measure IQ, give a drug, measure IQ)

Regression to the Mean

tendency for people who score high/low on a particular measure to score closer to the mean on the next testing, extreme scores are unlikely to be extreme again. Confound

Participant Reaction Biases

Biases that occur when people realize they are being studied (usually threatens internal validity); happens to all experimental designs (experiment and non experiment). Includes Participant expectancies, Participant reactance, and Evaluation Apprehension.

Participant Expectancies

Behaving in ways that are consistent with the experimenter’s hypotheses (as revealed though demand characteristics).

Demand Characteristics

Tells in your study that reveal your hypotheses to the participant.