Research Methods. Ch 10. Introduction to Simple Experiments

design confound

a threat to *internal validity* in an experiment in which a second variable happens to vary *systematically* along with the independent variable and therefore is an alternative explanation for the results

-If experimenters had accidentally served more appetizing pasta in large bowl, would have design confound because 2nd variable (quality) would have varied along w/ the IV

Coke on hot day Pepsi on cold day

How do you control for design confounds?

You have a control variable - any variable that an experimenter holds constant - hot day, cold day

Experiment example

Wanted to see if plate size effects how much a person eats

-flipped coin and assigned undergraduates to a medium or large size bowl group

-Refilled pasta continuously and weighted plate to see how much they were taking

-Measured at end to see how much they had actually eaten

-Results showed participants took more pasta from the large bowl than medium.(about 140 calories more)

experiment

a study in which one variable is manipulated and one is measured

manipulated variable

A variable in an experiment that a researcher controls, such as by assigning participants to different levels (values)

-Ex: flipping coin to see who takes gets large bowl or medium

measured variable

A variable in a study whose levels (values) are observed and recorded

-Ex: how much pasta people took and how much they ate

independent variable

In an experiment, a variable that is manipulated in a multiple regression analysis, a predictor variable used to explain variance in the criterion variable

-causal variable

-Ex: serving bowl size

Comes first in time

condition

one of the levels of the independent variable in an experiment

-Ex: serving bowl size had 2 conditions, medium and large

dependent variable

In an experiment, the variable that is measured. In a multiple regression analysis, the single outcome, or criterion variable the researchers are most interested in understanding or predicting. Also called a outcome variable.

-outcome variable

-Ex: amount of pasta eaten (grams of pasta taken and calories consumed)

Measured after

control variable

in an experiment, the variable that the researcher holds constant on purpose

*eliminates potential design confounds*

-Ex: quality of the food: all same kind of pasta, size of serving spoon and plate

-Important for internal validity

3 rules needed for causal claims

1) Co-variance (Is IV related to DV)

-size of bowl related to amount eaten (indicated by difference in group means)

2) Temporal precedence (IV comes before DV)

-experimenters manipulated serving bowl size before students served their own pasta

3) Internal validity (no alternative explanations)

-if pasta in larger bowl had been more appetizing than medium bowl

-This experiment could make causal claims because it meets all 3 criteria

comparison group

A group in an experiment whose levels on the independent variable differ from those of the treatment group in some intend, meaningful way

control group

treatment group

control group

A level of an independent variable that is intended to represent "no treatment" or a neutral condition

treatment group

the participants in an experiment who are exposed to the level of the independent variable that involves a medication, therapy, or intervention

placebo group

a control group in an experiment that is exposed to an inert treatment, such as a sugar pill

confound

A general term for a potential alternative explanation for a research finding: a threat to internal validity (confound= "confuse")

3 threats to internal validity

design confounds, selection effects, order effects

systematic variability

in an experiment, a description of when the levels of a variable coincide in some predicable way with experimental group membership, creating a potential confound

-Ex: If generous assistants work only with large-bowl group and reserved ones only with medium-bowl group (would be a confound)

unsystematic variability

in an experiment, a description of when the levels of a variable fluctuate independently of experimental group membership, contributing to variability within groups

can obscure differences in the dependent variable

-Ex: If research assistance demeanor was random across both groups

-would not be a design confound but could make it difficult to detect differences in the DV

selection effect (Between groups)

A threat to internal validity that occurs in an *independent-groups design* when the kinds of participants at one level of the independent variable are *systematically different* from those at another level

-Can happen when experimenters let participants choose which group they want to be in, or if experimenters assign one type of person (all women, or all who sign up early) to one condition, and another type (all men, those who signed up later) to another condition

selection effects make it impossible to tell the reasons for the results

men front of class

random assignment

the use of a random method (e.g., flipping a coin) to assign participants into different experimental groups. (*avoids selection effects*)

spreads differences out more evenly

How does random assignment prevent selection effects?

Assigning participants at random to different levels of the independent variable - by flipping a coin, rolling a die, controls for all sorts of selection effects

every participant has an equal chance of being selectef

matched groups

An experimental design technique in which participants who are similar on some measured variable are grouped into sets; the members of each matched set are then randomly assigned to different experimental conditions. Also called matching (*avoid selection effects*)

-Ex: for experiment on note taking, researchers might match based on GPA by taking 2 highest GPAs and within that matched set, randomly assign to the 2 conditions (notes on laptop or handwritten), then take pair with next highest GPA/s and continue until they are done

-Prevents selection effects while ensuring groups are equal on some important variable before manipulation of the IV

-idea of "treating each participant as his or her own control" means it can be treated as within-groups design

What is a disadvantage of matched groups?

it requires an extra step so needs more time and resources than random assignment

How does using matched groups prevent selection effects?

Matching has the advantage of randomness. Because each member of the matched pair is randomly assigned, the technique prevents selection effects.

independent groups design (Between Groups)

An experimental design in which different groups of participants are exposed to different levels of the independent variable, such that participants experience only one level of the independent variable.

Ex. large bowl condition, medium bowl condition

-Also called between-subjects design or between groups design.

Post test only

pre-test/post-test

within-groups design

One group of participants.

each person is presented with all levels of the independent variable.

-Ex: if experimenters asked participants to take notes twice-once with laptop and once handwritten

-Also called within-subject design.

-Generally requires fewer participants (if wanted 20 people in each condition, independent groups would need a total of 40 but within-groups only need 20 bc they are experiencing all levels of IV)

Concurrent measures

reported measures

posttest only design

An experiment using an independent groups design in which participants are randomly assigned and are tested on the dependent variable only once.

-Also called equivalent groups, posttest-only design (most basic ind. groups design)

-Note taking experiment is posttest-only design

-Satisfies all 3 criteria for causation

pretest/posttest design

An experiment using an independent groups design in which participants are tested on the key dependent variable twice: once before and after exposure to the independent variable.

-Might use when want to show that random assignment made groups equal (no selection effect)

-Not always plausible (for pasta experiment, could not ask people to pretest or they would be too full to do rest of study)

random assignment>verbal gre>mindfulness class> verbal gre

random assignment>verbal gre>nutrition class>verbal gre

can track peoples change in performance over time

repeated measures design

An experiment using a within-groups design in which participants respond to a dependent variable more than once, after exposure to each level or independent variable.

-Ex: To see whether a shared experience would be intensified even when people do not interact with another person, participants both tasted chocolate and then one tasted while a confederate looked at a painting, participant said chocolate was better when confederate was also tasting it even though chocolate was the same both times

-2 levels (sharing and not sharing experience), and participants experienced both levels, so within-groups design. People rated chocolate twice, so repeated-measures design

concurrent measures design

An experiment using a within-groups design in which participants are exposed to all the levels of an independent variable at roughly the same time, and a single attitudinal or behavioral preference is the dependent variable.

-Ex: infants shown male and female face at same time and experimenter recorded which face they looked at longest

-Ensures participants in 2 groups will be equivalent (same participant for both)

-Using each participant as his or her own control

What is an advantage of within groups design?

Ensures participants in two groups will be equivalent

Treating every participant as his or her own control means that a matched group design can be a within group design

requires fewer participants

Can within groups make causal claims?

It can establish covariance

It can establish temporal precedence

May establish Internal validity - no selection effects because each participant has same two conditions. There is order effects and design confounds to rule out

power

The likely hood that a study will show a statistically significant result when an independent variable truly has an effect in the population; the probability of not making a Type II error

order effect

In a within-groups design, a threat to internal validity in which the exposure to one condition changes the participant responses to a later condition.

Confound

Participants later responses are systematically affected by their earlier one

practice effect

a type of order effect in which people's performance improves over time because they become practiced at the dependent measure (not because of the manipulation or treatment)

-Also called fatigue effects (long sequences might lead participants to get better, or tired or bored toward end)

carryover effect

A type of order effect, in which some form of contamination carries over from one condition to the next.

-Ex: sipping orange juice right after brushing teeth

counterbalancing

in a repeated-measures experiment, presenting the levels of the independent variable to participants in different sequences to *control for order effects*

Ex. recruit 30 participants to a study and randomly assign 15 to receive order A then B and assign 15 of them to receive order B then A

full counterbalancing

a method of counterbalancing in which all possible condition orders are represented

-as number of conditions increase, so do the number of possible orders needed for full counterbalancing so hard for larger numbers

when there are two conditions it is easy to counterbalance. two orders A then B, B then A

partial counterbalancing

a method of counterbalancing in which some, but not all, of the possible condition orders are represented. present the conditions in a randomized order for every subject

(ex: computer delivers conditions in new random order for each participant)

Latin square

A formal system of partial counterbalancing that ensures that each condition in a within-groups design appears in each position at least once.

3 main disadvantages to within subjects design

1. order effects (can be managed by counterbalancing)

2. may to be possible or practical. (can't teach a kid to ride a bike twice and measure which one worked better)

3. demand characteristics (catch on to the primary research question)

demand characteristic

a cue that leads participants to guess a study's hypotheses or goals; a threat to internal validity

-also called experimental demand

create alternative explanation for a study's results

What are the two types of within groups design?

concurrent measures and repeated measures

How does counterbalancing improve the internal validity of a within groups design?

because it eliminates order effects when data is collected.

Interrogating construct validity

how well the researchers measured dependent and independent variables

manipulation check

In an experiment, an extra dependent variable researchers can include to determine how well an experimental manipulation worked.

-most likely used when the intention is to make participants think or feel certain ways. manipulate feelings or empathy to see if the operationalization worked as intended

-Ex: if wanted to see how humor would improve students memory of lecture, one group would go to serious and another a funny lecture and then students rated lecture on how "funny" and "light" it was.

pilot study

a study completed before (or sometimes after) the study of primary interest, usually to test the effectiveness or characteristics of the manipulations

-Ex: exposing a separate group of students to either a serious or humorous lecture and then asking them how amusing they found it

When interrogating external validity, you ask about

random sampling

When interrogating internal validity, you ask about

random assignment

If an experimenter has to sacrifice one which one will be sacrificed? Internal or external validity?

sacrifice external for internal validity

In experiments, what do they often use as a different indicator of standardized effect size

d. it takes into account the difference between the means and the spread of scores within the group

When d is large, it usually means the independent variable caused the dependent variable to change for more participants in the study. Overlap less

When d is smaller, it usually means the scores of participants in the two experimental groups overlap more.

Cohen's guidelines for effect size strength

What are three threats to internal validity?

1. Were there any confounds or did some other variable accidentally covary along with the intended independent variable?

2. If an independent group design was used, did they control for selection effects by using random assignment or matching

3. If within groups design was used, did they control for order effects by counterbalancing?

How do manipulation checks provide evidence for the construct validity of an experiment?

Manipulation checks help researchers determine whether the operationalization worked as intended

Why does theory matter in evaluating construct validity?

because interrogating construct validity requires you to evaluate how well the measured and manipulations researchers used in their study to capture the conceptual variables in their theory

Besides generalization to other people, what other aspect of generalization does external validity address?

Generalization to other situations, places and things

What does it mean when an effect size is too large (as opposed to too small) in an experiment?

When the effect size large, it usually means the independent variable caused the dependent variable to change for more participants in the study. There is less overlap

Why does Max's experiment satisfy the causal criterion of temporal precedence?

a. Because Max found difference in rated friendliness between the two conditions?

b. Because the participants shook the experimenter's hand before rating her friendliness? CORRECT ANSWER

c. Because the experimenter acted the same in all conditions, except having cold or warm hands.

d. Because Max randomly assigned people to the warm hand or cold hand condition.

In Max's experiment described above, what was a control variable

a. The participant's rating of the friendliness of the experimenter?

b. The temperature of the experimenter's hands?

c. The gender of the students in the study?

d. The standard greeting the experimenter used while shaking hands? CORRECT ANSWER

What type of design was Max's experiment?

a. posttest-only design CORRECT ANSWER

b. Pretest/posttest design

c. Concurrent measures design

d. Repeated measure design?

Max randomly assigned people to shake hands either with the "warm hands" experiment or the "cold hands" experiment. Why did he randomly assign participants?

a. because he had a within-groups design

b. because he anted to avoid selection effects CORRECT ANSWER

c. because he wanted to avoid order effects

d. because he wanted to generalize the results to the population of students at his university

Which of the following questions would be interrogating the construct validity of Max's experiment?

a. How large is the effect size comparing the rated friendliness of the warm hands and cold hands conditions?

b. How well did Max's "experimenter friendliness" rating capture participant's actual impression of the experimenter? CORRECT ANSWER

c. Were there any confounds in the experiment?

Can we generalize the results from Max's friend to the other experimenters with whom people might shake hands?