Research Methods Exam 2

External validity

extent to which the findings from any one study apply in other settings, at other times, with different participants, when different procedures are used

Generalizability

whether the results of a study apply to settings, populations, treatments, or outcomes that were not included in the study, think “across”

Ecological validity

how well do the results of the study apply to this particular type of patient in this setting? Degree to which the methods, materials, and procedures used in a study mimic the conditions of the natural setting to which they are to be applied, think “to”

Structural component

concerned with how a study is carried out, includes factors such as setting, procedures used, nature of the participant sample. Would the findings remain consistent if they were tested in new studies that used different methods?

Boundary condition

conditions under which the effect operates, do lab setting effects hold up in natural settings?

Functional component

degree to which the psychological processes that operate in a study are similar to the psychological processes at work in a particular setting. How mental processes evoked in a research setting mimic those evoked in a natural setting.

Example - do people in a mock jury research study make the same decisions as real jurors? Likely no because there is no real world outcome

Conceptual component

degree to which the problems studied in research correspond to problems considered important in a natural setting.

Structural Component

Focus is on methodology. Issues addressed is whether the results of research are consistent across variations in settings, procedures, populations, etc, and whether they apply to a particular setting (boundary condition). This component is relevant to generalizability and ecological validity.

Functional component

Focus is on the psychological process. Issue addressed is whether the psychological processes that operate in research settings similar to those operating in applicable natural settings. This component is relevant to ecological validity.

Conceptual component

Focus is on the research question. Issue addressed is whether the research question under investigation is one that is important in the applicable natural setting. This component is relevant to ecological validity.

The structure component wants to generalize across five different dimensions including….

Setting, participants, research procedures, time, and culture

Setting (structural component)

physical and social environment in which the research takes place.

Physical - mirror, air quality, animal in the lab can impact responses

Coparticipant attributes - behavior and characteristics from other participants can influence responses 

Researcher attributes - can influence behavior form other participants’ behavior and characteristics

Participants (structural component)

People from whom the data are collected. 

Convenience sampling - MTurk, college students, have different characteristics compared to the general population

Restricted sampling - restricted to one category of people

Volunteer participants - those who volunteer tend to be different from those who don’t

Person by situation interactions

Research procedures (structural component)

Includes operational definitions, instructions, and tasks. Can be artificial - hypotheticals, short-term tasks that are different from everyday ones, have situations that wouldn’t happen in real life, may not resemble the natural phenomenon. Responses can vary by operational definitions.

Time (functional component)

possibility relationships between variables may change over time as a function of events and social change

Culture (structural component)

Shared history, traditions, and worldview of a social group; unique meaning and information system, shared by a group and transmitted across generations, that allows the group to meet basic needs of survival, pursue happiness and well-being, and derive meaning from life

Responses to stimuli can change responses. Example - Western cultures more susceptible to Muller-Lyer illusion of the same length of line that looks different because of the shape of the end point compared to non-Western cultures.

Differences in value system can change responses. Example - individualistic cultures - focus on self and personal needs and goals whereas collectivist cultures - focus on the needs of the group

Experimental realism

Can reduce reactivity in an experimental study by creating a high degree of realism

Mundane realism

Creating a setting that mimics the natural setting as much as possible

Example - jury research in a mock courtroom of a law school

Person by situation interactions

Different types of people might respond differently to the same IV

Example - event-related potential studies usually only have right-handed people, hand preference relates to asymmetries in the brain and behavior, so this might not generalize to left-handed people

Time factors (structural component)

Time sampling and changes in relationships between variables over time

Time sampling

Some behaviors are more frequent at different times, rise and fall

Example - Car accidents more common in the winter months than summer, so if research was looking at car accidents this is something that would need to be considered

Also take into account time required for an IV to have an effect

Relationship between therapy and patients improving take time, there will be at difference at 26 sessions but maybe not at 2

Chronic manipulations

those found in natural situations and often result from an accumulation of small effects over time, have been experienced for some time, and are expected to continue into the future, such as anxiety, self-esteem, and stress

Acute manipulations

Result from a single event, occur relatively suddenly, and end with the research situation, ethical considerations may require that the manipulation ends then

Example - Threat of an electric shock

Nonspecific treatment effects

other causes of any differences between the conditions, the equivalence between experimental and control groups lets you rule out these effects

Placebo effect

 nonspecific treatment effects are similar to this, people taking part in a drug or psychotherapy study improve because they think they are receiving a treatment even if they are being given an inert substance or are not actually undergoing therapy

Comparison group

Experimenter compares the effects of two conditions rather than having a group with an absence or substitute for treatment

Example - Having a quiz each day after lecture or having a quiz the day before the exam, look at the relationship to exam scores

Manipulation check

 construct validity of a manipulation or IV is tested

Example - high-stress group should report more stress than the low-stress group

Can check through interviews after data collection or include dependent variables that assess the construct being manipulated

Example - had a superman pose as a picture, had participants rate how powerful the person was in the picture

Pilot testing

Test run manipulations to see that they actually work, can work out bugs in the procedures

What are the three defining features of an experiment?

Manipulation of the IV, holding extraneous variables constant, controlling for individual differences

Construct validity (of a manipulation)

A manipulation intended to operationally define a hypothetical construct should accurately represent the construct, may include a manipulation check

Experimental group

participants undergo an experience or treatment that the research thinks will affect the behavior

Control condition

establishes a baseline against which you can assess the effect of the treatment

Reliable manipulation

The manipulation is applied the same way every time, every participant is receiving the manipulation in the same way

Automation - if you don’t need the experimenter present, might do the experiment online

Scripts - have a script for the experimenter to use

Rehearsal - practice and train experimenters

Strong manipulation

Conditions of the IV are different enough to differentially affect behavior

Example - 60 watt vs 75 watt might not find a different but would for 15 watt vs 100 watt

Salient manipulation

A manipulation stands out against everything else that is happening in the experiment; for a manipulation to affect research participants, they have to notice it in the context of everything else that is happening in the experiment

What are characteristics of a good manipulation?

Construct validity, reliability, strength, salience

Stimulus

person, object, or event that represents the operational definition of a condition of the independent variable

Example - study of physical attractiveness impacting perception, IV is physical attractiveness and have picture of attractive or unattractive people

Extraneous variable

variables that are related to or can influence the dependent variable in a study but are not a focus of the research

Example - characteristics of the participants like gender

Between subjects designs (independent groups design)

 participant takes part in either the experimental or control condition, but not both

Simple random assignment

uses a random procedure to determine if a participant will be the experimental group or the control condition; Assume that characteristics across the groups will be the same, larger group size the more likely this is

Quasi-random

Assign participants alternately to the experimental and control conditions as they come to the lab

Example -1st person in experimental, 2nd person in control, 3rd person in experimental

Matched random assignment

Measure the characteristics and then balance group membership on the characteristic

Ex - matched dogs on training history and breed, one member assigned to control or experimental condition

Within-subjects designs (repeated measures design)

each participant takes part in both the experimental and control conditions so that the effects of participant variables are perfectly balanced across conditions

Order effects

participants’ scores on the dependent variable are affected by the order in which they experience the conditions of the independent variable

Practice effects

differences in the dependent variable that result from repeatedly performing the experimental task

Example - May do poorly at first on an unfamiliar task and then improve with practice; Might habituate to background noise (the IV) and it becomes less impactful

Fatigue effects

participants become tired or bored from repeatedly completing the same task

Carryover effects

Effect of one condition of the experiment carries over to and affects participants’ performance in another condition

Ex - reaction time study, experimental group gets caffeine and control condition gets nothing, if you get caffeine first the drug might still be present in the body and affect performance in the control condition

Sensitization effects

form of reactivity, experiencing one condition of the experiment affects performance in another condition

Ex - watch three interview videos, one condition see two great and one average, another condition see two poor and one average, in isolation they had different ratings

Counterbalancing

Half the participants undergo the experimental condition first and the other half undergo the control condition first

• Number of groups for 2 conditions is 2 orders

• Number of groups for 4 conditions is 24 orders

• Increases greatly when you have more conditions

Latin square design

number of orders is equal to the number of conditions, with each condition appearing in each place in the order

Washout period

Period of time over which the effects of a condition dissipate

Ex - for the caffeine study, if the group has caffeine one day then you have them come back for the other condition on another day without consuming any caffeine in the meantime

Differential order effect

order of participation in the conditions has a greater effect when the experimental condition is experienced after the control condition

Factorial design

each independent variable is a factor in the design, uses multiple independent variables

Main effect

Effect of a single independent variable on a dependent variable, ignoring all other independent variables. Of an independent variable represents what you have found if you conducted the experiment using only that IV and ignored the possible effects of the other independent variable

Interaction effects

Whether the different conditions of your independent variable produce results that differ when considering a second variable

Example - exercise or not depends on diet or not when considering your dependent variable

Population parameters

statistics to estimate the characteristics of a given group of people

Sample surveys

participants are selected to produce a sample of respondents whose characteristics closely mirror those of a population from which the sample was drawn

Convenience survey

members of whatever group or groups that the researcher found it convenient to sample from; Cannot be used to estimate population parameters but are commonly used in hypothesis-testing research

Open-ended questions

allow participants to say anything they want and to say it in their own words

Closed-ended questions

require respondents to select one or more responses from a list of choices provided by the researcher

Double barreled question

asks two or more pieces of information at once

Example - Do you think this official should be impeached and compelled to leave office?

Example - Your class functions as a stable, cohesive group, with a balance of leadership, which facilitates learning

Leading question

Implies a certain response is desired

Example - Do you agree that…. Instead say do you agree or disagree that…

Loaded question

uses emotional content to evoke a desired response, such as appealing to social values, such as freedom, and using terms with strong positive or negative connotations

Example - Are you in favor of allowing construction union czars the power to shut down an entire construction site because of a dispute with a single contract, thus forcing even more workers to knuckle under to union agencies?

Numerical rating scales

 respondents assign numerical values to their responses

Anchors

 the meaning of the values in a numerical rating scale are defined by written labels

Sensitivity of measurement

can detect small differences in the level of a variable if the measure is sensitive, if it is insensitive the measure can only detect large differences

Multi-item scale

Composed of two or more items in rating scale format, each of which is designed to assess the same variable; Scores are combined to form an overall average scale score

Subscales

Each measures a component of a multidimensional variable

Likert scales

Summated rating scale, presents respondents with a set of statements about a person, thing, or concept, and has them rate their agreement or disagreement with the statements using a numerical scale that is the same for all the statements, half the statements are worded positively and half are worded negatively

Reverse scores

 numerical values are changed so that a higher number indicates a higher score on the characteristic being assessed

Response bias

participant responds in a particular way for other reasons so their true score is not reflected

Scale ambiguity

numerical or graphic rating scales are used to assess frequency or amount. Frequently or an extreme amount can have different meanings to different people

Category anchoring

Respondents use the amounts provided in the response options as cues for what constitutes an appropriate responses

Example - How long do you watch tv? Starts at up to half an hour and goes to more than 4.5 hours, so people might give lower responses because they assume he 4.5 hour is bad

Social desirability response bias

tendency to respond in a way that makes the respondent look good to others

Self-deceptive enhancement

leads people to respond in a way that puts them in a more positive light than is warranted by their true scores on the construct, but they do so unknowingly

Impression management

people respond in ways intentionally designed to make a positive impression

Primacy effect

people pay more attention to items that come early in a list than those that come late in the list

recency effect

People tend to focus on the items that come late in the list compared to early in the list

Acquiescence response bias

show a tendency to agree are called “yea-sayers” and those who show a tendency to disagree are called “nay-sayers”

Extremity response bias

people tend to give extreme responses on a measure such as by using only the end points on a scale

Cultural response bias

cultural tendency to respond in a certain way on tests or response sets

Literal interpretation fallacy

Taking the meanings of scale anchors at face value even though they have no inherent meaning

Example - rate an automobile using one of five terms - excellent, very good, good, poor, very poor, but the true opinion of some is it is “fair” but they don’t have that option so they pick “good”

Context effect

occurs when responding to one question affects the response to a later question

What are the three ethical principles for research?

Respect for persons, beneficence, justice

Respect for persons

protection for people’s privacy and freedom of choice to participate in research

Beneficence

refers to protecting research participants from harm and is reflected in ethical guidelines for risk-benefit analysis

Justice

 ensuring that research participants are fairly compensated for their participation and that the burdens of research participation and benefits of research are shared by all members

Minimal risk

probability and magnitude of harm or discomfort anticipated in the research are not greater than those ordinarily encountered in daily life

What are 5 categories of risk?

Inconvenience, physical, psychological, social, economic or legal

Risk benefit analysis

Consider anticipated benefits and the anticipated risks, whether benefits outweigh the risks; is subjective because there is no way to meaningfully quantify risks and benefits; can only deal with anticipated costs and benefits; can be problematic since the researcher does the original analysis

Voluntary participation

freedom to decide about participation free from any coercion or excessive inducement and the freedom to withdraw from research without penalty once it has begun

Informed consent

Requires researchers to provide information before data are collected so that participants have all necessary information to decide whether to participate or not

Affirmative assent

need the potential participant to also agree after the parent or guardian says “yes”

Deception

Providing false information about the research purpose or tasks, using confederates who act as another research participants. Used to prevent artificial responses, obtain information participants might be reluctant to give due to embarrassment or defensiveness if they knew the true purpose of the study, may be a IV in the study, create a simulated natural environment, minimize risk of harm

Right to withdraw consent

participants can withdraw from the study, should not be threatened with the loss of any benefits. They may feel an implicit contract with the researcher. Children may feel uncomfortable say no to an adult. Can also decline to answer specific questions with “I prefer not to say”

Debriefing

post-experimental interview, educate participants about the research and explain any deception used. Should take place directly after participation and include crisis hotline information and email link or telephone numbers.

Confidentiality

protecting the data people provide from unauthorized disclosure to safeguard their right to privacy

Anonymity

Not collecting identifying information such as name, identification number, address. Can replace names or other unique identifiers with aliases

Institutional Review Board

Purpose to examine proposed research to ensure it will be carried out ethically. Membership needs 5 members who have varied backgrounds that allow them to look at ethics from diverse viewpoints, must have one member who is a scientist and one who isn’t, one member needs to not be affiliated with the institution, and doesn’t require a representative for specific populations but will often consult with them

Research protocol

 researcher submits this and begins a review process with the IRB, then reviewed for compliance with ethical standards