Research methods

Testimonial evidence

Consumer quotes that encourage other people to purchase the product or service

Empiricism

Using evidence from the senses or from instruments that assist the senses as the basis for conclusions. Empiricists are systematic, rigorous and submit their work to be peer evaluated.

Theory-data cycle

Collecting data to test, update or change a theory.

Theory

Ideas about how a set of variables relate to each other.

Falsifiable

Having numerous hypotheses that are testable and a way in which the theory could be proven wrong.

Parsimony

Knowing that the simplest option is most likely the correct one.

Hypothesis

A prediction of what you think will happen in a study, based on theory and evidence.

Null hypothesis

Hypothesis that the variables will have no relation.

Alternative hypothesis

Hypothesis that there is some relationship between the variables.

Preregistration

Before performing the study, experimenters may register their study and hypothesis beforehand.

Data

A set of empirical observations. Unbiased and systematic.

Universalism

Claims are based on scientific merit, not credentials.

Communality

Science is created and shared and belongs to a community

Disinterestedness

Seek to find the truth, regardless of personal belief

Organized skepticism

Science will find its own mistakes and correct them

Basic research questions

Fundamental or pure research driven by curiosity. Does not address a specific and practical problem, but may lay the foundations for that. Intended to develop a body of knowledge.

Translational research questions

Takes the lesson from basic research to test applications in health care, psychotherapy or intervention. Probably doesn’t yet mirror a real-life context.

Applied research questions

Has direct and immediate relevance to the solution of a real-world, practical problem.

Comparison group

Allow us to compare what would happen both with and without the thing that we are interested in (experience often doesn’t have this). Allows for a systematic and unbiased way of comparison.

Confound

Alternative explanation for an outcome. Something that affects an outcome.

Intuition

Accepting ideas because we "feel" they are true or because you have a "gut" feeling

Probabilistic

Findings are not expected to explain all the cases all the time. Behavioural research is probabilistic and allows for some exceptions to the rule.

Availability heuristic

Accepting the first thing that pops into our mind as truth

Present/present bias

Failure to consider appropriate comparison groups that are absent (failure to think about things that are not there).

Confirmation bias

Tendency to only look at information that we already believe to be true.

Bias blind spot

Belief that we are unlikely to be biased, but that other people are.

Authority bias

Accepting ideas because they came from a respected source. Could be drawing from their own experience, which can be biased. They may also be an expert in one area, but not in another.

Variable

Something that changes, may have levels

Measured variable

Observed and recorded

Manipulated variable

Controlled, doing something to the participants to instigate some sort of change

Conceptual variable

Abstract, theoretical concept

Operational variable

To turn a conceptual definition into a measured or manipulated variable.

Constant

Something that does not change

Frequency claim

Describes a particular rate or degree of a single variable. (Ex. How many, how often, etc.)

Association claim

Argue that one level of a variable is likely associated with a particular level of another variable. Involve at least two measured variables. Variables that are associated are said to be correlated. There will be no attempt to control or manipulate the variables.

Causal claim

One of the variables is responsible for changing the other. There will be one measured variable and one manipulated variable. Can only be made when using an experimental study design (Three variables are associated, one variable comes before the other, and no other explanations exist for the relationship).

Validity

Appropriateness of a conclusion or decision. A valid claim would be reasonable, accurate and justifiable. You also need to specify which type of validity we are basing the judgement on.

Construct validity

How well a conceptual variable is operationalized. Do the scores from your measure actually assess the construct that it is supposed to?

External validity

How well the results generalize to, or represent, people or contexts besides those in the original study. Can you generalize the study? This has to do with how the sample was chosen from the population.

Statistical validity

Extent to which a study’s statistical conclusions are accurate and reasonable. Do the study numbers support the claim? Can look at the strength of the association and the precision of the estimated association. Smaller studies will likely be less precise.

Margin of error

Helps us decide how well our sample estimates the true percentage.

Internal validity

The degree of truth around causal claims. Only applicable to causal claims. Can the outcome be attributed to only the intervention or could something else have caused it?

Covariance

Changes in one variable are associated with changes in another.

Temporal precedence

Causal variables must come before the changes in the measured variable.

Independent variable

The causal variable, what we think affects the outcome. Causes a change in the other variable.

Dependent variable

The outcome of interest, what we measure, not manipulated.

Core principles

Respect for persons, concern for welfare, justice

Deception

When a researcher is not completely forthright about a study such that a participant cannot give informed consent. Passive deception = withholding information and active deception = misleading information.

The Three Rs of Animal Research

Replacement: The use of inanimate systems or less sentient animals when possible.
Reduction: A decrease in the number of animals used previously with no loss of information.
Refinement: A change in some aspect of the experiment that results in a reduction of pain, stress or harm to the animal.

Data fabrication

Instead of recording what really happened in a study, the researcher invents data to fit their hypothesis.

Data falsification

Researchers influence a study’s results, perhaps by selectively deleting observations from a dataset or by influencing the subjects to respond in a certain way.

Ways to operationalize a variable

Self report, observational, physiological

Scales of measure

Categorical (may be coded with meaningless numbers) or quantitative (uses numbers with real value).

Ways to rank in quantitative research

Ordinal, interval, ratio

Reliability

Consistency of scores given by a measure

Test/retest reliability

Whether the researcher gets the same results every time they use the measure. Requires time and/or more than one assessment. Must use the same procedure with the same people, just at different points. People are expected to score similarly both times, applying to all measurement types. This is only relevant if it is not expected to change dramatically over time

Interrater reliability

Consistent scores are found no matter who measures the variable, requiring more than one assessor simultaneously. Used when using observational-type assessments. Measures the level of agreement between the two assessors, but is not relevant to self-report questionnaires.

Internal reliability

Participant gives a consistent pattern of answers, no matter how the researcher phrases the question. Requires more than one question measuring the same concept on a self-report questionnaire. Measures whether the scores on the items correlate with each other.

Cronbach's alpha

An average of all the item-total correlations, higher scores are better. Shows how correlated two variables are.

Construct validity

Assesses the quality of measurement responses. Do the scores measure what you’re intending to measure?

Face validity

Subjectively decided whether your operationalization of the variable was effective. Simplest and easiest to achieve, no statistical tests and often based on expert opinion and theory.

Content validity

Measure must capture all parts of the defined construct. No statistical tests for this, often based on expert opinion and theory.

Criterion validity

Scores from a measure are associated with a concrete outcome. Look at how your measure holds up against the ideal measure (gold standard). One of the strongest forms of construct validity, especially important for self-report data, however, it needs a gold standard to compare your measure with. Ex. comparing self-report weight to weight on scale in a doctor’s office.

Convergent validity

Statistically related to other measures of the same or similar constructs. If you expect your responses to be correlated with something else (Ex. Happiness and exercise) they should show that in the results.

Discriminant validity

Statistically unrelated (or weakly related) to constructs that are different from the construct of interest. Ex. Comparing sedentary behavior to exercise, they should show a weak correlation. If they do, your study on exercise would have high discriminant validity.

Forced-choice question

People give their opinion by choosing from two or more options.

Likert scale

People are asked to rate a statement based on the level of agreement or disagreement.

Semantic differential

People are asked to rate a statement based on a scale that is anchored by adjectives.

Leading questions

One whose wording lead people to a particular response because it explains why someone should respond in a certain way.

Negatively worded question

Questions contains negative wording which causes confusion.

Double-barreled question

Asks two questions at once.

Question order

Earlier questions can influence the way people answer later questions.

Response sets

Adopting a consistent way of responding to all the questions, especially at the end of a long questionnaire.

Acquiescence (Yea-saying)

Saying "yes" or "strongly agree" to every item.

Fence-sitting

Answering in the middle of the scale.

Socially desirable responding (faking good)

Giving answers to make yourself look better than you are.

Faking bad

Giving answers to make yourself look bad.

Observational research

Systematically recording others' behaviours. Used in all types of claims.

Observer bias

When observer expectations influence the interpretations of the participants' behaviours or the outcome of the study. Seeing what they want to see.

Observer effects

When observers inadvertently change the behaviour of those they are observing. Also known as expectancy effect.

Clever Hans

Horse that was thought to be able to do math. He was actually picking up on subtle cues from the people who were asking the questions, then getting the correct answer.

Reactivity

A change in behaviour because one knows they're being observed. Can happen in humans and animals.

Population

The entire set of individuals of interest to a researcher.

Sample

A subset of a population.

Sampling

The ways in which people are recruited or selected for a study.

Biased sample

Some members of a population of interest have a much higher chance of being included in a sample than others.

Unbiased sample

All members of the population have an equal chance of being included in the sample.

Sample representativeness

The extent to which the characteristics of a sample reflect those of the population.

Probability sampling (random selection)

Every member of the population of interest have an equal and known chance of being selected for the sample, regardless of convenience or if they're a motivated volunteer. All members of a population are equally represented.

Simple random sampling

Most basic form of probability sampling.

Cluster sampling

When people are already divided into arbitrary groups

Multi-stage sampling

Two random samples are selected (1) random sample of clusters (2) random sample of people within the sample.

Stratified random sampling

Researcher purposefully selects particular demographic categories. Population consists of identifiable groups, or strata.

Oversampling

Researcher intentionally overrepresents one or more groups, a version of stratified random sampling.

Systematic sampling

Using a random number generator, a researcher selects a number at random (Ex. 7) and then counts people in a room. Every 7th person would be included in the sample.

Convenience sampling

Using a sample of people who are easy to contact and readily available participants.

Self selection sampling

When a sample is known to contain only people who volunteer to participate.

Purposive sampling

Used when you want to study certain types f people, so you only recruit that type of participant.

Snowball sampling

A variation on purposive sampling in which participants are asked to recommend other participants for the study. Not representative because the sample isn't random.

Quota sampling

The researcher identifies subsets of the population and then sets a target number from each subset, then the researcher uses nonrandom sampling until the quotas are filled.