M2 - methods & bias

3 kinds of research within cultural psychology

• Cross-cultural validation studies: these test whether an instrument/test/measure that was developed in one cultural context can also be used in other cultural contexts. These studies test if the measure is valid across cultures (i.e. it measures what it intends to measure).

• Indigenous cultural studies: these are focused on providing insights on a target culture by offering rich descriptions of said culture, and they use complex theoretical models to predict and explain cultural differences. The basic philosophy of these studies is that psychological processes and behaviour are best understood within their own cultural environment.

• Cross-cultural comparative studies: psychological constructs are compared between participants from different cultures, to see whether there are cultural differences and what these differences are.

bias vs equivalence (3 types)

Bias = differences that do not have the same meaning within and across cultures → if bias exists in cross-cultural comparative research, the comparison loses its meaning. Bias is not “noise” or random—it’s a systematic difference.

Equivalence = state/condition of similarity in conceptual meaning and empirical method between cultures → this allows a comparison to be meaningful.

There are 3 types of equivalence, in hiërarchical order:

• Construct equivalence = the same construct is measured across cultures.

• Measurement unit (metric) equivalence = scores share similar units of measurement, and when converted, we can obtain equivalent scores (e.g. comparing kilometres and miles).

• Full-score equivalence = there is no bias; everything is comparable.

construct bias (+ model bias)

Construct bias = whether the constructs are defined the same way across cultures (are you studying the same thing in both cultures?).

• Model bias = whether the theoretical framework/hypotheses mean the same thing in all the cultures being tested.

To minimise this bias, you can acknowledge the incompleteness of a construct or sample all its relevant behaviours across cultures.

E.G.: study on happiness in different cultures where participants are asked “how happy do you feel today?” In North-America, people generally derive happiness from personal achievement and try to maximize their positive experiences; in East-Asia, people generally derive happiness from interpersonal connections and try to balance positive and negative affect. The construct “happiness” is vastly different across these two cultures. Therefore, a western-focused questionnaire could underestimate East-Asian people’s happiness.

method bias

Method bias = whether specific methods are equally valid and reliable across cultures → there’s bias when methods are not measuring the same thing or inconsistently across cultures.

Types of method bias are measurement/instrument bias, linguistic/item bias, response bias, sampling bias & procedural/administration bias.

measurement/instrument bias

Measurement/instrument bias = whether the measures/tests/instruments used are equally valid and reliable across cultures.

This is a form of method bias.

stimulus familiarity

Stimulus familiarity = whether participants are familiar with the measurement/stimuli presented during a study. If they’re already familiar, they may perform much better than those who are not familiar with it.

This is a form of method bias.

E.G.: Chinese children outperformed Greek children on visuo-spatial processing tasks because Chinese children have had more visuo-spatial practice by learning how to write Chinese characters. Because of this difference, the task was already easier for Chinese children and harder for Greek children, and therefore there was measurement bias. The measure (the test the kids had to do) didn’t measure viso-spatial processing fairly across the cultures.

linguistic/item bias (different types) + forward/backward translation + committee of experts + cognitive interviewing

Linguistic/item bias = whether research protocols are semantically equivalent across the languages used in the study. An item is biased when it has different psychological meaning across cultures. This is a form of method bias.

• The item is not applicable → an item that has been used in personality tests to assess conscientiousness is “I never make a long trip without checking the safety of my car”. This is only a good item if the respondent has a car; otherwise it’s useless.

• The item has cultural connotations → an item asks whether a person relates to “I do things my own way”. It varies across cultures whether or not this is desirable; in more individualistic cultures, this would be considered a good thing, whilst in collectivistic cultures, this would be considered more of a negative thing. Because of social desirability, this can influence a person’s answer.

• The item cannot be easily translated → the sentence “I feel blue” makes sense in English, but when you literally translate it into Dutch it doesn’t make sense. Figurative/metaphorical language should therefore be avoided if possible.

One way to solve this bias is using forward/backward translation:

• Translator nr.1 translates the item from the original language to the target language.

• Translator nr.2 translates the translated item from the target language back to the original language.

However, this only helps figure out if you’ve used the right literal linguistic equivalent; it doesn’t say anything about the meaning of the item.

A committee of experts can be used to help determine whether an item is properly translated, both literally and meaning-wise, or whether questions should simply be avoided because they are inappropriate/inapplicable in certain cultural contexts.

An additional step is cognitive interviewing: whilst participants are engaging with the material (e.g. filling out the questionnaire), use focus groups or think-aloud protocols to see what they are thinking.

response bias

Response bias = whether people from different cultural groups use the same instrument in the same way and respond in the same way or not → there’s bias if they don’t respond similarly, when this is not caused by cultural differences.

This is a form of method bias.

E.G.: some cultures are more likely to reply with extreme scores, others stick more to the midpoint on a likert-scale.

sampling bias

Sampling bias = whether the samples are appropriate representatives of the culture and equivalent on variables that have nothing to do with cultural differences (no confounding differences). There’s bias if there’s intercultural vairance in certain confounding characteristics of the sample, such as a sample from a wealthy country having an overall higher SES than that from a poorer country.

This is a form of method bias.

E.G.: Educational levels differs across cultures, so keeping education constant might be useful. But because access to education is different across cultures, if you keep education constant, other variables such as wealth / social status can be very different (e.g. little education can still be a privilege in some countries, but can be a sign of poverty in other countries).

procedural/administration bias

Procedural/administraton bias = whether the procedures used to collect data mean/are the same across all cultures → there’s bias if the procedure is experienced differently across cultures.

This is a form of method bias.

E.G.: are the interviewers addressing both cultures the same, or are different interviewers used so each group gets an interviewer from their own culture? Depending on cultural differences, both of these could cause bias.

interpretational bias

Interpretational bias = whether the findings are interpreted correctly.

There’s bias when you interpret data from a differen culture based on your own cuture → this can cause misinterpretation (bias).

4 things you can do against bias (Poortinga, 1989)

• Ignore the non-equivalence (obviously the worst option).

• Preclude the comparison: you don’t make that comparison, e.g. you leave those items out of the results.

• Interpret the non-equivalence: there is clearly a systematic group difference between the cultural groups, so what could have caused this? This is the best option.

• Reduce the effect of the non-equivalence: using statistical techniques to make the non-equivalent data less influential or delete outliers.

exploratory cross-cultural studies (strength + weakness)

Exploratory cross-cultural studies = these studies examine and describe the existence of corss-cultural similarities and/or differences. This is most important in understudies domains/populations.

• Strength: broad scope for identifying similarities/differences.

• Weakness: limited capability to solve the causes of differences.

hypothesis-testing cross-cultural studies (types: structure-oriented, linkage, unpackaging & experiments)

Hypothesis-testing cross-cultural studies = these studies examine why cultural differences exist, based on theoretical models, context vairables and statistical techniques to show whether variables actually account for differences in data.

There are multiple types of hypothesis-testing studies:

• Structure oriented studies = comparison of cultures’ constructs, structures, or relationships; how variables are related across different samples (correlational studies). This can be difficult, because there are many factors to consider that may explain differences beside the variable that you’re interested in.
  E.G.: is a specific style of parenting consistently associated with positive developmental outcomes across multiple cultural samples?

• Linkage studies = try to asses an aspect of culture that is hypothesised to produce cultural differences (the independent variable), and then empirically link this measured aspect of culture with the variable of interest (dependent variable).

  • Unpackaging studies = translates global/unspecific concepts of culture into specific/measurable psychological constructs to explain cultural differences. These constructs are called contextvariables.
    These are quasi-experimental studies, because researcher cannot assign people to be part of a certain culture; they use pre-existing groups.
    E.G.: why are people from Japan so orderly? In Japan natural disasters (earthquakes, tsunamis, etc.) are common, and so they frequently have drills and education on what to do in such situations (which causes orderly behaviour)

  • Experiments = researchers create conditions to establish a cause-effect relationship. Participants are randomly assigned to conditions and results are compared across conditions.

    • Priming studies = participants are exposed to different culturally relevant mindsets, and then their behaviour is compared across conditions.

    • Behavioural studies = involve manipulations of the environment and observation of changes in behaviour.

quantitative vs qualitative research

quantitative = Independent and dependent variables are used, e.g. research in an (quasi-)experimental setting.
In psychology, quasi-experiments are frequently conducted, in which participants are not randomly assigned to an independent variable. It is often difficult to control the variables, and post-hoc cultural interpretations are highly susceptible to bias.
This is useful for hypothesis-testing studies.

qualitative = research is performed within a natural environment or “in the field”. This is subjective and therefore more dependent on interpretation; it’s hard to formalise procedures → this leaves room for bias.
This is useful for exploratory studies.

3 levels of data used in studies

• Individual-level studies = individual participants provide data and are the unit of analysis. It shows how personal cultural orientation affects the person.
  E.G.: a study which measures whether a person is more independent or dependent and how this relates to their cognitive style.

• Ecological (cultural) studies = countries/cultures are the units of analysis. This can also consist of merged/averaged data of individuals, but the subject of study is the culture as a whole.
  E.G.: a study which compares different countries on their socio-cultural value orientation.

• Multilevel studies = involve data collection at multiple levels of analysis.
  E.G.: a study in educational psychology which is interested in how students do in school, using a nested design and comparing data of the individual students, the class, the entire school, and comparing countries.

isomorfism & simpsom’s paradox

Isomorphism = when variables have the same structure, shape, or relationship / the nature of the variable is the same across levels (individual/ecological) → this is what you want to achieve.

Simpson’s paradox = when the relationship between variables is reversed based on which level of data you are researching. This is the oposite of isomorphism (the relationship is reversed depending on invididual or ecological level).

This is illustrated in the figure below. When looking at each individual Simpson, the relationship between the variables is clearly negative. However, when you take the average of each Simpson and start comparing those, the relationship turns positive.

ecological fallacy

Ecological fallacy = when you take the mean of multiple groups (ecological level) and infer individual characteristics from this mean.  

When you are comparing two groups (e.g. two cultures), you’re essentially comparing two distributions. Most people align with the mean scores, but a lot of individuals don’t, and so you can’t say “every Dutch person is exactly like the average” because that is just not true.

Distributions can also overlap: on average, culture X may be more individualistic than culture Y, but part of their distribution may overlap, so there are also individuals from culture Y who actually score higher on individualism than individuals from culture X. Differences within groups can actually be larger than differences between groups. This is illustrated in the figure below

dichotonomous thinking

Thinking in “black and white”, that means only one out of 2 options can be true.

When applying this to cultural characteristics, such as individualism OR collectivism, this is wrong → characteristics should be seen as dimensions, not categories.

cultural essentialism

The beleif that a culture is a central aspect of someone’s personality and determines their characteristics.

This is not true, because their are inter-individual differences within cultures.