MUX - Interviews & Participant Observation

Attitude

What people say (e.g., “Getting at least 8 hours of sleep a day is important to me.”)

Understand or measure people’s stated beliefs, but is limited by what people are aware of and willing to report

Behavior

What people do (e.g., Going to sleep at 2-3AM regularly)

Formation of attitudes in media

People develop opinions and evaluations toward a subject or object.

One example is how algorithms work on curating feeds. These algorithms are complex, curating, selecting, and presenting information, but are often opaque.

These attitudes and beliefs people have about how algorithms work can affect behavior.

Folk theories about algorithmic curation

People often use observations of their own feed to make sense of behavior of algorithms.

People can hold multiple theories even when they can be logically mutually exclusive.

Personal-Engagement Theory

Belief: Your own interactions affect what you see. The more interactions you have (e.g., comment, like), the more someone’s content will show up.

Behavior/Thought: “I’m going to like videos of Harry Potter so I get on DracoTok.”

Global Popularity Theory

Belief: Popular content gets more visibility. Likelihood that content will appear is measured by interactions with others.

Behavior/Thought: “I’m going to unfollow people who produce popular content to ‘make space’ for other content. Or “I’m going to like my own post in the hopes that it will appear in others’ feeds.”

Narcissus Theory

Belief: Similarity to another person affects what you see

Behavior/Thought: “TikTok knows I’m a whimsical person, so it’ll show me fun DIY videos that other whimsical people like.”

OC Theory

Belief: Original self-composed content is shown more than shared content

Behavior/Thought: “If I upload my own photo, it will be seen more than if I repost someone else’s.”

Fresh Blood Theory

Belief: New connections have increased visibility

Behavior/Thought: “I just added this person, so I’m going to see more of their posts now.”

Loud and Quiet Friends Theory

Belief: How frequently someone posts affects how much you see them

Behavior/Thought: “This person posts constantly so I’ll see more of their posts more often.”

Randomness Theory

Belief: Algorithm acts randomly and there is no discernible pattern

Behavior/Thought: “Sometimes, I see these posts and sometimes I don’t, it’s pretty random.”

Control Panel Theory

Belief: User’s own settings influence what the algorithm shows

Behavior/Thought: “I am going to click ‘hide’ AI-generated posts so the algorithm should learn not to show this to me.”

Eye of Providence Theory

Belief: Algorithm knows everything and uses multiple factors to curate feeds (it knows all)

Behavior/Thought: “TikTok knows what I’m interested in, even if I don’t explicitly do anything”

Format Theory

Belief: Type and format of content affects visibility

Behavior/Thought: “Video content does better than pictures, so Instagram will show me more Reels.”

Interviews

Can capture expectations, attitudes, and biases people may hold, and bring forth what is salient in their minds

To capture these, researchers meet with a participant one-on-one to discuss in depth what they think about the topic in question

Structured Interviews

Set list of questions that are asked to every participant, in the same order

Semi-Structured Interviews

Guiding questions that are asked (not necessarily all) that can lead to follow-up questions or expanding on a participant’s response

Unstructured Interviews

No set questions, but open-ended conversation guided by the specific participant

Researcher demands

A factor that may influence how attitudes are reported

When a participant responds according to what they believe the researcher is looking for

Social impressions

A factor that may influence how attitudes are reported

When a participant responds in a way that they believe is socially acceptable

Differences in interpretation

A factor that may influence how attitudes are reported

When a question is framed or phrased in a way that can be interpreted in different or vague ways (which may differ from what the researcher is actually trying to ask)

Order effects

A factor that may influence how attitudes are reported

Questions asked in the beginning and at the end are more salient, and can influence how people respond in subsequent questions

Fatigue

A factor that may influence how attitudes are reported

If there are too many questions, participant can grow tired and stop answering genuinely over time

Memory limitations and access

A factor that may influence how attitudes are reported

Some responses may be buried in memory and not recalled readily. Participants may respond based on what thought happens to come to mind at the moment … which may be influenced by events that happened more recently

Identify the “What”

How to conduct a user interview

Identify what your research goals are

Guiding Questions

How to conduct a user interview

Prepare well-designed, open-ended questions that get people talking and sharing

Consider follow-up questions to gather more details

Example: “Tell me about your experience using Twitter/X”

Piloting Interviews

How to conduct a user interview

Tweak your interviews after running pilots with a friend or family, or a sample target user

This will:

• Identify what questions can be misunderstood or cause confusion

• Help identify new questions that may need to be added

• Help identify questions that should be removed

• Figure out the order of the questions

Start with easy questions

How to conduct a user interview

Start with simple questions that get people comfortable talking

Avoid personal, judgmental questions

How to conduct a user interview

“What was the last book you read” If question assumes a person read a book recently and they didn’t, they may feel stupid

Build Rapport

How to conduct a user interview

People are more likely to talk and let their guard down if they feel relaxed and trust the interviewer. This doesn’t necessarily mean the participants have to like you or think you’re funny

Verbal Cues

• Neutral acknowledging

• Noises to confirm you’re paying attention

• Echoing what participant says

• Adjusting speed, volume, intonation of questions

Nonverbal Cues

• Eye contact

• Nodding

• Raise eyebrows

• Mirroring participant smiles

Follow up and Probing

How to conduct a user interview

Ask follow-up questions and probe responses to help uncover motivations and attitudes

Closed-ended questions

How NOT to conduct a user interview

Example: “Do you like using Twitter/X?”

Super broad questions without any specific direction

How NOT to conduct a user interview

Example: What are your thoughts about the internet? (versus “What are some ways the internet has affected how you communicate with your friends?”

Double-barreled questions

The following are two separate questions:

Example: Is Instagram’s interface intuitive and visually appealing?

Iterative Interviewing

As you interview participants, identify and open-code the themes that emerge, update these themes as you continue interviews

Eventually, you may reach a point where additional interviews do not add substantially new information and you reach a point of saturation

1-7 participants

New codes are identified

Note: These are not “set” numbers

7-15 participants

Recurring themes emerge and themes start solidifying

Note: These are not “set” numbers

15+ participants

Little new information is added (doesn’t necessarily mean you’ve seen everything, but rather no new big themes emerge)

Note: These are not “set” numbers

Field Studies

Researcher studies participant in an environment where they would likely encounter the product being used in a real environment

Contextual Inquiry

Researcher and participant work together in the participant’s own environment to observe participant interacting with product in a natural setting

Marginal Observations

Researchers blend into an environment as natural observers of an activity or event

Example: Researcher rides the bus to watch transit commuters

Full Observations

Researcher becomes complete member of a group, subculture, or culture (sometimes even requiring infiltration or taking on a covert role)

Rarely done due to significant time investments and ethical considerations

Example: Researcher becomes a waitress to study restaurant behavior