Advanced Research Methods Final Exam

What is a confidence interval?

A range of values likely (95%) to include the true population parameter. For small samples, adjusted-Wald confidence intervals are recommended.
Confidence level, variability, and sample size
Task completion rates
Scores (e.g., from the System Usability Scale) and
Ratings (e.g., from 7-point Likert scales).

What is a recommended post-task question in a usability test?

• Objectivity - Can be verified
• Replicability - Are reliably repeated
• Quantification - Can report quantities
• Economy - Easy to reuse, once developed
• Communication - Promotes a familiar set of questions
• Scientific generalization - Can be generalized
Warning: Post test questions
Post TEST questions:
-Questionnaire for User interaction Satisfaction (QUIS) - First published
-Software Usability Measurement Inventory (SUMI) - Sensitive questionnaire
-Post-Study System Usability Questionnaire (PSSUQ) - For computers and apps
-System Usability Scale (SUS) - Most recommended

How does a user researcher compare experimental data to a benchmark?

Completion rates
Ratings or scores
Task times

What statistical test is best for a between-subjects comparison?

Two-sample t-test or ANOVA

How does one reduce variability in a usability test?

Ensure that test participants understand the tasks
Have them complete trial sessions
Simplify the tasks
Use expert participants, if appropriate
Use a within-subjects experimental design, if appropriate.

When is it acceptable to conduct a usability test with only 5 participants?

Researchers have concluded that 4-5 participants is usually satisfactory to detect problems if they are reasonably detectable.
However 4-5 is appropriate if the user interface is improved and another iteration of tests are conducted

How do you increase the likelihood of discovering usability problems?

Include diverse participants and create realistic task scenarios.
Research UIs that are likely to have problems
Conduct tests with users who represent your target user

What is a recommended post-test set of questions in a usability test?

Use the SUS (System Usability Scale) for overall usability ratings.

What statistical test is best for a within-subjects comparison?

Paired t-test

What should one do when an expected cell value in a Chi-squared test is < 5?

Combine cell values until the cell value is greater than 5. Fisher Exact test, chi squared test

What is an advantage of using an odd number of values in a rating scale?

Allows for a neutral midpoint.
Positive tone
Scale is more conservative
Allows reasonable perception of neutrality

What do the 118 Microsoft Product Reaction Cards measure?

They focus on the aesthetic qualities and attitudes of the product or site.

What are the attributes of the Net Promoter Score?

Made for business Executives. Uses an 11-point scale to classify detractors, passives, and promoters, calculated as.
Promoters: 9, 10; Neutrals/Passives: 7,8; and Detractors: 0,1,2,3,4,5,6
Difficult to judge what is good or bad
Can be tracked over time

Is it acceptable to average ratings from a 7-point number scale?

Yes, provided the scale is treated as interval-like and responses are consistent. Ratings can be summarized as better but not as ratio

What usability test metric is positively correlated with task time?

Error rates. Number of errors.

How does one use regression?

Regression predicts dependent variables based on independent variables by fitting data to a model.

How many stages should one optimize in Don Norman's Theory of Optimizing Stages of Human-Computer Interaction?

Four
1. Intention
2. Selection
3. Execution
4. Evaluation

How does the Keystroke Linear Model predict task times?

More keystrokes mean longer task times. Some have predetermined times so you add them together.

Can one predict a user's search time on a screen as a function of the screen elements?

Screen search time = function(number of character groups, average visual angle of character groups)

When should one perform an analysis of variance (ANOVA)?

When comparing means across three or more groups

What are surveys effective at measuring?

Attitudes, behaviors, characteristics

What are the characteristics of a Likert scale?

A Likert scale measures agreement or frequency on a balanced scale (e.g., 1-7)

What are the characteristics of a Semantic Differential scale?

A scale with opposing adjectives at each end, measuring attitudes (e.g., "Easy-Difficult")

What is a reasonable sample size for a survey on general trends?

Typically 100-300 participants for generalizable results

What is a reasonable first step for analyzing survey responses?

Main analysis tasks
Counting
Comparing
Counting is called tabulating for survey analysis.
Example: cross-tabulation or cross-tab
Explore the raw distributions, and not just the measures of central
tendency (mode, median, mean on next slide)

What cultural factors should one consider when conducting global research?

Is product ownership individual or shared among a family or friends?
Is it permissible to talk directly or alone with women?
How do you show respect to elders in a society?
Is some work seen as being performed only females, males, children?
How do the people in the culture view data privacy?
How likely will users ask for technical support from strangers or a company?
What are the important values for the people?
How collective or individualistic are the people in a culture?
How unusual is it to meet with a participant in their home?

What is a reasonable first step when conducting user research?

Determine the publications the company already know and if you have access and budget then buy more info from other organizations

How should one analyze qualitative comment data?

Record it and make sure you know what they're doing.
Look at all of the comments and group them.

How should one analyze a sequence of tasks?

Review participant comments for each task sequence
Concatenate individual sequences to form an overall task sequence.
Use a spreadsheet to tally how often topics emerge/repeat
Use flowcharts or timelines to map sequences and identify bottlenecks or inefficiencies

What questions do website analytics answer about user behavior?

What users click, time spent, navigation paths, and bounce rates, Click stream analysis, how many pages are being viewed.

What are the characteristics of an A/B test for a website?

A/B test definition - A live-site study in which thousands of visitors
randomly view either the A pages or the B pages.
There should be only one difference between the A and B pages.
Test for statistical significance to analyze any differences in user behavior.
Switch to the better pages, make a change, and start a new A/B test.

What are the characteristics of the Web Content Accessibility Guidelines (WCAG)?

Guidelines ensure content is perceivable, operable, understandable, and robust.
Design Heuristics and now law.

What are the characteristics of a user persona?

Focus personas on abilities, primary goals, and preferences.
Personas can also include demographics, technological expertise, physical and other product environment, values and attitudes, responsibilities. Personas should fit on one slide or one page. Consider ethnicity, age and gender when you choose a first name and picture.
Can be provisional or verified

What are the characteristics of a swimlane diagram?

• Used MS Visio software
• Task: Parts Catalog setup
• Personnel:
• IT manager
• CAD design manager
• 3rd party software developer
• Purpose: Visualize dependencies, find bottlenecks, increase staff utilization, and improve activity delegation
Used to map processes that may involve many different actors an end result

Through what deliverables does a user researcher report the user research findings?

Presentations, written reports, and workshops.
-Reports: Detailed documentation for replicating research or informing future projects.
-Presentations: Summarize findings to facilitate discussions and decision-making.
-Visual representations: Flowcharts, scenarios, and task analyses to contextualize user experiences.
-Highlight videos: Short videos showcasing user successes and challenges during research.
-Workshops: 2-3 hour sessions to prioritize solutions based on research findings

What metric did the case study use concerning the Netflix TV user interface?

Participant Interviews / eye tracking
Observational notes and eye tracking
Observational notes, silent discovery
Dwell time, gaze time, heat map (fixation duration)

What is a usability test best for measuring?

Completion rates, Task success rates, efficiency (ease of use), and user satisfaction. Performance and preference.

What is a focus group best for measuring?

Exploration of user attitudes, preferences, and group dynamics

What is a field visit best for measuring?

Real-world behavior and contextual factors. Can uncover unseen needs and work arounds.

What is a diary study best for measuring?

Longitudinal data about user behavior, thoughts, and preferences. Useful for things over time. Can capture learnability. Is unobtrusive.

What are performance and preference metrics best for measuring?

Performance metrics measure efficiency, task success, and user satisfaction. Preference metrics measure the likability of a product.

What are eye tracking metrics best for measuring?

Gaze patterns and areas of interest. Doesn't measure users' attention.

What are self-reported metrics best for measuring?

Can help show frustration and stress.

What are emotional metrics best for measuring?

Hedonics, engagement, trust, joy, frustration, and surprise, and confidence.

What are combined metrics best for measuring?

More metrics help triangulate a better result. Helps with overall usability.

What is the APA reference format of the author's first and last names?

Authors Last name, First initial

What is the APA reference format of the title concerning letter capitalization?

Capitalize the first letter after punctuation

What is the APA reference format of the journal name concerning letter italics?

Italicize the title and the volume number

What is the APA format for ordering references in the Reference section?

Alphabetical by last name
If there is multiple works front the same author you list them in chronological order.

What is the APA format of ordering citations in a citation?

Put in alphabetical order and separated by semicolons, you put the publication year before you put the semicolon