Qualitative and Quantitative Data Notes

Identifying Quantitative and Qualitative Data

  • Two kinds of evaluation data:

    • Qualitative ("Qual")

      • Focuses on descriptions and experiences.

      • Examples:

        • "Participants reported being confused about how to login."

        • "Many users said that, when using their GPS system, they assumed that 'home' was set to their real home location."

        • "Most users reported finding Task 2 more difficult than Task 4."

    • Quantitative ("Quant")

      • Focuses on numerical data and measurements.

      • Examples:

        • "78% of users could not complete the login without requesting help."

        • "In 10,000 sampled GPS journeys, 0.1% involved the user having to correct the home location coordinates."

        • "On average, users took twice as long to complete Task 2 than to complete Task 4."

Surveys: Quantitative and Qualitative Data

  • Surveys can collect both quantitative and qualitative data.

    • Quantitative: e.g., rating scales (Excellent, Good, Poor)

    • Qualitative: e.g., open-ended questions about opinions

  • Example of quantitative data from a survey: Distribution of students in different courses.

  • Example of qualitative data from survey questions like "What have you liked most about the subject so far?" and "What would you like to change about this subject?"

Observations: Quantitative and Qualitative Data

  • Observations can yield both types of data.

    • Quantitative: direct measurements

      • Time to complete a task.

      • Number of errors made.

      • Number of users who can perform a task.

      • Number of times a task is successfully completed.

      • Offset from a target

    • Qualitative: descriptive insights into user behavior

      • Example: A user's thought process while trying to log in, revealing confusion about multiple login options.

Collecting Quantitative Data

  • Quantitative data involves:

    • Quantity, metrics, numbers.

    • Analysis through statistics.

    • Measurable aspects.

    • Not necessarily non-subjective, but not interpretative.

  • Common quantitative methods:

    • Surveys (with quantitative questions, including Likert scales).

    • Experiments.

    • Task-based evaluation.

What to Measure in Quantitative Data

  • Measures of usability should cover:

    • Effectiveness: Users' ability to complete tasks successfully.

    • Efficiency: Users' ability to perform tasks without expending excessive resources (e.g., time).

    • Satisfaction: Users' subjective reactions to using the system.

      • Measured by the SUS (System Usability Scale).

  • Five basic types of performance metrics:

    • Task success.

    • Time on task.

    • Errors.

    • Efficiency.

    • Learnability.

Types of Performance Metrics

  • Task Success:

    • Tasks must have a clear end state or goal.

    • Binary success (True/False).

    • Consider types of task failure:

      • User gives up.

      • Moderator gives up.

      • User takes too long.

      • User does something incorrectly.

    • Define success levels clearly before you start!

  • Time on Task:

    • Important for frequent and routine tasks.

    • Considerations:

      • When to start/stop the timer.

      • Whether to record only successful attempts or all attempts.

    • Issues:

      • Users may take longer if using a concurrent think-aloud protocol.

      • How to establish a target time.

  • Errors:

    • Errors lead to:

      • Loss of efficiency.

      • Costs to organization or end-user.

      • Task failure.

    • Consider how to collect, measure, analyze, and present errors.

    • Be careful not to “double count” errors.

  • Efficiency:

    • How much effort the user expends to achieve their goal (both cognitive and physical).

    • To consider:

      • What action(s) are to be measured?

      • When does the action start and end?

      • How many ‘meaningful’ actions are required to complete the task?

    • Measuring “backtracking” is common.

  • Learnability:

    • Whether/how users learn to use a new system.

    • Short term learnability:

      • Within the span of a single user session/a few hours or days.

    • Longer term learnability:

      • Users come back to the task after weeks, months, or years.

Types of Data

  • Categorical:

    • Nominal.

    • Ordinal.

  • Numerical:

    • Interval.

    • Ratio.

Categorical Data

  • Nominal Data:

    • Separate categories that do not have a hierarchy or structure.

    • No category is better or worse than another.

    • e.g. "Yes/No" responses; hair color.

    • Analysis: counts, frequencies, proportions.

    • Visualisation: bar charts, pie charts.

      • Example: Favorite animals - 60% cats, 25% dogs, 15% mice.

  • Ordinal Data:

    • Groups with a clear order or hierarchy.

    • The elements are not necessarily equally distanced from each other.

    • e.g. Educational experience (primary -> PhD); "low/med/high".

    • Analysis: (aggregated) counts and frequencies.

      • e.g. 40% of users rated the site “good” or “excellent” (“at least good”).

    • Visualisation: bar charts, pie charts.

Numerical Data

  • Interval Data:

    • Data that is measured along a scale.

    • Each value is of equal distance.

    • No 'true 0 point'; can be negative (e.g. -10 degrees Celsius).

    • e.g. Temperature (C or F); time of day.

    • Analysis: mean, mode, median, standard deviation.

    • Visualisation: Line graphs, histograms.

  • Ratio Data:

    • Data that is measured along a scale.

    • Each value is of equal distance.

    • Has a true 0 point.

    • Can compare, multiply and divide, calculate a ratio.

    • e.g. Height; age; distance.

    • Analysis: mean, mode, median, percentile, interquartile range, standard deviation.

    • Visualisation: Line graphs, histograms.

  • The importance of data types lies in how they can be mathematically manipulated.

    • Ratio data allows for statements such as "a participant can take twice as long on Task A as on Task B," while nominal/categorical data does not permit such proportional comparisons.

Analyzing Quantitative Data

  • Describe the overall tendency:

    • Mean (average), median (the middle value), mode (the most frequent value).

  • Describe the distribution:

    • Range (difference between highest and lowest).

    • Standard deviation (average distance of data point from the mean).

  • Identify relationships:

    • Correlation (how strongly two variables are correlated).

  • Interpreting the results:

    • Look into the results.

    • Propose theories & explanations.

    • Compare to other researchers’ results.

Correlation and Causality

  • Correlation: A and B both happen at the same place.

  • Causality: B happens because A happens. Much harder to demonstrate!

  • Correlation is relatively easy to demonstrate.

  • Example: Storks and Babies:

    • Birth-rate and number of storks correlate.

      • Explanation 1: storks cause children.

      • Explanation 2: children cause storks.

      • Explanation 3: a third unknown aspect causes both (Tertium Quid).

Presenting Quantitative Data

  • Describe:

    • Central tendency – the ‘middle’ of the data.

    • Dispersion or variability – how spread out are the data points?

  • Measures of central tendency:

    • Mean – the average for all the data points.

    • Median – the midpoint in the distribution (half the data are above this, half are below).

    • Mode – the most commonly occurring value.

  • Standard Deviation:

    • How different are the values? (i.e. what is the standard amount that these values deviate from the mean?)

    • A low standard deviation indicates that values are more similar; higher standard deviation shows that they are quite different.

  • How to report these statistics:

    • Where relevant, you will describe your data in the text in your findings, as well as depict them in visualisations.

      • e.g. "On average, users found this website to be easy to use (M=5.84, SD=0.7)".

Small Samples

  • Instead of percentages, present information in tables/sentences using raw numbers.

    • “Two of the seven users were unable to find their bin collection dates."

Outliers:

  • An outlier is a single instance of a finding, something you have observed in one user only.

    • Is this person unlike your target users? (disregard)

    • Is this a problem that is relevant to (some of) your target users? (discuss, but use number not %).

    • Do you need to know more? (note for future testing).

  • Reporting outliers example: discuss the specific case and its potential relevance rather than generalizing with percentages.

Quantitative Data Visualization

  • The Basics of Data Plotting:

    • Make sure there are clear labels for:

      • All variable names (with units) - often on graph axes.

      • Values.

    • Choose a sensible range of values for each variable.

    • Remove anything that is unnecessary.

    • Give each figure a title and a label (e.g. "Fig 1. The number of…").

  • Pie Charts:

    • Pie charts are tempting - but not always appropriate.

      • People aren't good at identifying percentages via slice sizes.

      • Lots of slices can look too busy.

    • If you do use pie charts, totals must add up to 100%.

  • Bar Charts

  • Line Charts

  • Nightingale Rose Charts

Collecting Qualitative Data

  • Qualitative data involves:

    • Words, transcripts.

    • Analysis through coding and sorting.

    • Focus on meaning and experience.

    • Often interpretative.

  • Typical methods:

    • Interviews.

    • Talk-aloud observations.

    • Surveys (open responses).

Analyzing Qualitative Data

  • The Basics:

    • Look for themes and patterns in the data.

    • What did you notice during the study?

    • What did participants say?

    • Consider critical incidents – helps to focus in on key events.

    • Remember the original motivating questions for your research – what were your objectives?

  • Methods for Qualitative Analysis:

    • Generally will involve coding and/or sorting the data.

    • Formal Coding (Codebook analysis).

    • Affinity Mapping.

    • Thematic Analysis.

  • Qualitative Coding:

    • Coding can be:

      • Top-down: using a pre-existing framework to code your data.

      • Bottom-up: coding your data based purely on what the data says (no pre-existing framework).

  • Formal Coding of Usability Problems:

    • Interface Problems:

      • Dissatisfaction about an aspect of the interface.

      • Confusion/uncertainty about an aspect of the interface.

      • Confusion/surprise at the outcome of an action.

      • Physical discomfort.

      • Fatigue.

      • Difficulty in seeing particular aspects of the interface.

      • Having problems achieving a goal.

      • The user has made an error.

      • Unable to recover from error without external help.

      • Suggestion for redesign of the interface.

    • Content Problems:

      • Dissatisfaction about aspects of the content.

      • Confusion/uncertainty about aspects of the content.

      • Misunderstanding of the content (the user may not have noticed this immediately).

      • Suggestion for re-writing the content.

  • Affinity Mapping:

    • Organising your codes across users into distinct clusters to help summarise your data.

    • You might use a codebook to help you generate clusters (top-down), or you might just let the data lead you (bottom-up).

Presenting Qualitative Data

  • First Step: Summarise Your Data:

    • What did you analyse?

    • How did you analyse it?

    • What are your key themes/clusters?

    • What does the reader need to know in order to understand your report?

  • Presenting Your Themes/Clusters:

    • Structure presentation by task or theme.

    • Group related findings together.

    • Present positive findings before problems.

    • Present the highest priority problems first.

    • Finish with “nice to have” suggestions.

  • Presenting Qual Data - Tips:

    • Only make claims that your data can support.

    • Graphical representations may be appropriate for presentation (e.g. visual groups, clusters, hierarchies).

    • Combine the analysis from the various sources (pre-test, post-test, observations, comments).

    • You can use tables & dot points to present information.

    • Use representative quotes!

  • Using Quotes:

    • Use quotes to provide evidence of your claims.

    • Be selective - use the most representative quotes.

    • Explain your quotes!

  • Ethics: Anonymisation:

    • Full disclosure – name participants, provide all details (not generally used).

    • Change or remove names and key information (most commonly used).

    • Change or remove names and key information, rephrase quotes, “rogetise” information (not often used – maximum anonymity).

    • Do not use this data (consent obtained inappropriately or withdrawn).