User Evaluation and Usability Testing in Design

Course Logistics and Final Exam Information

• Final Exam Overview:     * The final exam information is located on the Canvas page under week $16$.     * Structure: The exam consists of $50$ questions and is multiple-choice, following a similar style to the midterm.     * Materials: Students are permitted to bring a note sheet to the exam.     * Conflicts: If a student has a conflict, such as three exams within $24$ hours or another exam scheduled at the same time, an alternate exam is available on the following Tuesday. To utilize this, students must fill out the final exam alternate request form.
• Study Materials: Practice exams, including past finals, are available at the bottom of the Canvas home page to help students test their understanding.

Understanding User Evaluation and Design Philosophy

• Purpose of Evaluation: Evaluation is about creating products that are meaningful and relevant for users, rather than the designers themselves.
• The Expertise Trap: When an individual creates a product (e.g., a website or game), they inherently become an expert on that product. This makes it difficult to view the creation from any perspective other than their own.
• Design Thinking Process: In the overall design thinking framework, user testing occurs toward the end of the lifecycle to gather feedback from the target audience before final implementation.

Defining Usability and Expert Review Methods

• Usability vs. Accessibility: While related, these are distinct concepts. Usability focuses on how effective and efficient a product is, and the sense of satisfaction it provides the user.
• Expert Review-Based Methods:     * Heuristic Evaluation: Utilizes well-established guidelines, such as the $10$ heuristics developed by Jacob Nielsen.     * Process: A team of experts critiques a design based on these specific, established guidelines.
• User Testing-Based Evaluations:     * Contrasted with expert reviews as the "Wild West," these involve actual, non-expert human beings.     * Empirical approach: Evaluations must be empirical and repeatable to determine if iterations actually result in product improvements.

The Five E Model of Usability

• 1. Effective: This measures how completely and accurately a user can reach a specific goal or complete an experience.     * Evaluation Strategy: Designers must identify the specific work or experience to evaluate (e.g., the physical comfort of a phone vs. its camera functionality).     * Metrics: Determined by whether the task is completed or not ($yes/no$) or via scored tests.
• 2. Efficiency: This measures how quickly a task can be completed.     * Supplemental Factors: Beyond time, efficiency can involve mental load (cognitive capacity), resource consumption (money/tokens used), or physical power.
• 3. Engaging: This measures how well the interface draws the user in and how pleasant/satisfying it is to use.     * Relative Context: Success varies by task. For instance, TurboTax moved from a boring linear progress bar (which led to users abandoning the app for a CPA) to a "hub and spoke" model with intermittent breaks, which reduced abandonment rates.     * Metrics: Abandonment rates or stay-time indicate engagement, though specific "satisfaction" ratings can be subjective.
• 4. Error Tolerance: This addresses how well the software prevents user errors or allows for recovery.     * The User Reality: Users often do not read directions (e.g., the IKEA furniture assembly metaphor) and do not devote full mental energy to tasks.     * Designer's Job: Catch mistakes early enough so they do not noticeably impact the user.
• 5. Easy to Learn: This evaluates how much effort is required to master the product.     * Measurement Methods:         * Repetition: Measuring time to complete a task over successive trials to see if completion time decreases.         * Think Aloud: Observing a user and having them vocalize their confusion or thought process during the experience.         * Comparison: Asking users to compare the learning curve to a similar product they have used in the past.

Strategic Application of Metrics

• Contextual Priority: Not all of the "Five Es" are equally important for every product.     * Example (Museum Forensics Exhibit): High importance on being "Engaging"; low importance on "Error Tolerance" (taking a wrong turn doesn't ruin the experience).     * Example (TurboTax): High importance on "Error Tolerance" (to avoid reporting incorrect numbers to the IRS); moderate importance on being "Engaging."

Formative vs. Summative Testing

• Formative Testing:     * Occurs during the design process as various iterations are formed.     * Uses a small number of testers (typically $\text{5–6}$ users). Information gain has diminishing returns beyond this number.
• Summative Testing:     * Occurs at the end of the process on a finished product.     * Typically involves a much larger pool of testers to identify outlier errors (e.g., beta testing software and games).

Components of a Usability Test Plan

• The Why: Establishing specific goals for the study.
• The What (Scope and Task): Defining specifics.     * Horizontal Prototype: Broad scope, shallow functionality (many features, none fully developed).     * Vertical Prototype: Narrow scope, deep functionality (one feature fully developed).
• Scenarios (Play-Acting): Since users aren't usually in the exact moment of need during a test, designers create scenarios.     * Badger Buddies Example: Instead of saying "try to adopt a cat," the scenario might be: "Imagine you are helping your non-computer literate grandmother find cats to save and show her later."     * Effective Scenarios: Match real-life experiences, start generally before moving to specific tasks, and include a sequence of actions.

Execution and Environment

• Personnel Roles:     * Moderator: The single point of contact for the participant to ensure consistency in interaction.     * Note-taker: Directly records factual observations without internal opinion.     * Observers (UX Team): Designers who provide follow-up questions based on their product expertise.     * Technician: Manages equipment (keystroke logging, recordings) and ensures setup is identical for all users.
• Testing Locations:     * Lab Testing: Controlled environment. Designers must make the lab feel realistic (e.g., using a couch to simulate a home interaction) because physical surroundings influence behavior.     * Field Testing: Testing in the user's natural environment (e.g., home or workplace). Uses methods like observation and think-alouds, as structured surveys are harder in the field.

Data Collection and Reporting

• Types of Data:     * Qualitative: Observations, comments, and behaviors. Should remain structured (e.g., comparing Version A vs. Version B).     * Quantitative: Standardized, objective metrics like completion times or standard error reports. Subjective ratings (scales of $1-10$) are often treated as qualitative due to individual variance.
• The Usability Report Structure:     1. Methods: Factual record of how data was collected.     2. Results/Factual Findings: What was found (including qualitative narratives and quantitative graphs).     3. Interpretation: Explanation of what the facts mean (e.g., if user time doesn't decrease, is the product too easy or too hard?).     4. Recommendations: Actionable steps for the next iteration.
• Reporting for Different Audiences:     * Internal Reports (e.g., Western Michigan University style): Factual, demographic-heavy, long records for designers.     * Stakeholder/Executive Reports: Story-driven, highlight-focused summaries that interpret facts quickly (e.g., a bimodal graph showing $19\%$ found navigation "very easy" while others gave it a $4$ or $5$ on a scale of $10$).