Usability and Evaluation

What is usability? (3)

Systems should be easy to use, easy to learn and flexible (Shackel, 1990); should engender a good attitude in people; now mainly concerned with the efficiency and effectiveness of systems

What is accessibility? (2)

Removing the barriers that would otherwise exclude some people from using the system at all; designers focus on the demands their designs make on people's abilities

What is the difference between accessibility, usability and acceptability? (3)

Accessibility: removing barriers that exclude people; usability: quality of the interaction (time taken, number of errors, time to become competent); acceptability: fitness for purpose in the context of use

Why is a usable, accessible system not always successful? (2)

It may be assessed as accessible and highly usable; it may still fail to be adopted or to satisfy people (acceptability)

Why is accessibility important? (3)

Access to physical spaces for disabled people is a legal and ethical requirement, increasingly so for information spaces; legislation such as the UK Disability Discrimination Act and US Section 508 requires accessible software; the UN and W3C have declarations and guidelines on access to information

In what ways can people be excluded from using a system? (5)

Physically (location, input and output devices); conceptually (instructions, commands, mental model); economically (cost); culturally (inappropriate metaphors, differences); socially (time, location, social group)

What are the two main approaches to designing for accessibility? (2)

Design for all (universal design); inclusive design

What are the seven principles of design for all? (7)

Equitable use; flexibility in use; simple, intuitive use; perceptible information; tolerance for error; low physical effort; size and space for approach and use

What is the template for explaining a universal design principle? (3)

Name the principle; say what the design must do (e.g. not disadvantage any group); say who it protects or helps (e.g. regardless of ability, experience or conditions)

What are the four premises of inclusive design? (4)

Varying ability is a common characteristic of being human, not a special condition of the few; a design that works well for people with disabilities works better for everyone; self-esteem, identity and well-being are affected by the ability to function with comfort, independence and control; usability and aesthetics are mutually compatible

How is inclusive design different from design for all? (2)

It is a more pragmatic approach; it accepts there are often technical or financial reasons why total inclusion is unattainable

What should designers do to ensure an accessible system? (4)

Include people with special needs in requirements analysis and testing of existing systems; consider whether new features affect users with special needs and note this in the specification; take account of guidelines and evaluate against them; include special needs users in usability testing and beta tests

What is accessibility testing? (3)

A subset of usability testing; checks the application is usable by people with disabilities such as hearing loss, colour blindness, old age and other disadvantaged groups; tests the product with assistive technologies and web compliance testers

What assistive technologies can be used for accessibility testing? (4)

Speech recognition software; screen readers; magnification software; special keyboards

What is WAVE? (2)

Web Accessibility Evaluation Tool (wave.webaim.org); checks the accessibility of websites

What are the characteristics of a highly usable system? (4)

Efficient (people do things with an appropriate amount of effort); effective (has appropriate functions and information, organised appropriately); easy to learn and remember, and safe to operate in its contexts of use; high utility (does the things people want to get done)

What are usability heuristics used for? (3)

Help designers choose between design alternatives; help evaluators find problems in interfaces (heuristic evaluation); several sets exist to choose from (Nielsen, Norman, Tognazzini, Shneiderman)

What are Nielsen's 10 heuristics? (10)

Match the real world; consistency and standards; help and documentation; user control and freedom; visibility of system status; flexibility and efficiency; error prevention; recognition, not recall; error reporting, diagnosis and recovery; aesthetic and minimalist design

How are Nielsen's heuristics grouped? (3)

Learnability (match the real world, consistency and standards, help and documentation); safety (user control, visibility of status, error prevention, recognition not recall, error reporting); efficiency (flexibility and efficiency)

What are Norman's principles? (4)

Affordances; natural mapping; visibility; feedback

What are Tognazzini's 16 first principles? (16)

Anticipation; autonomy; colour blindness; consistency; defaults; efficiency; explorable interfaces; Fitts's law; human interface objects; latency reduction; learnability; metaphors; protect users' work; readability; track state; visible navigation

What are Shneiderman's 8 golden rules? (8)

Consistency; shortcuts; feedback; dialog closure; simple error handling; reversible actions; put the user in control; reduce short-term memory load

What is usability testing? (3)

A technique in user-centred design that evaluates a product by testing it on users; usually systematic observation under controlled conditions to see how well people can use the product; gathering opinions alone is market or qualitative research, not usability testing

What are the types of usability testing? (3)

Expert based; participant based; A-B testing

What is acceptability? (3)

About fitting technologies into people's lives; can only be understood in the context of use; cannot be evaluated in a laboratory (usability can, though with limits)

What is acceptance testing? (2)

Determines whether the software is acceptable to users in the context of use; forms include user acceptance, business acceptance, alpha and beta testing

What is the goal of evaluation? (3)

Validate the design, which is essential to a product's success; make products more usable, easy to learn and intuitive; closely tied to requirements, conceptual and physical design, and prototyping

What are the key decisions when planning an evaluation? (3)

Evaluate early and throughout the design process; choose a method appropriate to the stage; decide who is involved (experts such as usability specialists, or participants such as end users and stakeholders)

Why evaluate early? (3)

Get feedback to inform early design concepts, especially for novel applications (paper or quick software prototypes help); evaluate competitor products or previous versions; decide between design options (e.g. voice input vs touchscreen)

What is formative evaluation? (3)

Evaluation that helps shape the design; identifies usability problems during development; needs a stable version and time left to fix problems

What is summative evaluation? (2)

Assesses the usability of a finished product; checks against in-house guidelines or formal standards such as ISO 9241, or provides evidence of usability a customer requires

What is the difference between horizontal and vertical prototypes for evaluation? (2)

Horizontal: the whole system is not fully operational; vertical: the system is completely functional but only some parts

What is the participatory design approach? (3)

Involves people in the design process; stakeholders help set the goals for the evaluation and this improves eventual uptake and use; avoid evaluating your own designs unless you have no alternative

What are the two methods of expert evaluation? (2)

Heuristic evaluation; cognitive walkthrough

What is heuristic evaluation? (4)

Measures a design against a list of principles, guidelines or heuristics; useful to include a severity rating; scenarios can be a useful addition; useful for formative evaluation only

What is the template for a heuristic evaluation process? (3)

Plan (what to achieve, who the users are, which heuristics); execute (evaluators go through product flows and note issues and suggestions); review (summarise findings, remove duplicates, prioritise, report)

Why use a team of experts for a heuristic evaluation? (2)

One expert cannot find a high number of usability issues; a team of usability experts finds more

What is the template for recording a heuristic problem? (4)

Which heuristic is broken; where in the product it happens; a description of the problem; a severity rating

What is participant evaluation? (3)

Involves real people in the process; ranges from designers sitting with users to observing alone through a two-way mirror; types include cooperative evaluation, participatory heuristic evaluation, co-discovery and controlled experiments

What are the types of participant evaluation? (4)

Cooperative evaluation; participatory heuristic evaluation (users rather than experts); co-discovery (watching and thinking aloud in pairs); controlled experiments

What are the steps of cooperative evaluation? (9)

Write a draft list of tasks from the scenarios; try out the tasks and estimate timings (allow 50% longer per session); prepare a task sheet; get ready for the session; tell participants the system is under test, not them; participants start tasks and give a running commentary; encourage them to keep talking; interview them afterwards; write up notes as soon as possible in a usability report

What should tasks in a cooperative evaluation be like? (3)

Realistic; doable with the prototype; explore the system thoroughly

What are sample questions during a cooperative evaluation? (5)

What do you want to do; what were you expecting to happen; what is the system telling you; why has the system done that; what are you doing now

What are sample questions after a cooperative evaluation? (5)

What was the best or worst thing; what most needs changing; how easy were the tasks; how realistic were the tasks; did giving the commentary distract you

What is the template for a task sheet? (3)

A specific, clear description of the task; any starting information the participant needs; what counts as finished

What is concurrent think aloud? (5)

Users think aloud as they work; nothing they say is wrong; don't interrupt; spontaneous and encourages positive suggestions; can be done in teams

What is retrospective think aloud? (3)

Asks people afterwards what they were thinking; has issues with accuracy; does not interrupt users, so timings are more accurate

What is a controlled experiment? (4)

Appropriate when the designer is interested in particular features (e.g. comparing one design to another); needs careful design and running; establish the independent variable (what you are looking at); establish the dependent variables (how you measure the difference)

What should be considered in a controlled experiment? (4)

Learning effects; effects of different tasks; effects of different background knowledge; a measurable outcome (e.g. number of clicks needed for a task)

What should be planned for a usability lab? (3)

Which types of testing and how often; what space is available (dedicated testing room, observation room, one-way mirror, or a multi-purpose room); who will observe and who is on the team

What is the template for evaluation in practice? (3)

Establish the aims, participants, context of use, state of the technology and scenarios; select methods and metrics, combining an expert review (cheaper) with participant testing (more costly, focused by the expert review); analyse results, document and report back to designers

What is the formula for task success rate? (1)

Total successful attempts divided by total attempts (multiply by 100 for a percentage)

What is the template for calculating task success rate from a results table? (3)

Count the successes across all users and tasks; count the total attempts (users multiplied by tasks); divide successes by attempts and convert to a percentage

Who should take part in an evaluation? (4)

The people who will use the system, identified in the analysis work (e.g. personas); recruit at least three, preferably five; a small number only makes sense for a homogeneous group; for a heterogeneous group, run 3 to 5 people from each group