COMP719-2025-S1-Wk7-1

Cognitive Task Analysis

• Cognitive Task Analysis (CTA) is a family of psychological research methods used to examine and understand the mental processes of people involved in performing a task.

• It investigates:

  • How people think, what they know, and how they apply that knowledge in complex or critical tasks.

  • Mental processes such as decision-making, problem-solving, attention, memory, reasoning.

  • Reasoning and inference making processes, decision-making strategies, and mental models that individuals use to perform complex tasks.

• CTA identifies what is needed and how they think to perform a task, including:

  • Knowledge, automatic and procedural skills, representation of processes, understanding of decision making.

• It is used to match human and system, design controls and displays, and how processes should be portrayed.

• Triangulation, using multiple perspectives, is always very useful.

The Critical Decision Method (CDM)

• The CDM is a CTA technique designed to study how experts make decisions under conditions of high time pressure, uncertainty, and dynamic changes.

• Key features include:

  • Retrospective Interviews: Experts recall specific, non-routine incidents where critical decisions were made.

  • Cognitive Probes: Structured questions explore the reasoning, cues, and situational assessments that influenced their decisions.

  • Focus on Expertise: It aims to uncover the mental models and strategies that differentiate expert decision-makers from novices.

• Based on the Critical Incident Technique.

• CDM was developed in Naturalistic Decision Making (NDM) research, specifically Klein’s Recognition-Primed Decision (RPD) model.

• It describes how decisions were made in real situations, such as by fire ground commanders.

• Probes are centered on decision models, querying about options developed or considered and the basis of their choice.

• CDM provides insight into how experts make decisions, critical judgements, skills, expertise, knowledge, strategies, and use of cues.

• This can guide training design and the design of decision aids/displays.

• Insights, representation design principles, and information structuring analysis lead to display design.

Usage of CDM

• Identifying perceptual cues in Neonatal Intensive Care Units (NICU).

• Identifying the level and nature of expertise in systems analysis and programming.

• Identifying cues for virtual reality interaction.

• Understanding decision making and identifying perceptual needs for aiding decision making in:

  • Military command and control.

  • Emergency ambulance command and control.

  • Electricity dispatch systems.

• Field studies to describe target phenomena.

What CDM is Good For

• Exploring and identifying, not hypothesis testing.

• Descriptive modeling and theoretical conceptualization rather than proving a priori hypotheses; use Grounded Theory approach.

• Focusing on a specific incident, not general trends.

• Focusing on a single incident, unlike Ethnographic studies that collect many details, some useful and some not.

• Investigating personally experienced events, not hypothetical scenarios.

• It is not an "expert review or evaluation" second-guessing someone else’s decisions; it relies on personal experience.

• Conducting individual interviews, not focus groups.

• The ‘brainstorming’ effect can misrepresent the incident; better to triangulate across individuals in a group.

• Analyzing demanding situations, not routine tasks; other tools are more appropriate for describing routine activities.

CDM Advantages

• Cheap: Relatively easy to learn with minimal learning time and effort.

• Minimal equipment: Requires only a tape recorder, A3 sheet, sticky paper, and pencils.

• Portable: Few pieces of equipment make it easy to deploy in field situations.

• Powerful: Useful insights obtainable in a short time, e.g., Level 1 assessment, and from a small number of interviewees.

• Flexible: Easily modified for use in different domains, such as emergency ambulance dispatch, hydro-electric power generation and dispatch, interaction in VR, and knowledge acquisition in operative dentistry.

The CDM Interview

• An open-ended, semi-structured interview method.

• It needs a guide, not a questionnaire.

• Decide on areas to inquire and hence probes to use; prepare it as a guide.

• Have fully written-out questions, but also have prompts about what the questions are about for easy glancing to know where you are at during an interview.

• Pausing too long to read a question can break the flow of an interview discussion/conversation.

• Remember to both steer and document the interview.

• Why?

  • Semi-structured interviews can become un-structured, and following a ‘tight’ questionnaire can be counter-productive.

  • Sometimes you need to follow a line of discussion which could divert and then return to the topic - set questions can become awkward.

  • Questionnaire questions lead to questionnaire answers.

CDM Interview Probes

• Cues: What were you seeing, hearing, smelling?

• Knowledge: What information did you use in making this decision, and how was it obtained?

• Analogues: Were you reminded of any previous experience?

• Goals: What were your specific goals at this time?

• Option: What other courses of action were considered by or available to you?

• Basis: How was this option selected/other options rejected? What rule was being followed?

• Experience: What specific training or experience was necessary or helpful in making this decision?

• Aiding: If the decision was not the best, what training, knowledge, or information could have helped?

• Time pressure: How much time pressure was involved in making this decision? (Scales varied.)

• Situation Assessment: Imagine that you were asked to describe the situation to a relief officer at this point; how would you summarize the situation?

• Hypotheticals: If a key feature of the situation had been different, what difference would it have made in your decision?

CDM Interview Procedure

1. Select incident:

   • Choose a memorable incident that is challenging and demanding, not gory nor morbid.

   • It should be an abnormal situation, operating at limits of performance, not routine.

   • It must be personally experienced, not supervisory.

2. Obtain an Unstructured Account of the Incident:

   • Record events on sticky paper, without concern for sequence or placement at this stage.

   • Re-tell the incident to the interviewee.

   • Invite interviewee to help re-organize the events on a timeline.

   • This encourages interviewee participation, relaxes them, and encourages them to take control and talk.

   • Identify key events and decision points.

   • Verify that the sequence of events is correct.

   • Identify key decision points - the skeleton for further analysis.

3. Construct Incident Timeline.

4. Decision Point Identification.

5. Decision Point Probing.

The CDM Data Analysis

• Qualitative data is gathered:

  • Audio recordings => verbal protocols and written transcripts.

  • Video recordings => observational data => verbal protocols and written transcripts.

  • Written reports.

• Transcribe the interviews.

• Transform data into useful information.

• Extract information to help form conclusions.

• Consider selectivity: What’s the basis for discarding and selection? Bias?

• Extraction of hidden patterns vs verification and proof of models or hypothesis.

CDM Data Analysis: Structured, a priori Approach

• Incident Summary: Synopsis useful in seeing what the key aspects of incident are.

• Decision charts:

  • Document the decision timeline.

  • Progressive deepening to show depth and helps to identify key decisions.

  • Include relevant details that may have arisen from other parts of the interview.

• Tabulate the decisions:

  • Enables systematic analysis and comparison across cases.

  • Especially useful if more than one analyst.

  • Identify and compare similar decisions, common data, themes, and patterns.

  • Compare across cases.

CDM Data Analysis: Emergent Themes Analysis Approach

• Activities, cues, knowledge, difficulties, decision, strategies, representation, and design concepts are identified through narratives and transcripts.

• Broad themes are identified, such as goals and assessments of situation and resources.

• Related excerpts from transcripts are collated.

• Specific themes: Excerpts relating to specific concepts in a theme, e.g., types of activities, examples of cues.

• Identify, Index & Collate -> Structuring & Data Reduction -> Interpret & Conceptualise.

• Index, Structure, Theorise, Repeat: Transcripts -> Broad patterns, e.g., goals -> Decision Process.

• Decision Process: Indexing framework and structure. Collate transcripts to describe process. Structure the result: activities, cues, knowledge, mistakes.

• Emergence: Strategies, themes, issues. Theories emerge which are grounded in the data structure.

From Transcripts into Goals

• Indexing, collating, and drawing out common themes:

  • To get the right number of ambulances to the right place as quickly as possible (ORCON) to ensure a good patient outcome (survivability).

  • To ensure that the rest of the sector continues to perform.

  • To keep critical incidents at the forefront of considerations.

  • To understand what is happening in order to assess the need.

  • To get the right vehicle to the right amount of patients is difficult.

  • To communicate relevant information to the crews as quickly as possible (radio operator).

  • To give [the ambulance crews] the correct information clearly and concisely.

From Goals into Decision Process

• Structuring and Theorising:

  • One overall goal: To get the right ambulances to the right place at the right time.

  • Three sub-goals which correspond with main decision processes in IDM:

    • Assessment of situation

    • Planning

    • Co-ordination and control

  • The other goals set priorities and constraint actions:

    • Minimise disruption to the rest or other sectors.

    • Control other ambulances simultaneously.

    • Pay attention to critical incidents.

From Theorising back to Structure

• Index for next stage of collating:

  • Goal: To get the right ambulance to the right place at the right time.

  • Priorities and constraints: Minimise disruption to rest or other sectors; pay attention to critical incidents (while…); control other ambulances simultaneously.

  • General processes:

    • Assessment of problem: Mental picture, understanding of what is going on.

    • Assessment of resource status: What is available to deal with the problem?

    • Planning: What plans do the situation correspond with? Is the situation novel and require construction of a new plan?

    • Co-ordinating and controlling: Directing and communicating with crews.

    • Situation awareness: What is happening within and around sector.

    • Management of major incidents.

Thematic Analysis using Affinity Diagram

• Affinity Diagram Purpose: Assigning the right KPIs to the right tribe.

• Listing current and possible KPIs.

• Grouping KPIs.

• Labeling a group of KPIs: Attention, Engagement, Customer Performance, Service Quality, Business Model.

From CDM to Design

• Decision Strategy for Search and Planning: Zoom in on incident and circle out to locate next nearest and available ambulance.
  
  

COMP719-2025-S1-L7 User Research 1: Cognitive Task Analysis

Design Thinking Phases

• A non-linear approach to tackle ill-defined or unknown problems involving five phases:

  • Empathize

  • Define

  • Ideate

  • Prototype

  • Test

  • Implement

• Note that the process is not linear.

The Double Diamond Design Thinking Process

• Adapted from the Design Council (2004).

• Involves a cyclical process: User Problem -> Tasks / Activities -> Context Phenomenon -> Design Problem.

HCI Analysis-Design-Evaluate Process

Analysis / User Research (Empathize)

• Understand the User Problem.

• Problem Analysis:

  • Who are the users?

  • What do they do?

  • What's difficult?

  • Why is it difficult?

• User – Task, Activities, Work:

  • What is the primary task?

  • Context, Purpose

  • Work Phenomenon

• Methods:

  • Interviews

  • Task Analysis (TA): HTA, Process chart / process maps, model the problem

  • Cognitive Task Analysis (CTA): cognitive strategies

  • Observation

  • Surveys and Questionnaires

  • User Journey Map

  • Personas

• Understanding Human Capabilities and limits e.g. ABCS Framework

Design (Define & Ideate)

• Define the Design Problem:

  • What will the design do to address the problem and primary task?

• Ideate:

  • Generate several design alternatives

  • Conceptual Design

Prototype

• Rapid prototyping

• Lo-fidelity prototyping

  • Sketching, storytelling

  • Storyboards

  • Wireframes

  • Mock-ups

• Hi-fidelity prototype (interactive)

• Implement as software

Evaluate (Test)

• Evaluate without people (analytic methods)

  • Heuristic Evaluation (expert)

  • Cognitive Walkthrough

  • e.g. Human Working Memory Capacity (7±2)(7±2), GOMS / KLM, PUMA, Fitts Law, etc.

• Evaluate with people

  • Controlled settings

    • Usability Testing

    • Usability Experiments

    • Usability labs

  • Natural settings

    • Living labs

    • Field studies (studying actual users use a device in-situ)

Stage comparison

• Double Diamond stages: Discover, Define, Develop, Deliver

• NN/g Design Thinking stages: Empathise, Define, Ideate, Prototype, Test, Implement