Decision Making Under Uncertainty
SCIE90011: Decision Making Under Uncertainty
Course and Institutional Metadata
Institution: The University of Melbourne, Faculty of Science.
Course Code: SCIE90011.
Course Title: From Lab to Life.
Presenters:
A/Prof. Greg Kubik.
Dr. Daniel Czech.
Dr. Heshan Peiris.
Land Acknowledgment: The University of Melbourne acknowledges the Traditional Owners of the unceded land: the Wurundjeri Woi-wurrung and Bunurong peoples, the Yorta Yorta Nation, and the Dja Dja Wurrung people. It recognizes histories dating back more than years.
Intended Learning Outcomes
By the end of this lecture, students should be able to:
Describe main sources of uncertainty.
Explain sensitivity analysis in terms of performance measures and interpret its implications.
Distinguish between one-way and scenario-based sensitivity analysis for performance outcomes.
Describe structured challenge techniques and their role in improving design decisions.
Recognise key cognitive biases that distort decision making under uncertainty and outline concrete mitigation tactics.
Types and Sources of Uncertainty
Uncertainty arises from various facets of translational science and project management, categorized as follows:
Technical Performance Uncertainty:
Assay sensitivity and specificity in real samples (as opposed to just buffers).
Device accuracy, precision, and reliability under actual field conditions.
Bioprocess yields, stability, and reproducibility when moving to scale.
Clinical and Biological Variability:
Performance across different patient groups (accounting for co-morbidities, age, and genetic diversity).
Variation in pathogens or cell lines.
Within-patient variability over time.
Usability and Workflow Uncertainty:
Whether users operate the device as intended.
Adherence to protocol correctly.
Performance degradation in "messy" real-world workflows characterized by interruptions and shortcuts.
Safety and Risk Uncertainty:
Rare but severe adverse events.
Biosafety incidents, contamination, and off-target effects.
Economic and Political Uncertainty:
Changes in economic or political priorities.
Macroeconomic shocks affecting staffing levels, maintenance, and replacement cycles.
Public attitudes and political debates.
Categorization of Uncertainty
Aleatory (Inherent) Uncertainty: Unavoidable variation, such as biological noise or random equipment failure. This type cannot be reduced.
Epistemic (Knowledge) Uncertainty: Uncertainty due to a lack of knowledge, resulting from limited sample sizes, limited use cases tested, or early-stage data. This type can be reduced with more information.
Sensitivity Analysis
Sensitivity analysis is applied to determine which uncertainties constitute actual threats and where conclusions are robust versus fragile. It helps decide where to invest effort for learning or redesigning.
Core Elements
Sensitivity analysis investigates a system by looking at a model or mapping from inputs to outputs and asking how changes in inputs affect the outputs.
Types of Sensitivity Analysis
One-way (Univariate) Sensitivity Analysis: Varies one parameter at a time while holding others constant. It is simple, transparent, and identifies which single parameters have the greatest influence.
Multi-way Sensitivity Analysis: Varies two or more parameters simultaneously to capture interactions (e.g., performance and adherence changing together).
Scenario-based Sensitivity Analysis: Defines a small number of coherent scenarios (e.g., "optimistic", "base-case", "challenging"). Each scenario specifies a bundle of parameter values, allowing a comparison of outcomes to assess robustness across different conditions.
Global Sensitivity Analysis: Systematically explores a wide range of parameter combinations, often using sampling methods in complex simulation models.
Structured Challenge for Better Decisions
Because parameter values are defined by humans, there is a residual risk of bias, such as under-discussing risks or focusing only on average performance. Structured challenge ensures alternative perspectives are considered and failure modes are explored.
Devil’s Advocate
A thinking technique where a team member deliberately takes an opposing or critical position to test the strength of an idea.
Execution: Assign one or two members to this task; rotate roles between decisions.
Tasks:
Question the realism of assumptions.
Highlight edge cases.
Argue against an idea or for an alternative.
Pre-mortem Analysis
A team imagines a project has already failed disastrously and works backward to identify the causes. This differs from a post-mortem, which analyzes actual failures after they occur.
Benefits: Reduces overconfidence and groupthink, surfaces hidden assumptions, and generates mitigation strategies.
Steps of Pre-mortem Analysis:
Set the scene: The facilitator states: "It’s months from now. This project has failed badly. Tell me what went wrong."
Silent brainstorming: Individuals write down plausible reasons for the hypothetical failure.
Group discussion: Everyone shares "failure causes," which are clustered into themes (technical, financial, operational, etc.).
Risk ranking: The team prioritizes the most likely and severe potential failures.
Mitigation planning: High-priority risks receive an action plan to avoid or reduce impact.
Important Rule: The failure is assumed; the project is not to be defended during this exercise.
Red Teaming
The deliberate use of an independent group (the "Red Team") to challenge the plans and assumptions of the main group (the "Blue Team").
Independence: The Red Team must be organizationally or intellectually separate to reduce social pressure.
Adversarial Stance: Adopts the role of critic with constructive intent to find weaknesses before they occur in reality.
Structured Probing: Uses frameworks to probe technical/behavioral assumptions, boundary conditions, missing scenarios, and blind spots.
Estimation Techniques: Delphi & PERT
Delphi Technique
A technique to extract and summarize knowledge from a group of experts through an iterative process:
Brief the group on the topic.
Each individual makes a forecast based on their experience.
Results are tabulated and presented.
Participants in the outer quartiles (highest they and lowest estimates) share their reasoning.
Members estimate again after discussion.
Process repeats ( to passes) until a uniform estimate is reached.
PERT (Program Evaluation and Review Technique)
Originally a method for calculating task timelines, it uses probabilistic estimates. It can be combined with the Delphi technique for powerful estimation under uncertainty.
Cognitive Biases in Decision Making
Biases shape judgments in translational science, affecting data selection, assumption ranges, and the interpretation of conflicting evidence.
Common Individual Biases
Confirmation Bias: Seeking and interpreting information that confirms pre-existing beliefs. This includes overweighting early lab results and ignoring negative data.
Overconfidence: Overestimating judgment accuracy and underestimating uncertainty. Example: Predicting sensitivity as a narrow when evidence supports .
Anchoring: Relying too heavily on an initial value. For example, a first prototype's performance or an initial time-to-deployment estimate dictates later planning.
Sampling and Availability Bias:
Sampling Bias: Drawing conclusions from unrepresentative samples (e.g., testing only "easy" patients).
Availability Bias: Judging likelihood based on how easily examples come to mind (e.g., overestimating a rare event due to one memorable case).
Sunk Cost Fallacy: Persisting with an action due to past investment (e.g., continuing a failing assay because money/time has already been spent for years).
Group-Level Biases and Decision Traps
Groupthink: Suppression of dissent to maintain cohesion. Signs include rapid consensus and interpreting silence as agreement.
Polarisation: Group discussion pushing positions to becomes more extreme/over-confident.
Social Loafing and Diffusion of Responsibility: Individuals feeling less responsible for decision quality, resulting in less effort to check assumptions.
HiPPO Effect: The "Highest Paid Person’s Opinion" dominates, leading junior experts to remain silent and decisions reflecting hierarchy over evidence.
Designing Teams to Reduce Bias
Team Composition
Diversity of Expertise: Include biology, engineering, clinical, humanities, and statistics. Clinicians identify workflow issues; engineers detect reliability issues.
Diversity of Seniority: Junior members are often closer to the day-to-day data; senior members provide strategic/regulatory context.
Cognitive and Demographic Diversity: Different backgrounds notice different blind spots.
Psychological Safety and Norms
Psychological Safety: The belief that it is safe to ask questions, admit uncertainty, and offer dissenting views.
Norms: Explicitly state that critiquing ideas is a valued contribution and that changing one’s mind based on evidence is a mark of professionalism.
Techniques for Inclusion
Independent Pre-commitment: Members write opinions down before discussion to reduce anchoring.
Structured Turn-taking: Use round-robin speaking to ensure junior or quieter members are heard and to counter the HiPPO effect.
Anonymous Input Channels: Use surveys to collect concerns, allowing for the expression of minority opinions without social pressure.
Use of Structured Prompts and Checklists: A checklist for major decisions should include:
Listing performance assumptions.
Considering subgroups and non-ideal conditions.
Consulting every discipline and seniority level.
Using a Devil's Advocate.
Discussing what evidence would change the decision.
Shadow Boards
Parallel decision forums comprised of younger or more junior staff. They mirror the main board's remit but provide alternative perspectives and stress-test strategic decisions implementation from a different generational or disciplinary background.