Delusions of Success: How Optimism Undermines Executives’ Decisions
The Idea in Brief
Most business initiatives fail (e.g., new manufacturing plants, mergers, start-ups).
This high failure rate is attributed to delusional optimism, where potential benefits are overemphasized and costs are underestimated due to cognitive biases and organizational pressures.
To counteract this, reference forecasting is recommended: compare a project's potential outcomes with those of similar past projects for more accurate predictions.
Expansion on Failure Rates:
Delving deeper into the high failure rate, it's crucial to understand that these failures span across various industries and are not isolated incidents. Whether it's a tech startup or a multinational corporation, the risk of a project not meeting its intended goals is substantial.
The underlying reasons for these failures are multifaceted, ranging from inadequate market research to poor execution and unforeseen external factors. Therefore, a comprehensive understanding of potential pitfalls is essential for mitigating risks.
Cognitive Biases and Organizational Pressures
Cognitive biases, such as confirmation bias and the halo effect, can significantly distort decision-making processes, leading to unrealistic expectations and flawed strategies.
Organizational pressures, including internal politics and the pressure to meet short-term financial targets, can further exacerbate the problem by encouraging overoptimism and discouraging dissent.
Reference Forecasting in Depth
Reference forecasting involves gathering data from past projects and using statistical analysis to identify patterns and trends that can inform future predictions.
This approach helps ground expectations in reality by providing a more objective assessment of potential outcomes. By comparing a project's potential outcomes with those of similar past projects, organizations can gain valuable insights into potential risks and rewards.
The Idea in Practice: Rose-Colored Glasses
Cognitive biases contribute to overoptimism:
Anchoring: Initial, positively skewed project proposals bias subsequent analyses, preventing adequate adjustment for potential problems.
Competitor neglect: Ignoring the capabilities and plans of competitors leads to overestimation of market potential and profitability.
Exaggerating abilities and control: Taking credit for positive outcomes while blaming external factors for negative ones leads to an overestimation of one's ability to avoid problems and control outcomes.
Examples:
- Anchoring: Teams often fixate on initial, optimistic estimates, hindering realistic assessments of challenges. - Competitor Neglect: Companies overestimate market share by ignoring competitors' innovations and strategies. - Exaggerating Abilities: The 'we're different' syndrome leads to ignoring historical data.
Organizational pressures also play a role:
Investment is directed toward the most promising (and often overoptimistic) proposals.
Optimism is rewarded, while pessimism is viewed as disloyalty, stifling critical thinking.
Examples:
- Investment Skew: Funding gravitates towards ventures with overly optimistic projections. - Stifled Critical Thinking: Dissent is suppressed, leading to a lack of rigorous evaluation.
Counteracting Cognitive Biases and Organizational Pressures: The Outside View
To counteract cognitive biases and organizational pressures, especially for novel initiatives, the following steps provide an "outside view" to augment the intuitive "inside view."
Select a reference class: Identify past projects similar to the one being planned. For example, a studio executive forecasting film sales might select recent films of the same genre, featuring similar actors, and having comparable budgets.
Further Elaboration:
- Refining Reference Class Selection: Include projects with similar strategic goals, market conditions, and technological requirements. Consider both successes and failures to identify common pitfalls and best practices.Assess the distribution of outcomes: Identify the average and extremes in the outcomes of the reference class projects. Continuing the film example, the executive might find that reference films sold 40 million in tickets on average, with 10% selling less than 2 million and 5% selling more than 120 million.
Further Elaboration:
- Documenting Outcome Distributions: Collect detailed data on project costs, timelines, revenue, and customer satisfaction. Use statistical tools to analyze distributions and identify key performance indicators (KPIs).Predict your project’s position in the distribution: Intuitively estimate where the project would fall along the reference class’s distribution. The studio executive predicts 95 million in sales for the new film.
Further Elaboration:
- Calibrating Intuitive Predictions: Use scenario planning techniques to explore different potential outcomes and their associated probabilities. Engage stakeholders in the prediction process to capture diverse perspectives and insights.Assess your prediction’s reliability: Counteract biased prediction from Step 3. Estimate the correlation between your intuitive prediction and the actual outcome, expressed as a coefficient between 0 and 1 (0 = no correlation; 1 = complete correlation). The studio executive estimates this correlation coefficient as 0.6.
Correct your intuitive estimate: Adjust your intuitive prediction based on predictability analysis. Here, the corrected estimate is 62 million: 95M + [0.6 * (40M – 95M)].
Further Elaboration:
- Implementing Corrective Measures: Develop clear guidelines for adjusting initial estimates based on predictability analysis. Provide training and support to help decision-makers apply these guidelines effectively.
Delusions of Success: How Optimism Undermines Executives’ Decisions
Executives planning major initiatives routinely inflate benefits and downplay costs, setting the stage for failure.
In 1992, Oxford Health Plans initiated a complex computer system. By 1997, the company's stock price plummeted 63% due to system and account issues.
In the early 1980s, several European countries collaborated to develop the Eurofighter. Costs more than doubled, and deployment faced significant delays.
In 1996, Union Pacific's acquisition of Southern Pacific led to operational difficulties, lawsuits, and financial strain.
Most large capital investment projects are late, over budget, and underperforming.
Over 70% of new manufacturing plants in North America close within a decade.
About 75% of mergers and acquisitions fail to deliver expected returns.
Most efforts to enter new markets are abandoned within a few years.
Additional Examples:
Boston's Big Dig: The project increased traffic rather than lessening delay; it cost billions of dollars over budget and took years longer than expected
Standard Economic Theory vs. Flawed Decision Making
Standard economic theory explains failures as rational risks in uncertain situations.
The authors argue that failures result from flawed decision-making due to the planning fallacy where managers make decisions based on delusional optimism rather than a rational assessment of risks and rewards.
Executives overestimate benefits and underestimate costs, ignoring the potential for mistakes.
The Planning Fallacy Expansion:
The planning fallacy isn't merely about making overly optimistic estimates; it involves a systematic underestimation of the time, costs, and risks involved in future tasks while executives overestimate the benefits.
Sources of Overoptimism
Executives' overoptimism stems from:
Cognitive biases: Errors in information processing.
Organizational pressures: Accentuate the positive.
Counteracting these effects involves supplementing traditional forecasting with statistical analysis of analogous efforts.
This outside view provides a reality check on the inside view, reducing the likelihood of disastrous investments.
Rose-Colored Glasses: Cognitive Biases
Most people are highly optimistic, exaggerating their talents and abilities.
A College Board survey showed that most students rated themselves above average in leadership, athleticism, and interpersonal skills.
Attribution errors lead people to take credit for positive outcomes, attributing negative outcomes to external factors.
Executives attribute favorable outcomes to their control (e.g., strategy, R&D) and unfavorable outcomes to external factors (e.g., weather, inflation).
People exaggerate their control over events, discounting the role of luck.
Studies show executives are overoptimistic about capital investments, M&A, and market entries.
Over 80% of start-ups fail to achieve their market-share targets.
Executives overestimate their abilities, leading them to believe they can avoid or overcome potential problems.
Overconfidence Expansion:
This overconfidence contributes to the illusion of control, where individuals believe they have more influence over outcomes than they do, often leading to risky decisions.
Accentuating the Positive: Anchoring
The initial plan accentuates the positive, causing subsequent analysis to skew toward overoptimism.
Anchoring is a strong cognitive bias where people insufficiently adjust estimates away from an initial value.
Executives often set budgets with contingency funds but don't adjust sufficiently for problems and delays.
Competitor Neglect occurs when executives focus on their company's capabilities and plans, neglecting competitors' actions.
Anchoring Expansion:
This bias stems from the human tendency to rely heavily on the first piece of information received (the "anchor") when making decisions.
Organizational Pressure
Every organization has only a limited amount of money and time to devote to new projects. Competition for this time and money is intense, as individuals and units jockey to present their own proposals as being the most attractive for investment. Because forecasts are critical weapons in these battles, individuals and units have big incentives to accentuate the positive in laying out prospective outcomes. This has two ill effects.
First, it ensures that the forecasts used for planning are overoptimistic, which, as we described in our discussion of anchoring, distorts all further analysis.
Second, it raises the odds that the projects chosen for investment will be those with the most overoptimistic forecasts— and hence the highest probability of disappointment.
Other organizational practices also encourage optimism.
Senior executives tend, for instance, to stress the importance of stretch goals for their business units.
This can have the salutary effect of increasing motivation, but it can also lead unit managers to further skew their forecasts toward unrealistically rosy outcomes
(And when these forecasts become the basis for compensation targets, the practice can push employees to behave in dangerously risky ways.) Organizations also actively discourage pessimism, which is often interpreted as disloyalty.
When pessimistic opinions are suppressed, while optimistic ones are rewarded, an organization’s ability to think critically is undermined. The optimistic biases of individual employees become mutually reinforcing, and unrealistic views of the future are validated by the group.
The Outside View
Optimism cannot be eliminated, but it can be tempered through objective forecasting methods.
In 1976, a curriculum development project in Israel estimated completion in 18-30 months. An expert recalled similar projects taking 7-10 years, with a 40% failure rate.
The team ignored this pessimistic information and completed the project in eight years, with limited impact.
The expert made two forecasts:
Inside view: Focusing on the specific project.
Outside view: Examining similar projects' outcomes.
The inside view is intuitive but optimistic, while the outside view (reference-class forecasting) is more accurate.
The Outside View Expansion:
The outside view helps mitigate the planning fallacy by providing a benchmark for project timelines, costs, and potential outcomes based on real-world data.
Contrasting Inside and Outside Views
The inside view focuses on the project's objective, resources, and obstacles.
The outside view ignores project details, examining outcomes of similar projects.
Studies confirm that the outside view leads to more objective forecasts.
Students asked to rate their academic performance expected to outperform 84% of peers. Providing outside-view information reduced this to a still overconfident but more realistic 20%.
Organizations prefer the inside view, but the outside view is more likely to yield realistic estimates, bypassing cognitive and organizational biases.
The outside view is most valuable for novel initiatives where optimism biases are strong.
Integration is Key:
Integrating both inside and outside views provides a balanced perspective. The inside view allows for detailed planning and execution, while the outside view offers a reality check, preventing overconfidence.
Putting Optimism in Its Place
Optimism is beneficial for enthusiasm and resilience but needs balance with realistic forecasts.
Aggressive goals can motivate, but outside-view forecasts should inform commitment decisions.
Separate decision-making roles (realistic outlook) from action-oriented roles (optimistic outlook).
CEOs need both optimism and realism, using the outside view for investment decisions.
Once committed, constant review of success odds can undermine morale.
Cultivate Intellectual Honesty Expansion:
Executives should encourage the open and honest exchange of information, even if it challenges prevailing assumptions or forecasts. This involves creating a culture where dissent is valued and where individuals feel safe to express concerns without fear of reprisal.
How to Take the Outside View - Steps:
Select a reference class:
Balance similarities and differences. Weigh which variables are the most meaningful in judging how the plan will play out.
For example, forecasting film sales; base reference class on recent films in the same genre, starring similar actors, with comparable budgets, and so on.
Choose a class broad enough to be statistically meaningful but narrow enough to be truly comparable to the project at hand.
Assess the distribution of outcomes:
Document the outcomes of the prior projects and arrange them as a distribution, showing the extremes, the median, and any clusters.
Make an intuitive prediction of your project’s position in the distribution:
Based on your own understanding of the project at hand and how it compares with the projects