I am very appreciative that Runde, Feduzi, and Cabantous (hereafter RFC) took the time to write a commentary on my recent article, which argued that managers could improve the quality of their decisions by adopting a Bayesian approach.

I begin with my interpretation of areas where we seem to have similar views. Both the original article and the commentary have a positive view of Bayesian updating when decision makers can attach probability values to prior beliefs and beliefs about strength of evidence. I believe RFC and I also share the view that the better the inputs (i.e., the accuracy of the probability estimates), the better the outputs.

Moreover, we have a common belief that the ability to accurately assess probability varies across contexts, and the distinction between risk and uncertainty is relevant to that variance.

I am also hopeful that even if RFC remain convinced of the concerns identified in their commentary related to formal Bayesian updating, they still see many of the benefits to managers associated with thinking informally in Bayesian terms.

RFC identify two areas of key concern.

Assigning definite probabilities
The first is whether Bayesian updating is useful in situations where it is difficult (or impossible) to assign numerically definite subjective probabilities. As RFC note “there are many decision situations without objective probabilities to converge to” and in such cases, there is no guarantee that estimates updated in a Bayesian fashion based on new evidence would be any more accurate than non-updated estimates.

I certainly agree that a lack of objective probabilities is an issue in many real-world decisions. Imagine a manager considering a new product launch in Serbia around 2010. Serbia had applied for European Union (EU) membership in 2009, and a potentially important factor influencing the profitability of a new product launch was whether Serbia would be granted EU candidate status. This is obviously a decision surrounded by many sources of uncertainty. Is it possible to assign and update meaningful probability estimates to a question like that? Well, it turns out that is exactly the type of question that superforecasters from the Good Judgment Project (GJP) tackled as part of the Intelligence Advanced Research Projects Activity (IARPA) tournament. As Tetlock and Gardner (2015) describe, these superforecasters were able to form and update meaningful estimates such that the GJP team soundly beat a control group benchmark and other forecasting teams. These individuals forecast hundreds of events of deep uncertainty and consistently updated those forecasts over time in the light of new information. ¹ The superior performance of these individuals led Tetlock to comment “Subjective probability estimation of messy real-world events is a skill that can be cultivated and is worth cultivating” (Kenny, 2016), and the research pointed to the habit of superforecasters to continually update their probability estimates in the face of new evidence. Overall, although I agree that a lack of objective probabilities is definitely an issue in many managerial decisions, it is not clear to me that this precludes the usefulness of Bayesian updating. The performance of superforecasters suggests there is value in the Bayesian approach even in highly uncertain contexts.

The usefulness of Bayesian updating
RFC’s second concern is that Bayesian updating is inapplicable in cases where a decision maker is unable to define all possible outcomes relevant to a decision. I agree with RFC’s point that “decision makers are often unable to specify in advance all conceivable states of the world.” But, I fail to see why that would necessarily rule out the possibility of engaging in meaningful Bayesian updating. First, a non-trivial number of decision problems can be framed in the dichotomous categorization scheme used in the profitable / non-profitable example in the original article (similarly, a patient either has a condition or not). Certainly, a decision maker might be surprised at the reasons one state of the world or the other ensues, but this would not prevent Bayesian updating. Second, the possibility of surprise states of the world can be accommodated by assigning a non-zero probability to a catch-all state of the world (“something unforeseen happens”). Again, this would still allow estimate updating. That said, this approach does require that decision makers are open to the possibility of surprise and incorporate that awareness into their probability estimates. To the extent that surprises are hard to foresee or decision makers underweight those possibilities, I concur that this raises concerns about the accuracy of the estimation and updating process.

Despite some differences in our views on the above issues, I do share the concern identified by RFC that a risk of following an explicit Bayesian approach and being willing to “trade in numerically definite probabilities” is that it might engender a sense of false confidence in decision makers. Although the original article noted that inputs of low quality lead to outputs of low quality in Bayesian reasoning, it should have emphasized the importance of interrogating the information basis for the inputs and maintaining an appropriate sense of confidence in those estimates. Thank you to RFC for highlighting this important issue.

Finally, I also agree with RFC’s suggestion of two practices outlined at the end of their commentary, specifically, more attention to a) framing of decision problems and b) the social and material aspects of decision making. I view many of the decisions that managers make as attempts to address a problem, and the better that problem is understood, the higher the quality of the decision. I also share RFC’s endorsement of tools that help decision makers update probabilities without explicit instruction in Bayesian reasoning. Indeed, Sedlmeier and Gigerenzer’s (2001) study suggests that people can be effectively trained to update in Bayesian ways through the use of frequency grids and frequency trees. Perhaps where RFC and I see things somewhat differently is whether these more visual approaches are a substitute or complement to more formal applications of Bayesian updating.

In closing, I appreciate the efforts of Runde, Feduzi, and Cabantous to communicate their concerns. As a good Bayesian, I have taken their comments and slightly downgraded my probability estimate for the belief that Bayesian updating is useful even in cases where definitive probability estimates are available. Thanks to RFC for giving me the opportunity to practice what I preach. And, as a final response to the two key concerns raised by RFC, I note that I have managed to do this even considering the uncertainty around this issue and the fact that the RFC commentary came as a surprise to me!

Previously: Runde et al. commentary on McCann’s original article