Imagine your pharmaceutical company is struggling with an innovation problem related to molecular biology. Who would you seek out for help - a biologist who is an expert in the problem domain or a social anthropologist?

If you are like most, you will consult the biologist. And this is for a good reason; research has provided many strong arguments for why an insider like a biologist rather than an outsider like an anthropologist is more likely to solve your innovation problem. After all, insiders know the territory, which is often viewed as a necessary condition for being able to make a creative contribution to a domain.

While consulting an anthropologist for a molecular biology problem might seem far-fetched, many major breakthroughs in biology have actually originated from unexpected sources including anthropologists. This is often attributed to fresh perspectives of outsiders that might be novel for the problem domain. A case in point outside the field of biology is the Auto-Tune technology. Described as “the invention that changed music forever”, it was developed by a complete outsider to music industry, a petroleum engineer named Harold Hildebrand.

Outsiders versus insiders

In light of these conflicting perspectives and evidence, it remains unclear whether insiders or outsiders are a more promising source for creative breakthroughs. I explored this in my research together with Jan van den Ende. Our intuition was that the answer depends on the ways solutions are explored for the problem; whether outsiders are more innovative than insiders is determined mostly by how they explore solutions for the problem. 

To test our intuition, we collected data from 230 solvers who generated solutions for various innovation problems posted on InnoCentive—one of the largest crowdsourcing platforms globally. We asked solvers about their knowledge distance from the problem domain; they indicated the extent to which the domain is inside or outside their field of expertise. Solvers then indicated their reliance to two specific cognitive search processes: cognitive search variation and cognitive search effort. While the former measured the breadth of domains that solvers have drawn on (ranging from a single domain to a wide variety of domains) when developing the solution, the latter addressed overall hours solvers spent on it (including the time spent thinking about the solution, reading and researching it, and discussing it with other people). Next, we collected archival data on whether a solution is selected as a winning solution in the platform. We analyzed how one’s knowledge distance and cognitive search processes interactively influence the odds of generating a winning solution to a problem.

A problem tackled right is a problem half solved

The results confirmed our expectation: both insiders and outsiders can be a key source of creative breakthroughs as long as they engage in the right cognitive search processes.

The path to breakthrough solutions was entirely different for insiders and outsiders. Insiders were most creative when they engage in a broad search that span across various knowledge domains. For example, insiders to the field of biology benefit from talking to outsiders—such as those who are experts in sociology or computer science—when tackling a thorny innovation problem in their own domain. This is because the theories and solutions in these outside domains have the potential to inspire a novel solution in biology field.

One mechanism by which this happens is analogical transfer—identification and application of analogies (deep structural patterns) from other domains. For example, the theory of bacterial mutation was driven through an analogy between bacteria and slot machines. Likewise, exposure to diverse knowledge from different domains can prompt insiders going beyond the conventional routes of thinking. This could help overcoming widely documented problem of cognitive fixation; experts often struggle to consider alternatives because they tend to fixate on initial ideas triggered by previous experience. In addition, a broad search allows insiders to access additional knowledge elements which can be a basis for forming novel associations between different elements.

In contrast to insiders, outsiders were most innovative when they undertook a focused search. For instance, to solve a chemistry problem, a biology insider should focus on getting familiarized with the field of chemistry (instead of searching broadly across other domains like anthropology or computer science). One explanation is that being able to transfer novel perspectives to a new domain require familiarization with that new domain which will most likely occur when outsiders engage in a narrow search focused on that unfamiliar domain. This way outsiders can turn their lack of knowledge in a domain into an advantage.

For both outsiders and insiders, benefiting from these search processes in generating creative breakthroughs was conditional on one key factor: investment of substantial effort. In other words, regardless of their background, solvers need to devote considerable time to tackle the problem by, for example, thinking, researching or talking about it. Generating breakthroughs actually takes blood, sweat and tear, in sharp contrast to the common portrayal of creativity as a sudden flash of insight.

What happens when both insiders and outsiders engage in the right processes? The results show that outsiders have an edge over insiders. Nevertheless, the results overall suggest that how a problem is tackled is at least as important as knowledge background. Both insiders and outsiders can significantly improve their chances of generating breakthroughs by following the right search processes. It turns out that a problem tackled right is a problem half solved.