Where Good Ideas Come From: The Natural History of Innovation

Thatcher’s study suggests a counterintuitive notion: the more disorganized your brain is, the smarter you are. It’s counterintuitive in part because we tend to attribute the growing intelligence of the technology world with increasingly precise electromechanical choreography. Intel doesn’t advertise its latest microprocessors with the slogan: Every 55 milliseconds, our chips erupt into a blizzard of noise! Yet somehow brains that seek out that noise seem to thrive, at least by the measure of the IQ test.

Bill Gates (and his successor at Microsoft, Ray Ozzie) are famous for taking annual reading vacations. During the year they deliberately cultivate a stack of reading material — much of it unrelated to their day-to-day focus at Microsoft — and then they take off for a week or two and do a deep dive into the words they’ve stockpiled. By compressing their intake into a matter of days, they give new ideas additional opportunities to network among themselves, for the simple reason that it’s easier to remember something that you read yesterday than it is to remember something you read six months ago.

Old-style browsing does indeed lead to unplanned discoveries. But thanks to the connective nature of hypertext, and the blogospheres exploratory hunger for finding new stuff, it is far easier to sit down in front of your browser and stumble across something completely brilliant but surprising than it is to walk through a library, looking at the spines of books. Does everyone use the Web this way? Of course not. But it is much more of a mainstream pursuit than randomly exploring the library stacks, pulling down books because you like the binding, ever was. This is the irony of the serendipity debate: the thing that is being mourned has actually gone from a fringe experience to the mainstream of the culture.

Google and Wikipedia give those passing hints something to attach to, a kind of information anchor that lets you settle down around a topic and explore the surrounding area. They turn hints and happy accidents into information. If the commonplace book tradition tells us that the best way to nurture hunches is to write everything down, the serendipity engine of the Web suggests a parallel directive: look everything up.

Error often creates a path that leads you out of your comfortable assumptions. De Forest was wrong about the utility of gas as a detector, but he kept probing at the edges of that error, until he hit upon something that was genuinely useful. Being right keeps you in place. Being wrong forces you to explore.

As William James put it, The error is needed to set off the truth, much as a dark background is required for exhibiting the brightness of a picture.” When we’re wrong, we have to challenge our assumptions, adopt new strategies. Being wrong on its own doesn’t unlock new doors in the adjacent possible, but it does force us to look for them.”

Nemeth has gone on to document the same phenomenon at work in dozens of different environments: mock juries, boardrooms, academic seminars. Her research suggests a paradoxical truth about innovation: good ideas are more likely to emerge in environments that contain a certain amount of noise and error. You would think that innovation would be more strongly correlated with the values of accuracy, clarity, and focus. A good idea has to be correct on some basic level, and we value good ideas because they tend to have a high signal-to-noise ratio. But that doesn’t mean you want to cultivate those ideas in noise-free environments, because noise-free environments end up being too sterile and predictable in their output. The best innovation labs are always a little contaminated.

Big organizations like to follow perfectionist regimes like Six Sigma and Total Quality Management, entire systems devoted to eliminating error from the conference room or the assembly line, but it’s no accident that one of the mantras of the Web startup world is fail faster. It’s not that mistakes are the goal — they’re still mistakes, after all, which is why you want to get through them quickly. But those mistakes are an inevitable step on the path to true innovation. Benjamin Franklin, who knew a few things about innovation himself, said it best: “Perhaps the history of the errors of mankind, all things considered, is more valuable and interesting than that of their discoveries. Truth is uniform and narrow; it constantly exists, and does not seem to require so much an active energy, as a passive aptitude of soul in order to encounter it. But error is endlessly diversified.”

A match you light to illuminate a darkened room turns out to have a completely different use when you open a doorway and discover a room with a pile of logs and a fireplace in it. A tool that helps you see in one context ends up helping you keep warm in another. That’s the essence of exaptation.

Legendary innovators like Franklin, Snow, and Darwin all possess some common intellectual qualities — a certain quickness of mind, unbounded curiosity — but they also share one other defining attribute. They have a lot of hobbies.

For John Snow, there were fundamentally different modes of intellectual activity involved in his many projects: building mechanical contraptions to control the temperature of chloroform required different skills and a different mind-set from tending to patients or writing papers for The Lancet. It is tempting to call this mode of work serial tasking, in the sense that the projects rotate one after the other, but emphasizing the serial nature of the work obscures one crucial aspect of this mental environment: in a slow multitasking mode, one project takes center stage for a series of hours or days, yet the other projects linger in the margins of consciousness throughout. That cognitive overlap is what makes this mode so innovative. The current project can exapt ideas from the projects at the margins, make new connections. It is not so much a question of thinking outside the box, as it is allowing the mind to move through multiple boxes. That movement from box to box forces the mind to approach intellectual roadblocks from new angles, or to borrow tools from one discipline to solve problems in another.

In a funny way, the real benefit of stacked platforms lies in the knowledge you no longer need to have. You don’t need to know how to send signals to satellites or parse geo-data to send that tweet circulating through the Webs ecosystem. Miles Davis didn’t have to build a valved trumpet or invent the D Dorian mode to record Kind of Blue. The songbird sitting in an abandoned woodpeckers nest doesn’t need to know how to drill a hole into the side of a poplar, or how to fell a hundred-foot tree. That is the generative power of open platforms.

What happens when you take the distant approaching to reading novels is that you’re able to see patterns that simply aren’t visible on the scale of paragraphs and pages, or even entire books. You could read a dozen silver fork novels and bildungsromans and yet miss the most striking fact revealed by Moretti’s chart: that the diversity of forms is strikingly balanced by their uncannily similar life spans, which Moretti attributes to underlying generational turnover. Every twenty-five to thirty years a new batch of genres becomes dominant, as a new generation of readers seeks out new literary conventions. If you’re trying to understand the meaning of an individual work, you have to read closely. But if you’re interested in the overall behavior of the literary system — its own patterns of innovation — sometimes you have to read from a long way off.

Look at the past five centuries from the long view, and one fact confronts the eye immediately: market-based competition has no monopoly on innovation. Competition and the profit motive do indeed motivate us to turn good ideas into shipping products, but more often than not, the ideas themselves come from somewhere else.

Stephen Jay Gould makes this point powerfully in the allegory of his sandal collection: The wedge of competition has been, ever since Darwin, the canonical argument for progress in normal times, he writes. But I will claim that the wheel of quirky and unpredictable functional shift (the tires-to-sandals principle) is the major source of what we call progress at all scales. The Nairobi entrepreneur selling sandals in an open-air market may indeed be in competition with other cobblers, but what makes his trade possible is the junkyard full of tires waiting to be freely converted into footwear, and the fact that the good idea of converting tires into sandals can be passed from cobbler to cobbler by simple observation, with no licensing agreements to restrict the flow.

Does this mean we have to do away with intellectual property law? Of course not. The innovation track record of the fourth quadrant doesn’t mean that patents should be abolished and all forms of information allowed to run free. But it should definitely put the lie to the reigning orthodoxy that without the artificial scarcity of intellectual property, innovation would grind to a halt. There are plenty of understandable reasons why the law should make it easier for innovative people or organizations to profit from their creations. We may very well decide as a society that people simply deserve to profit from their good ideas, and so we have to introduce a little artificial scarcity to ensure those rewards. As someone who creates intellectual property for a living, I am more than sympathetic toward that argument. But it is another matter altogether to argue that those restrictions will themselves promote innovation in the long run.

You may not be able to turn your government into a coral reef, but you can create comparable environments on the scale of everyday life: in the workplaces you inhabit; in the way you consume media; in the way you augment your memory. The patterns are simple, but followed together, they make for a whole that is wiser than the sum of its parts. Go for a walk; cultivate hunches; write everything down, but keep your folders messy; embrace serendipity; make generative mistakes; take on multiple hobbies; frequent coffeehouses and other liquid networks; follow the links; let others build on your ideas; borrow, recycle, reinvent. Build a tangled bank.