A Data Science Central Community
We are ruined by our own biases. When making decisions, we see what we want, ignore probabilities, and minimize risks that uproot our hopes.
What's worse, "we are often confident even when we are wrong," writes Daniel Kahneman, in his masterful new book on psychology and economics called "Thinking, Fast and Slow."
An objective observer, he writes, "is more likely to detect our errors than we are."
The new year will bring plenty of splashy stories about iPads and IPOs. There is a more important theme gathering around us: How analytics harvested from massive databases will begin to inform our day-to-day business decisions. Call it Big Data, analytics, or decision science. Over time, this will change your world more than the iPad 3.
Computer systems are now becoming powerful enough, and subtle enough, to help us reduce human biases from our decision-making. And this is a key: They can do it in real-time. Inevitably, that "objective observer" will be a kind of organic, evolving database.
These systems can now chew through billions of bits of data, analyze them via self-learning algorithms, and package the insights for immediate use. Neither we nor the computers are perfect, but in tandem, we might neutralize our biased, intuitive failings when we price a car, prescribe a medicine, or deploy a sales force. This is playing "Moneyball" at life.
It means fewer hunches and more facts. Think you know something about mortgage bonds? These systems are now of such scale that they can analyze the value of tens of thousands of mortgage-backed securities by picking apart the ongoing, dynamic creditworthiness of tens of millions of individual homeowners. Just such a system has already been built for Wall Street traders.
Crunching millions of data points about traffic flows, an analytics system might find that on Fridays a delivery fleet should stick to the highways— despite your devout belief in surface-road shortcuts.
You probably hate the idea that human judgment can be improved or even replaced by machines, but you probably hate hurricanes and earthquakes too. The rise of machines is just as inevitable and just as indifferent to your hatred.
Business people have been having such fantasies of rationalism for decades. Until the last few years, they have been stymied by the cost of storage, slower processing speeds and the flood of data itself, spread sloppily across scores of different databases inside one company. These problems are now being solved.
"We've just got to the point where the technology really starts to work," says Michael Lynch, chief executive of Autonomy Corp. Hewlett-Packard Co. just spent $11 billion to buy Autonomy, which vacuums up "unstructured data" then applies it to these analytic approaches.
Read full article at http://online.wsj.com/article/SB10001424052970203462304577138961342...