Last week's Supreme Court ruling, today's jobs report, politicians who just don't get it — there's plenty to be gloomy about. But despite all this, I'm an optimist in the medium term. From my vantage point here in Silicon Valley, there's a lot to be hopeful about. As I write at the Wall Street Journal (paywall), all across the commercial and industrial landscape, there are exciting developments that—if America can get its economic policies in order—could set off a burst of growth and wealth creation as big as any we've ever seen in this country. Take, for instance, recent breakthroughs in cloud computing.
In the world of information technology, the big story these days is the shift of data management from largely in-house computing centers to rented, easily scalable computing and storage from anonymous servers located somewhere out in the Internet. Much of this shift, driven by leading providers such as Amazon, is already well under way, rapidly driving down costs and making information management much more affordable both for industry and, increasingly, consumers.
This in turn has kicked off a true revolution in what is being called "big data." Big data is the application of all of this new computing power to reach beyond the individual application of mass information to the mass application of individual data—for instance, by tracking a billion sensors in real time to monitor weather across a continent. It could mean capturing every step in the path of every shopper in a store over the course of a year, or monitoring every vital sign of a patient every second for the course of his illness.
Big data offers measuring precision in science, business, medicine and almost every other sector never before possible. It could ultimately have an impact as great as mass production did more than a century ago—creating a new world of mass personalization of products and services. The big-data revolution is already happening, with hundreds of applications already in use, for instance, tracking millions of chickens from farms in Thailand to family tables around the world, or monitoring the location in real time of every emergency vehicle in a major city like Chicago. Over the next few years, it will spread across every industry and scientific discipline.
Trends like this offer the potential for a golden era. A better world awaits us if we can elect the leaders who are courageous enough to unleash the animal spirits of the market to get us there.