Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
In the first 15 minutes of the latest Ricochet podcast (Episode #483), Rob said a couple of things that caught my attention. At one point, when talking about our communication- and data-centric technical culture, he suggested that the answers to all our big problems were probably in the wealth of data we’ve collected.
What came to my mind when he said that was the movie WarGames (1983), in which a wayward defense computer is discouraged from initiating Armageddon when it crunches the numbers and concludes that there’s no way to win a nuclear war. Setting aside the question of whether or not that’s a correct conclusion (and I recently re-re-re-watched Dr. Strangelove, in which Buck Turgidson makes a compelling contrary argument, so I’m really not so sure), what the computer in WarGames did was reach a kind of meta-conclusion. A thorough examination of the available information suggested that no good answers could be found.
If our “biggest problems,” whatever those are, can be “solved” by the application of ever more refined policy driven by ever more detailed knowledge, then perhaps Rob is right. But if, as I think is more likely, our biggest problems are complex — complex in a formal, and not casual, sense — then all the vast trove of linked and correlated factoids buried in the deep dungeons of Google’s digital fortress will, upon examination, yield not solutions but misplaced hopes for the next generation of technocratic central planners.
On the other hand, maybe all that data mining will lead to the same sensible conclusion reached by WarGames’ WOPR: that, despite all our data and all our brains, there’s no way to plan our way out of complexity’s maze. I’d be down with that meta-conclusion.
A few minutes later, Rob does it again, when he imagines New York City’s congestion and gridlock optimally managed by a centralized oracular database. As Rob sees it, all that’s missing is a knowledge of the destination toward which each driver is bound: given that, it’s merely a matter of crunching the numbers to determine the optimum trajectory for every vehicle, and voilà, unruly chaos is once again brought to heel by superior math skills.
That sounds nice, but it isn’t without its problems.
First, the specific task of optimal routing falls into a very large class of problems that are believed to be non-polynomial, or NP, in their complexity. This means that, as the number of parts of the problem grow, the difficulty of solving the problem grows much faster. Practically speaking, many, perhaps most, NP problems are not considered strictly solvable computationally; at best, decent approximations can be made.
It’s easy to imagine that anything approaching an optimal solution to New York City traffic routing implemented at the level of individual commuters would be so computationally intensive that one would be better off simply walking while the computers worked on the problem.
But this is all begging the question, because Rob was mistaken in his original assumption that knowing the destinations of all those millions of individuals is sufficient. It isn’t. One also has to know the urgency, the priority, the value placed on punctuality, of each of those millions of individuals. Otherwise, the thousand women off to buy new hats will likely have their transportation needs met at the expense of the three surgeons off to perform emergency transplants. Etc.
What Rob seems to be imagining is that central planning for traffic, lacking as it does the knowledge of the market, will nonetheless be better than central planning for anything else.
What the analysis of big data reveals is that simple rules govern independent actors, and that complex behavior arises from the interplay of those independent actors following their simple, but personalized, rules. That’s useful knowledge — and most useful if it discourages technocratic interventions in complex systems.Published in