Rob Long’s Data-Driven Utopian Dream

 

In the first 15 minutes of the latest Ricochet podcast (Episode #483), Rob said a couple of things that caught my attention. At one point, when talking about our communication- and data-centric technical culture, he suggested that the answers to all our big problems were probably in the wealth of data we’ve collected.

What came to my mind when he said that was the movie WarGames (1983), in which a wayward defense computer is discouraged from initiating Armageddon when it crunches the numbers and concludes that there’s no way to win a nuclear war. Setting aside the question of whether or not that’s a correct conclusion (and I recently re-re-re-watched Dr. Strangelove, in which Buck Turgidson makes a compelling contrary argument, so I’m really not so sure), what the computer in WarGames did was reach a kind of meta-conclusion. A thorough examination of the available information suggested that no good answers could be found.

If our “biggest problems,” whatever those are, can be “solved” by the application of ever more refined policy driven by ever more detailed knowledge, then perhaps Rob is right. But if, as I think is more likely, our biggest problems are complex — complex in a formal, and not casual, sense — then all the vast trove of linked and correlated factoids buried in the deep dungeons of Google’s digital fortress will, upon examination, yield not solutions but misplaced hopes for the next generation of technocratic central planners.

On the other hand, maybe all that data mining will lead to the same sensible conclusion reached by WarGames’  WOPR: that, despite all our data and all our brains, there’s no way to plan our way out of complexity’s maze. I’d be down with that meta-conclusion.

A few minutes later, Rob does it again, when he imagines New York City’s congestion and gridlock optimally managed by a centralized oracular database. As Rob sees it, all that’s missing is a knowledge of the destination toward which each driver is bound: given that, it’s merely a matter of crunching the numbers to determine the optimum trajectory for every vehicle, and voilà, unruly chaos is once again brought to heel by superior math skills.

That sounds nice, but it isn’t without its problems.

First, the specific task of optimal routing falls into a very large class of problems that are believed to be non-polynomial, or NP, in their complexity. This means that, as the number of parts of the problem grow, the difficulty of solving the problem grows much faster. Practically speaking, many, perhaps most, NP problems are not considered strictly solvable computationally; at best, decent approximations can be made.

It’s easy to imagine that anything approaching an optimal solution to New York City traffic routing implemented at the level of individual commuters would be so computationally intensive that one would be better off simply walking while the computers worked on the problem.

But this is all begging the question, because Rob was mistaken in his original assumption that knowing the destinations of all those millions of individuals is sufficient. It isn’t. One also has to know the urgency, the priority, the value placed on punctuality, of each of those millions of individuals. Otherwise, the thousand women off to buy new hats will likely have their transportation needs met at the expense of the three surgeons off to perform emergency transplants. Etc.

What Rob seems to be imagining is that central planning for traffic, lacking as it does the knowledge of the market, will nonetheless be better than central planning for anything else.

What the analysis of big data reveals is that simple rules govern independent actors, and that complex behavior arises from the interplay of those independent actors following their simple, but personalized, rules. That’s useful knowledge — and most useful if it discourages technocratic interventions in complex systems.

Published in Technology
This post was promoted to the Main Feed by a Ricochet Editor at the recommendation of Ricochet members. Like this post? Want to comment? Join Ricochet’s community of conservatives and be part of the conversation. Join Ricochet for Free.

There are 49 comments.

Become a member to join the conversation. Or sign in if you're already a member.
  1. kedavis Coolidge
    kedavis
    @kedavis

    Vince Guerra (View Comment):

    Well, Skynet took the same data as WOPR and came to a different conclusion, it also solved the traffic congestion problem.

    They were evaluating different “equations.”  WOPR was looking for ways for the US/population to survive.  Skynet was looking for ways for IT to survive.

    https://www.youtube.com/watch?v=p7VkjA8IQxE

    • #31
  2. Paul Schinder Inactive
    Paul Schinder
    @PaulSchinder

    Henry Racette (View Comment):

    Yes. The Foundation series, which I loved when I was a kid (Asimov was my favorite author when I was growing up), is unreadable dreck now, both in terms of literary quality (okay, that’s just Asimov) and philosophically.

    But then, Asimov was a self-declared socialist — a lot like Bernie, really. So, surprise.

     

    The way I remember it, the point of the original Foundation trilogy was that psychohistory didn’t work.  Seldon set up the Foundations, and then the Mule came along…

     

    • #32
  3. Gazpacho Grande' Coolidge
    Gazpacho Grande'
    @ChrisCampion

    Henry Racette (View Comment):

    Barfly (View Comment):

    Henry Racette (View Comment):
    By the way, I don’t think Rob Long actually imagines that anything can be perfected, and that utopia lurks in the data. I think he’s smarter than that, and more traditional in his sensibilities. I’m responding to what he said, because a lot of people actually believe that sort of thing. I don’t think he does.

    Misplaced charity. He said it, he made reasoning based on it, why would you think he doesn’t believe it?

    No, I don’t think it’s misplaced. While I often disagree with Rob, he’s never given me the impression that he believes complex processes can be managed, nor that humans are particularly good at doing so when they try. I think he was just riffing on the remarkable potential of networks and our modern data-gathering capabilities, and not really trying to make a serious point.

    Oh, and thanks for the nice comment. ;)

    Tex929rr (View Comment):
    As a kid I loved Isaac Asimov’s science fiction. He posited “psychohistory” which would predict the future of human events through statistical analysis.

    Yes. The Foundation series, which I loved when I was a kid (Asimov was my favorite author when I was growing up), is unreadable dreck now, both in terms of literary quality (okay, that’s just Asimov) and philosophically.

    But then, Asimov was a self-declared socialist — a lot like Bernie, really. So, surprise.

     

    Agree completely on Asimov, now, in retrospect.  The writing itself was agonizing to plow through.  The ideas, as I look back, were pure utopian garbage, socialist in nature, assuming that human behaviors can be managed and controlled and predicted by just the right people with enough computational power.

    Is Trump The Mule?

    • #33
  4. Gazpacho Grande' Coolidge
    Gazpacho Grande'
    @ChrisCampion

    Vince Guerra (View Comment):

    Well, Skynet took the same data as WOPR and came to a different conclusion, it also solved the traffic congestion problem.

     

    Skynet plowed the missiles into the ground?  Airbursts are more effective.

    See?  Failure of computational power.

    • #34
  5. David Foster Member
    David Foster
    @DavidFoster

    When Rose Wilder Lane visited the Soviet Union circa 1920, she was still a Communist.  After she explained the benefits of central planning to a disbelieving village leader, he shook his head sadly and said:

    It is too big – he said – too big. At the top, it is too small. It will not work. In Moscow there are only men, and man is not God. A man has only a man’s head, and one hundred heads together do not make one great big head. No. Only God can know Russia.

    The Soviet Union continued pursuing its dream that one hundred (or ten thousand) heads together would make one great big and brilliant head, in the 1950s and afterwards supplemented for even more brilliance by electronic computers.  In his book Red Plenty…part history, part novel…Francis Spufford tells the story from the standpoint of the people who actually had to try to make it work.  Review.

    • #35
  6. The Reticulator Member
    The Reticulator
    @TheReticulator

    David Foster (View Comment):

    When Rose Wilder Lane visited the Soviet Union circa 1920, she was still a Communist. After she explained the benefits of central planning to a disbelieving village leader, he shook his head sadly and said:

    It is too big – he said – too big. At the top, it is too small. It will not work. In Moscow there are only men, and man is not God. A man has only a man’s head, and one hundred heads together do not make one great big head. No. Only God can know Russia.

    The Soviet Union continued pursuing its dream that one hundred (or ten thousand) heads together would make one great big and brilliant head, in the 1950s and afterwards supplemented for even more brilliance by electronic computers. In his book Red Plenty…part history, part novel…Francis Spufford tells the story from the standpoint of the people who actually had to try to make it work. Review.

    I listened to that recently on Audio. I wouldn’t ordinarily read or listen to something that is “part novel,” but it was in my audio queue. As usual, I wondered who had recommended it, as I can never remember those things. I figured it was somebody from Ricochet, but I didn’t remember who. I’ll bet it was you.

    But yes, scenes from that book came to mind as I was reading the OP and the comments here.

    • #36
  7. Phil Turmel Inactive
    Phil Turmel
    @PhilTurmel

    Paul Schinder (View Comment):

    Henry Racette (View Comment):

    Yes. The Foundation series, which I loved when I was a kid (Asimov was my favorite author when I was growing up), is unreadable dreck now, both in terms of literary quality (okay, that’s just Asimov) and philosophically.

    But then, Asimov was a self-declared socialist — a lot like Bernie, really. So, surprise.

     

    The way I remember it, the point of the original Foundation trilogy was that psychohistory didn’t work. Seldon set up the Foundations, and then the Mule came along…

     

    I’m re-reading the trilogy right now (almost done), thanks to a prod here on Ricochet recently.  Your memory is faulty.  The Mule was dealt with. (No further spoilers.)

    But Henry’s assessment of the trilogy largely matches mine.  I don’t think it’s unreadable, though, just juvenile.

    • #37
  8. Cal Lawton Inactive
    Cal Lawton
    @CalLawton

    What are you talking about? Women don’t wear hats anymore.

    • #38
  9. OccupantCDN Coolidge
    OccupantCDN
    @OccupantCDN

    Henry Racette (View Comment):

    OccupantCDN (View Comment):

    More data doesnt equal more science. Look at climate “Science” its got all kinds of data to look at, but they end up selecting the data points that support their conclusions and ignoring anything else. The data is used to support the theory instead of the theory describing the data.

    The inaccuracies with their predictions are not caused by the data selection, its caused by a lack of fundamental understanding in how the atmosphere works and how it distributes heat – until we understand it and model it correctly more data will not help.

    I do think there is a lot broken in climate science right now. I think it’s an open question whether any amount of data and solid theory would allow long-term predictions to be made with significant accuracy. Chaotic systems defy reliable modeling.

    In other words, it isn’t a matter of weak data or bad models: even if those problems are addressed, complexity frustrates prognostication.

    Thats fair. Modeling the atmosphere to produce extremely long term forecasts may not be possible.

    I think its funny, that most people wont put the umbrella in the car on the strength of a 10 day forecast, demand the government spend 10s of Trillions on the strength of a 100 year forecast.

    • #39
  10. David Foster Member
    David Foster
    @DavidFoster

    Henry Racette: First, the specific task of optimal routing falls into a very large class of problems that are believed to be non-polynomial, or NP, in their complexity. This means that, as the number of parts of the problem grow, the difficulty of solving the problem grows much faster.

    We are about to get a real-world test of simultaneous routing of a lot of vehicles, with real-time timing constraints…

    Discussions of flying cars and widespread drone delivery rarely seem to address the problems of *traffic* and *air traffic control*.  Imagine ten thousand or a hundred thousand airborne vehicles, with a diversity of origins and destinations, in a dense metropolitan area.  

    There is work underway on this problem, sponsored I believe by both the FAA and NASA as well as some individual corporations.  Assuming that the tracking and communication problems can be solved (I don’t think the tracking can be done by radar and probably not by the relatively new ADS-B system, either), the routing problem remains.  The good news is that an optimal solution is *not* necessary; near-optimum is surely sufficient.  The bad news is that the solution needs to be extremely failure-resilient.

    • #40
  11. kedavis Coolidge
    kedavis
    @kedavis

    David Foster (View Comment):

    Henry Racette: First, the specific task of optimal routing falls into a very large class of problems that are believed to be non-polynomial, or NP, in their complexity. This means that, as the number of parts of the problem grow, the difficulty of solving the problem grows much faster.

    We are about to get a real-world test of simultaneous routing of a lot of vehicles, with real-time timing constraints…

    Discussions of flying cars and widespread drone delivery rarely seem to address the problems of *traffic* and *air traffic control*. Imagine ten thousand or a hundred thousand airborne vehicles, with a diversity of origins and destinations, in a dense metropolitan area.

    There is work underway on this problem, sponsored I believe by both the FAA and NASA as well as some individual corporations. Assuming that the tracking and communication problems can be solved (I don’t think the tracking can be done by radar and probably not by the relatively new ADS-B system, either), the routing problem remains. The good news is that an optimal solution is *not* necessary; near-optimum is surely sufficient. The bad news is that the solution needs to be extremely failure-resilient.

    One of the reasons why traffic collisions etc are only as bad as they are, is because of GRAVITY.  And FRICTION.  Flying cars have the potential to cause so much more carnage than what we have now, people don’t even seem to suspect or imagine.

    • #41
  12. kedavis Coolidge
    kedavis
    @kedavis

    Phil Turmel (View Comment):

    Paul Schinder (View Comment):

    Henry Racette (View Comment):

    Yes. The Foundation series, which I loved when I was a kid (Asimov was my favorite author when I was growing up), is unreadable dreck now, both in terms of literary quality (okay, that’s just Asimov) and philosophically.

    But then, Asimov was a self-declared socialist — a lot like Bernie, really. So, surprise.

     

    The way I remember it, the point of the original Foundation trilogy was that psychohistory didn’t work. Seldon set up the Foundations, and then the Mule came along…

     

    I’m re-reading the trilogy right now (almost done), thanks to a prod here on Ricochet recently. Your memory is faulty. The Mule was dealt with. (No further spoilers.)

    But Henry’s assessment of the trilogy largely matches mine. I don’t think it’s unreadable, though, just juvenile.

    The even larger problem, really, was that the “psychohistory” predictions snapped right back to being correct again.  Ridiculous.  But if Trump IS The Mule, then for sure Democrats are hoping that Asimov was right!

    • #42
  13. kedavis Coolidge
    kedavis
    @kedavis

    OccupantCDN (View Comment):

    Henry Racette (View Comment):

    OccupantCDN (View Comment):

    More data doesnt equal more science. Look at climate “Science” its got all kinds of data to look at, but they end up selecting the data points that support their conclusions and ignoring anything else. The data is used to support the theory instead of the theory describing the data.

    The inaccuracies with their predictions are not caused by the data selection, its caused by a lack of fundamental understanding in how the atmosphere works and how it distributes heat – until we understand it and model it correctly more data will not help.

    I do think there is a lot broken in climate science right now. I think it’s an open question whether any amount of data and solid theory would allow long-term predictions to be made with significant accuracy. Chaotic systems defy reliable modeling.

    In other words, it isn’t a matter of weak data or bad models: even if those problems are addressed, complexity frustrates prognostication.

    Thats fair. Modeling the atmosphere to produce extremely long term forecasts may not be possible.

    I think its funny, that most people wont put the umbrella in the car on the strength of a 10 day forecast, demand the government spend 10s of Trillions on the strength of a 100 year forecast.

    They would say that’s because if they don’t have an umbrella when it rains, they just get wet.  But if we don’t spend 10s of Trillions, All Life On Earth May End! And We Can’t Take The Chance!

    They would also likely say that even if the global-doom prediction isn’t correct, a world without fossil fuels etc is still better.

    At least, they might say those things if they were as smart as me. :-)

    https://www.youtube.com/watch?v=w2dK4UYAwSI&feature=youtu.be&t=96

    • #43
  14. Henry Racette Member
    Henry Racette
    @HenryRacette

    Interestingly, the drone coordination problem is a much simpler and more manageable one, if the only priority is the safe transit of large numbers of drones. What Rob imagines is optimized routing; all the drone network really needs is collision avoidance, since drones, by their nature, can be expected to be less dependent on optimal transit times.

    Safe drone flight in a crowded environment is a beautiful example of emergent behavior. Bird flocking behavior, the forming and reforming and swooping and apparently coordinated movement of large flocks of birds, can be simulated with uncanny verisimilitude using only two or three rules which every member of the flock applies independently — and with absolutely no central coordination. Similarly, drones can be programmed to travel in predefined drone corridors and to avoid collision with other objects, but otherwise manage their own routing.

    Flying cars? I don’t think that’s going to happen, not in my lifetime. 

     

    • #44
  15. Jon1979 Inactive
    Jon1979
    @Jon1979

    Henry Racette (View Comment):

    Interestingly, the drone coordination problem is a much simpler and more manageable one, if the only priority is the safe transit of large numbers of drones. What Rob imagines is optimized routing; all the drone network really needs is collision avoidance, since drones, by their nature, can be expected to be less dependent on optimal transit times.

    Safe drone flight in a crowded environment is a beautiful example of emergent behavior. Bird flocking behavior, the forming and reforming and swooping and apparently coordinated movement of large flocks of birds, can be simulated with uncanny verisimilitude using only two or three rules which every member of the flock applies independently — and with absolutely no central coordination. Similarly, drones can be programmed to travel in predefined drone corridors and to avoid collision with other objects, but otherwise manage their own routing.

    Flying cars? I don’t think that’s going to happen, not in my lifetime.

     

    Shot this four blocks from my house about a month ago. I think the problem with the flying cars is the landing….

    • #45
  16. Phil Turmel Inactive
    Phil Turmel
    @PhilTurmel

    kedavis (View Comment):

    Phil Turmel (View Comment):

    Paul Schinder (View Comment):

    Henry Racette (View Comment):

    Yes. The Foundation series, which I loved when I was a kid (Asimov was my favorite author when I was growing up), is unreadable dreck now, both in terms of literary quality (okay, that’s just Asimov) and philosophically.

    But then, Asimov was a self-declared socialist — a lot like Bernie, really. So, surprise.

     

    The way I remember it, the point of the original Foundation trilogy was that psychohistory didn’t work. Seldon set up the Foundations, and then the Mule came along…

     

    I’m re-reading the trilogy right now (almost done), thanks to a prod here on Ricochet recently. Your memory is faulty. The Mule was dealt with. (No further spoilers.)

    But Henry’s assessment of the trilogy largely matches mine. I don’t think it’s unreadable, though, just juvenile.

    The even larger problem, really, was that the “psychohistory” predictions snapped right back to being correct again. Ridiculous. But if Trump IS The Mule, then for sure Democrats are hoping that Asimov was right!

    No, the predictions did not snap right back.  Re-establishing the plan post-Mule (with complications) is the plot of the third book of the trilogy.  Asimov’s elitism and statist ideas are worthy of much criticism, but at least do so without misstating the plot of his books.

    • #46
  17. The Reticulator Member
    The Reticulator
    @TheReticulator

    OccupantCDN (View Comment):

    Thats fair. Modeling the atmosphere to produce extremely long term forecasts may not be possible.

    I think its funny, that most people wont put the umbrella in the car on the strength of a 10 day forecast, demand the government spend 10s of Trillions on the strength of a 100 year forecast.

    Predicting averages is a lot easier than predicting the sequences of variations about the mean. In other words, predicting long-term climate is a lot easier than predicting long-term weather.  It’s still pretty difficult, though, and I wouldn’t place a lot of credence in the predictions even if they weren’t politicized.

    Short-term predictions are another matter. I do a lot of planning for my bicycle rides or gardening chores at home based on 10-day weather forecasts, which are accurate enough to be very useful.  It’s a lot different than when I was a kid. 

     

    • #47
  18. Jerry Giordano (Arizona Patrio… Member
    Jerry Giordano (Arizona Patrio…
    @ArizonaPatriot

    Phil Turmel (View Comment):

    kedavis (View Comment):

    Phil Turmel (View Comment):

    Paul Schinder (View Comment):

    Henry Racette (View Comment):

    Yes. The Foundation series, which I loved when I was a kid (Asimov was my favorite author when I was growing up), is unreadable dreck now, both in terms of literary quality (okay, that’s just Asimov) and philosophically.

    But then, Asimov was a self-declared socialist — a lot like Bernie, really. So, surprise.

     

    The way I remember it, the point of the original Foundation trilogy was that psychohistory didn’t work. Seldon set up the Foundations, and then the Mule came along…

     

    I’m re-reading the trilogy right now (almost done), thanks to a prod here on Ricochet recently. Your memory is faulty. The Mule was dealt with. (No further spoilers.)

    But Henry’s assessment of the trilogy largely matches mine. I don’t think it’s unreadable, though, just juvenile.

    The even larger problem, really, was that the “psychohistory” predictions snapped right back to being correct again. Ridiculous. But if Trump IS The Mule, then for sure Democrats are hoping that Asimov was right!

    No, the predictions did not snap right back. Re-establishing the plan post-Mule (with complications) is the plot of the third book of the trilogy. Asimov’s elitism and statist ideas are worthy of much criticism, but at least do so without misstating the plot of his books.

    What a disturbing idea.  I’d prefer to think of Obama as the Mule, in which case Trump is the beginning of the Second Foundation’s frenetic efforts to put things back on track.

    • #48
  19. kedavis Coolidge
    kedavis
    @kedavis

    Jerry Giordano (Arizona Patrio… (View Comment):

    Phil Turmel (View Comment):

    kedavis (View Comment):

    Phil Turmel (View Comment):

    Paul Schinder (View Comment):

    Henry Racette (View Comment):

    Yes. The Foundation series, which I loved when I was a kid (Asimov was my favorite author when I was growing up), is unreadable dreck now, both in terms of literary quality (okay, that’s just Asimov) and philosophically.

    But then, Asimov was a self-declared socialist — a lot like Bernie, really. So, surprise.

     

    The way I remember it, the point of the original Foundation trilogy was that psychohistory didn’t work. Seldon set up the Foundations, and then the Mule came along…

     

    I’m re-reading the trilogy right now (almost done), thanks to a prod here on Ricochet recently. Your memory is faulty. The Mule was dealt with. (No further spoilers.)

    But Henry’s assessment of the trilogy largely matches mine. I don’t think it’s unreadable, though, just juvenile.

    The even larger problem, really, was that the “psychohistory” predictions snapped right back to being correct again. Ridiculous. But if Trump IS The Mule, then for sure Democrats are hoping that Asimov was right!

    No, the predictions did not snap right back. Re-establishing the plan post-Mule (with complications) is the plot of the third book of the trilogy. Asimov’s elitism and statist ideas are worthy of much criticism, but at least do so without misstating the plot of his books.

    What a disturbing idea. I’d prefer to think of Obama as the Mule, in which case Trump is the beginning of the Second Foundation’s frenetic efforts to put things back on track.

    Nice!  I like that much better too.

    • #49
Become a member to join the conversation. Or sign in if you're already a member.