Model Mania
Model building is now the preferred method of “doing science” in fields like economics, population control, and climatology. Very large and complex models are being produced purporting to predict the behavior of large, complex, adaptive systems. Those of us who study the history and application of complex models know that they don’t work as reliable predictors of events. But model builders are oblivious; "all we need," they say, "is to make better models."But they are missing a fundamental flaw in all that they do: they do not realize that complex models cannot work, for some reasons that I would like to discuss.
Small scale models, like the equations for supply and demand, have enormous explanatory power at the micro level. Simple models are fairly good at predicting the behavior of simple mechanisms like clocks; they are not good at predicting the behavior of clouds.
In contrast, complex large scale macro models have proved to have very little descriptive or predictive power. These macro models rely on equations which are themselves time and place dependent. The logic of the underlying models and their assumptions are buried in a sea of equations, calculations, and estimates or probabilities, all of which are seemingly theoretically and empirically sound although they are often based on relatively little theory and only a few carefully chosen empirical observations. According to Freeman Dyson, almost all funding in global warming research is now being devoted to model building and very little funding has been devoted to gathering actual data.
The fundamental logic of model building has not changed since the large scale econometric models of the 1930s. Behind the computer simulations, a model consists of a series of equations, a set of variables, and a list of relations between those variables expressed by the coefficients of the model. And, although the original in-put out-put equations for a macro-economic model may have been created by means of observations of a functioning market-based economy, once fixed into the model, the empirical and theoretical inputs became conceptually redundant. For the most part the predictions of the model are in fact merely restatements of the assumptions of the model and the critically important coefficients are derived by means of empirical generalizations more often than not from outdated and possibly irrelevant data. To take a recent example: the entire stimulus package was justified by the assumption of a coefficient (multiplier) of 1.5. Larry Summers assured the nation that for every new government job, at least 1.5 other jobs would be created in the economy. Perhaps that may have been the case at some other place and some other time, but it was not the case in America in 2009.
Let’s look at other examples. Population growth models based on data from 1950 -1970 are inapplicable to population growth patterns in 2009. Investment banking models based on mortgages relied on data from 1945 - 2005 a period during which time single family housing prices in aggregate increased. The models based on these data predicted that single home mortgages in aggregate had less than a 1% chance risk of default. As a result, neither the government nor the investment houses required margins for trading in these aggregated mortgages and their derivatives and the derivatives of their derivatives. Financial collapse was not predictable from within these particular models, although any economist with a scrap of paper and a pencil should have been able to predict the likelihood of that collapse.
And as some Ricohetiers have pointed out, in the hands of the policy makers, the model itself becomes a black-box mechanism for implementing and formulating policy. And so, for example, from 1930s to 1970s macro-economists built enormously complex models of economic systems designed to serve as basis for long term economic planning. Psychologically, the models were impressive in that they gave people the illusion of knowledge and control. Unfortunately, those models didn’t work except perhaps to provide employment and professional advancement for cadres of economists and to motivate whole populations to accept poor economic policies.
Motivating the public has always been essential to the planners. According to developmental economist Michael Todaro such plans provided important psychological benefits in "mobilizing popular sentiment and cutting across tribal factions with the plea to all citizens to 'work together,'” so that an “enlightened central government, through its economic plan, [could] provide the needed incentive to overcome the inhibiting forces of traditionalism in the quest for widespread material progress."
Complex phenomena like climate change cannot be predicted by a single model no matter how complex that model. Multiple models cannot simply be aggregated into a macro models. Unfortunately, the more manpower and resources are devoted to the model-building the more fiercely it is defended. No countervailing opinions even from within particular disciplines are allowed to undermine the faith in the model itself. Critics are castigated as heretics. Unlike genuinely scientific theories, these models cannot be tested. The models create the appearance of precision by the magic of long division and computer simulations, but there is little real precision in their predictive ability. When the model-builders encounter facts which seem to contradict their predictions, they ignore the facts or they tinker with the model. The proponents of these models will not admit even the possibility of being fundamentally in error. Tautological arguments are very convincing.
A model is not a theory although it will contain theoretical elements. And even a genuine scientific theory is a net which addresses only certain aspects of reality. In its essence a good scientific theory is and must be a simplification, and so while its powers to explain are high, its ability to predict is often limited except in very controlled conditions. What the theory cannot interpret in its terms, it must ignore. A sophisticated theorist will recognize the limitations of any particular theory and a responsible model builder will recognize the limitations of any particular model. Unfortunately, today’s model builders believe that they have somehow reproduced reality and in their zeal they are often able to use “enlightened central governments” “to mobilize public sentiment” to their various causes
- Comment (40)
- · Quote
- · UnfollowFollow (6)
- Pages:
- 1
- 2
- Pages:
- 1
- 2
Comments :
Mar '11
Re: Model Mania
Interesting. I'd like to understand the line between a simple model (that works) and a complex model (that does not). Is it because a complex model tries to model reality, while a simple model can define the parametersin a made-up world (like the classic "frictionless cows in space" vector physics)?
If so, does this mean that *all* real-world models about the unknown future are doomed? That seems to be untrue - we know that predictive models about things like motors and bridges work. I think if we can find the dividing line, then we can better understand the weaknesses of complex models.
Edited on Sep 1, 2011 at 6:38pmDec '10
Re: Model Mania
This is covered by Chaos Theory. In a nutshell Chaos Theory says that any model of a complex system will be wrong in the long term because small errors expand geometricly.
Edited on Sep 1, 2011 at 7:23pmAug '11
Re: Model Mania
Foxman: This is covered by Chaos Theory. In a nutshell Chaos Theory says that any model of a complex system will be wrong in the long term because small errors expand geometricly. · Sep 1 at 7:10pm
Edited on Sep 01 at 07:23 pm
This is correct. Chaos is a powerful reason for the failure of these models -- but it is not the only powerful reason.
Jun '10
Re: Model Mania
After working at my friend's small business the last few weeks, it is completely obvious that models for predicting the future will NEVER work, let alone a model that attempts to make sense of the American economy or climate. I've heard so many people say, "But computers are so much more powerful now!" Sorry, the world is a lot more complex now, too, so the bar just keeps getting set higher and higher. Large economies and climate act too much in the environment of randomness which no statistics or algorithms can chase effectively. Microeconomics, yes, to a reasonable degree. Macroeconomics and climate... forget it. We can't even agree with whether lower taxes help the economy or hurt it or whether FDR's policies helped save us from the Depression or whether they made things worse. I think I know, but that doesn't mean half the world will come around to my point of view. Only the market knows and she doesn't know until it happens (successful investors excepted, I guess). Great post. We'll never get the assumptions right, so the rest is just SWAG.
Re: Model Mania
Although I think many of these points are quite correct, the argument can be taken to the point of useless radical skepticism. Real policy decisions do have to be made, not making them is a decision, and we've got to use the best models we have.
Jun '10
Re: Model Mania
It seems that a poorly-designed computer model is worse than having no computer model. "My wife loves gardening, so my gift-buying computer model tells me that a great silver anniversary gift would be a silver-plated garden shovel." Has to be right. It's science.
Edited on Sep 2, 2011 at 6:00amDec '10
Re: Model Mania
But we should not make major policy decisions based on models that we low confidence in, i.e. climate change.
Aug '10
Re: Model Mania
Perhaps. But how does one address the inherent complication that, as often as not, the government IS the problem? Frankly, much of the time, what we need is not a new policy, or a different policy - but less policy.
Edited on Sep 2, 2011 at 6:06amFeb '11
Re: Model Mania
One of the things wise businesses have learned is that **agility trumps prediction**. Say you are a manufacturer and it takes a long time to change the tooling from one part to another. You could develop complex statistical forecasting programs to gauge the future demand for final products, acquire MRP/ERP software to explode the demand down through the product structure and estimate the future needs for each part, and hire PhDs to develop optimization models for the best length of run for each part.
Or you could focus on figuring out how to change the tooling from one part type to another more quickly, which is what Toyota did, particularly with their quick die change program.
The same point is valid in many situations, not just manufacturing.
Jan '11
Re: Model Mania
Thus my love of studying game theory, and strategy.
The interplay of past performance and current assumptions is shaky enough, but when you introduce human beings into the equation, you change the game. Human beings are strategic, meaning that there are situations where you do better by doing what you "shouldn't" do. No matter what the model decides is the best course, that decision alone prompts a strategic player to act differently.
Dec '10
Re: Model Mania
I don't think the issue is whether models are useful or not. It's which models are useful or not. More importantly, when analyzing the efficacy of a model the questions to ask are whether a model is appropriate and if there is an agenda. Clearly, Climagate showed that an agenda of "proving" AGW influenced the model makers to the point of falsifying data. There is a hubris to these model makers of which Kenysian economic modeling is the most obvious example. In the early 1970s I was in an economics class for an MBA. The earnest professor put up the economic graphs of Keynesian supply and demand vs inflation and unemployment explaining that high inflation is accompanied by high employment. When I asked him how the current situation (at the time stagflation) where there was high inflation and high unemployment fit on his model, he had no answer. Yet, he maintained the model was correct. Complex model makers want reality to conform to their theories. A humbler scientist knows the validity of his model is just the opposite.
Mar '11
Re: Model Mania
No, we don't. Models give us the illusion of being accurate and true. In fact, they are usually (based on climate predictions) worse than using dart-throwing monkeys.
So make policy decisions for philosophical reasons: lower taxes, increase freedom, allow individuals to pursue their own goals, and the like. If we don't believe in the invisible hand, we are doomed anyway.
But don't rely on complex models. If they get it right, it is just luck.
Sep '10
Re: Model Mania
Fair point, but I am reminded of SecDef Macnamara and his practice of scoring bombing targets scientifically by a number of variables in order to pick the best ones (i.e. using the best models we have). I think the result was Viet Nam 1 US nil.
It is a difficult question on how you prevent the slide into "know nothing" thinking, but we must all be aware when we are faced with these complex model results which direct us to take this action, that the models in almost all cases just reinforce confirmation bias. That is, when the results do not cofirm conventional wisdom, the model is tweaked until they do.
Mar '11
Re: Model Mania
Sorry. Every model is built with an agenda (after all, any experiment starts with a theory). And the model builders have their assumptions baked into their model - where I would see a vibrant economy, for example, a marxist would sees a class struggle. We would build models of what we see, and the results would be as different as chalk and cheese.
Dec '10
Re: Model Mania
Denial is not just a river in Egypt. Professors have adopted an interest in conforming to a teacher-parrot: No ruffled feathers if he repeats what he had been told by other teachers. Teacher-parrots live in the comfort of repeating what they were taught without having to give it any thought.
Teacher-priests hold the staff of model making like a monarch holds his scepter. The illusion of power reinforces the perception of power. These model makers are cynical about their perpetuating the illusory heuristic value of a bundle of worthless equations because they did make one useful discovery: To be credible, the price of the fraud has to be exorbitant and the appearance of its cost to the public in its absence catastrophic.
There are also the self-anointed pseudopriests who vicariously associate with the priesthood thinking they know what it means to be smart or intelligent without ever having to wonder what it is like to be wise.
No surprise then, why the principals surrounding our historic first Islamic apostate president who are in charge of his administration have experience only in academia or government, and why all of their social and economic models fail.
Feb '11
Re: Model Mania
Ross Conatser..."SecDef Macnamara and his practice of scoring bombing targets scientifically by a number of variables in order to pick the best ones"
OTOH, I understand that during WWII a network model of the German transportation system was developed in order to identify the most effective points to attack, apparently with useful results. And, of course, the aiming of artillery has long been accomplished via mathematical models of projectile flight, with calculations at first done by rooms full of human "computers" and later by analog and digital computing machinery.
The trick is to distinguish the useful models from the useless ones.
Jun '11
Re: Model Mania
Excellent post. Very interesting.
In our day to day business operations, we spend as much time planning as we can. Planning and predicting are what gets an organization to a certain level of functionality.
The act of planning, however, must always be assumed to be a somewhat blind process because you cannot plan for what you do not expect and in life or business, what you do not expect is what will rip your plan apart.
The probable is easy. The improbable is not.
Unless you can establish a computer model that can accurately calculate probabilities, you're making assumptions similar to those made by the designers of the Titanic.
Re: Model Mania
Kenneth Arrow, one of the great economists of the 20th century, spent WWII in a weather forecasting unit. Asked to do long-range forecasts and with data that indicated this was a hopeless task, he and his associates asked to have their unit disbanded and its manpower reassigned to something of some value. The response received was, "The general is very well aware that your division's forecasts are useless. But they are required for planning purposes."
Aug '11
Re: Model Mania
Sorry for the delay. I am on Pacific Time and just learning this system.
Real decisions must be made and they must almost always be made without perfect information. We do use models and we must use models. But we must use more than one model in making every decision and we must be always aware of the limitations of certain kinds of models. This is not radical skepticism, but a plea for caution and moderation.
Mar '11
Re: Model Mania
michael kelley: Excellent post. Very interesting.
The act of planning, however, must always be assumed to be a somewhat blind process because you cannot plan for what you do not expect and in life or business, what you do not expect is what will rip your plan apart.
As as Donald Rumsfeld put it:
There are known knowns.
There are things we know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don't know
We don't know.