The U.S. intelligence community has a very fine reputation. On screen it is portrayed as practically omniscient and omnipotent. When it is criticized, it is usually more for evil genius than for bumbling. But this so-called Arab spring, and the instabilities and dangers it has brought, seems to have caught U.S. intelligence asleep on the job. What went wrong?
I recommend this whole New York Times "Room for Debate." I particularly like the main points in Reuel Marc Gerecht's reply. He blames group-think and scientism:
The intelligence community comprises many bureaucracies, which even when small and bright -- the State Department’s intelligence and research bureau, for example — usually reflect the established views of the larger institutions that house them. The dreaded disease — “group think” — is an unavoidable, and usually undetectable, affliction for intelligence analysts who want to prosper professionally...
The power of conformity is even greater elsewhere in the intelligence community, where the intellectual claustrophobia that comes with working in isolated, highly classified environments is more acute. (State is a realm of audacious libertines compared to the Central Intelligence Agency’s “campus” in suburban Virginia.) Intelligence bureaucracies ruthlessly extirpate intuition, the key ingredient that makes first-rate analysis. For wholly understandable reasons, bureaucracies can’t handle intuition: you can’t quantify it, you can’t really teach it and those who don’t have it — the vast majority of analysts — will rise in indignation against its use.
I have no experiential knowledge of the intelligence community, but these seem like plausible explanations. A typical analyst may have a bright idea about what's going on in Egypt. But if that idea waivers too much from the consensus of her respected, higher-pay-grade superiors she will (1) either convince herself her idea is ridiculous or (2) think it's not worth the risk of alienating herself from higher-ups, thus risking her job, for a prediction that is unlikely and won't gain her very much if it is correct anyways. Her incentive is clearly to stay within the consensus. Institutions have substantial intellectual inertia, which may be increased by hierarchies. We should perhaps consider restructuring analysts' incentives (giving very high rewards for unlikely but proven-correct predictions, perhaps?) to overcome that inertia.
The second point -- that intelligence only considers those factors that can be quantified -- is part of the more general scientistic conviction that a thing only counts as knowledge if it can be measured. Rendering everything quantitatively is also a way of deflecting responsibility ("That's not my claim! It's just what the numbers say!") and so may also may be a problem of the incentives of the intelligence community.