Saturday Night Science: Plastic Fantastic

Plastic Fantastic by Eugenie Samuel ReichBoosters of Big Science, and the politicians who rely upon its pronouncements to justify their policy prescriptions often cite the self-correcting nature of the scientific process: peer review subjects the work of researchers to independent and dispassionate scrutiny before results are published, and should an incorrect result make it into print, the failure of independent researchers to replicate it will inevitably call it into question and eventually cause it to be refuted.

Well, that’s how it works in theory. Theory is very big in contemporary Big Science. This book is about how things work in fact, in the real world, and it’s quite a bit different. At the turn of the century, there was no hotter property in condensed matter physics than Hendrik Schön, a junior researcher at Bell Labs who, in rapid succession reported breakthroughs in electronic devices fabricated from organic molecules including:

In the year 2001, Schön published a paper in a peer reviewed journal at a rate of one every eight days, with many reaching the empyrean heights of Nature, Science, and Physical Review. Other labs were in awe of his results, and puzzled because every attempt they made to replicate his experiments failed, often in ways which seemed to indicate the descriptions of experiments he published were insufficient for others to replicate them. Theorists also raised their eyebrows at Schön’s results, because he claimed breakdown properties of sputtered aluminium oxide insulating layers far beyond measured experimental results, and behaviour of charge transport in his organic substrates which didn’t make any sense according to the known properties of such materials.

The experimenters were in a tizzy, trying to figure out why they couldn’t replicate Schön’s results, while the theorists were filling blackboards trying to understand how his incongruous results could possibly make sense. His superiors were basking in the reflected glory of his ascendence into the élite of experimental physicists and the reflection of his glory upon their laboratory.

In April 2002, while waiting in the patent attorney’s office at Bell Labs, researchers Julia Hsu and Lynn Loo were thumbing through copies of Schön’s papers they’d printed out as background documentation for the patent application they were preparing, when Loo noticed that two graphs of inverter outputs, one in a Nature paper describing a device made of a layer of thousands of organic molecules, and another in a Science paper describing an inverter made of just one or two active molecules were identical, right down to the instrumental noise. When this was brought to the attention of Schön’s manager and word of possible irregularities in Schön’s publications began to make its way through the condensed matter physics grapevine, his work was subjected to intense scrutiny both within Bell Labs and by outside researchers, and additional instances of identical graphs re-labelled for entirely different experiments came to hand. Bell Labs launched a formal investigation in May 2002, which concluded, in a report issued the following September, that Schön had committed at least 16 instances of scientific misconduct, fabricating the experimental data he reported from mathematical functions, with no evidence whatsoever that he had ever built the devices he claimed to have, or performed the experiments described in his papers. A total of twenty-one papers authored by Schön in Science, Nature, and Physical Review were withdrawn, as well as a number in less prestigious venues.

What is fascinating in this saga of flat-out fraud and ultimate exposure and disgrace is how completely the much-vaunted system of checks and balances of industrial scale Big Science and peer review in the most prestigious journals completely fell on its face at the hands of a fraudster in a junior position with little or no scientific track record who was willing to make up data to confirm the published expectations of the theorists, and figured out how to game the peer review system, using criticisms of his papers as a guide to make up additional data to satisfy the objections of the referees. As a former manager of a group of ambitious and rambunctious technologists, what strikes me is how utterly Schön’s colleagues and managers at Bell Labs failed in overseeing his work and vetting his results. “Extraordinary claims require extraordinary evidence“, and Schön was making and publishing extraordinary claims at the rate of almost one a week in 2001, and yet not once did anybody at Bell Labs insist on observing him perform one of the experiments he claimed to be performing, even after other meticulous experimenters in laboratories around the world reported that they were unable to replicate his results. Think about it—if a junior software developer in your company claimed to have developed a miraculous application, wouldn’t you want to see a demo before issuing a press release about it and filing a patent application? And yet nobody at Bell Labs thought to do so with Schön’s work.

The lessons from this episode are profound, and I see little evidence that they have been internalised by the science establishment. A great deal of experimental science is now guided by the expectations of theorists; it is difficult to obtain funding for an experimental program which looks for effects not anticipated by theory. In such an environment, an unscrupulous scientist willing to make up data that conforms to the prejudices of the theorists may be able to publish in prestigious journals and be considered a rising star of science based on an entirely fraudulent corpus of work. Because scientists, especially in the Anglo-Saxon culture, are loath to make accusations of fraud (as the author notes, in the golden age of British science such an allegation might well result in a duel being fought), failure to replicate experimental results is often assumed to be a failure by the replicator to precisely reproduce the circumstances of the original investigator, not to call into question the veracity of the reported work. Schön’s work consisted of desktop experiments involving straightforward measurements of electrical properties of materials, which were about as simple as anything in contemporary science to evaluate and independently replicate. Now think of how vulnerable research on far less clear cut topics such as global climate, effects of diet on public health, and other topics would be to fraudulent, agenda-driven “research”. Also, Schön got caught only because he became sloppy in his frenzy of publication, duplicating graphs and data sets from one paper to another. How long could a more careful charlatan get away with it?

Quite aside from the fascinating story and its implications for the integrity of the contemporary scientific enterprise, this is a superbly written narrative which reads more like a thriller than an account of a regrettable episode in science. But it is entirely factual, and documented with extensive end notes citing original sources.

Reich, Eugenie Samuel. Plastic Fantastic. New York: St. Martin’s Press, 2009. ISBN 978-0-230-62384-2.

 

Here is a BBC documentary about l’affaire Schön.  It is sensationalistic, repetitive, and poorly transferred to video, but the latter part does a reasonably good job of describing how Schön’s fraud was uncovered.

  1. raycon and lindacon

    To make the matter worse, a notably ambitious and somewhat scientific illiterate class of people, known by the term “politicians” have exploited the above failure of the scientific and business establishments to accrue power unto themselves. 

    This has led to the virtual death of the American experiment.

  2. Sisyphus

    Homo grantseekerus is the ugliest sort of malfeasant.

  3. Mike Rapkoch

    I’d like to say I’m surprised, but I’m not. I’ve seen many a successful man cook the books in business, medicine, law–you name it. That, I believe, is in man’s nature, especially when someone is driven to success with such power that he starts to delude himself that he is doing good by doing bad. I’m also not surprised by the complicity–unconscious but still regrettable–of the scientific community. When a stock starts to rise quickly everyone wants in on the action. I would guess that many of the scientists who sought to replicate Schon’s findings forgot their objectivity in hopes they would go him one further and thus ride the coat tails so to speak, What bothers me is that such scientists would not want to disprove his theories even for no other reason than the principle of falsification.

    But, then again, we are an inscrutable lot.

  4. civil westman

    This is not surprising, given that we live in a time where everyone is entitled not only to his own opinion, but to his own “facts.” If the latter do not exist, simply make them up and add “science” to the verbiage. Q.E.D.

  5. Mendel

    How does this story demonstrate a breakdown of the system? I would say it’s an example of how the system worked.

    John Walker:

    …peer review subjects the work of researchers to independent and dispassionate scrutiny before results are published, and should an incorrect result make it into print, the failure of independent researchers to replicate it will inevitably call it into question and eventually cause it to be refuted.

    Which is exactly what happened in this case, no? Within a year of the major publications, flags had been thrown and the researcher was under review. That’s a lot faster than fraud in many other fields is exposed.

    This post seems to misunderstand the peer review process. The pre-publication process is not designed to sniff out fraud – there is an explicit trust that manuscript authors are telling the truth, and no independent attempts are made to verify the veracity of results before publication.

  6. Mendel

    It is thus hard to blame the journals who published Schön’s work when detecting fraudulent research is not part of their job description. That does leave a great big door open to potential fraudsters, but there is really no way to completely eliminate fraud in a system which produces products with no basic need.

    On the other hand, I do think this is an indictment of Bell Labs. But Bell Labs isn’t part of the government-funded Big Science complex everyone is railing against here – it’s a private institution with a history of some of the greatest innovations of the last century.

  7. Randy Webster

    That bastard!  Lying about sputtered aluminum oxide.

  8. John Walker

    Mendel: This post seems to misunderstand the peer review process. The pre-publication process is not designed to sniff out fraud – there is an explicit trust that manuscript authors are telling the truth, and no independent attempts are made to verify the veracity of results before publication.

    But when you have an investigator who is publishing results every eight days which nobody else is seeing, shouldn’t that raise some eyebrows among the reviewers and his management?  In the book, there are documented extensive communications by others who attempted to replicate Schön’s work which were met with responses which, to anybody with experience of financial fraud, would set off all of the warning bells.  His management were aware of these requests.

    It was only after obviously bogus charts in his publications were found that the investigation began.  Why did not any of the peer reviewers ask to see his raw data (which, after the investigation began, he failed to produce: “dog ate my homework”)?

    Is the “explicit trust that manuscript authors are telling the truth” justified?  When you have such a stunning example of how the system can be gamed (and it is not alone), shouldn’t reviewers apply a greater level of scrutiny to extraordinary claims?

  9. Mendel

    John Walker:

    Mendel

    But when you have an investigator who is publishing results every eight days which nobody else is seeing, shouldn’t that raise some eyebrows among the reviewers and his management? 

    Management, yes – but that problem has nothing to do with the peer-review process (which only involves people outside of the researcher’s institution).

    The problem for the reviewers/editor is that they don’t know what other manuscripts the researcher currently has in the pipeline at other journals (after all, those journals are competitors and don’t talk about their submissions until they are published). Theoretically, a researcher can submit the exact same manuscript to 10 publications simultaneously and they could all get published – the system simply isn’t designed to catch it.

    And considering that it often takes one year or longer between initial submission of a manuscript and its final publication, it would make sense that Schön could have a fairly long streak of impossibly rapid publication before the journals started hitting the brakes.

  10. Mendel

    John Walker:

    Mendel

    Is the “explicit trust that manuscript authors are telling the truth” justified? When you have such a stunning example of how the system can be gamed (and it is not alone), shouldn’t reviewers apply a greater level of scrutiny to extraordinary claims?

     It’s less of a question of whether it’s justified than whether there’s a workable alternative. Poring through the underlying data of even 10% of the submissions to Science or Nature is a mind-numbing and time-consuming process (especially with advent of technologies that produces reams of data in seconds). Like with many manufacturing processes, it’s easier to perform QC after manufacturing and then do a retrospective analysis when a fault is found, then to do a full in-line QC on every item.

  11. Mendel

    I don’t disagree that the peer review process is much more imperfect than its supporters claim it to be. But that has little to do either with the peer review process itself, nor with the growth in academic scientific funding. Rather, it is an inevitable consequence of doing research which has no direct market demand.

    The best way to find out if research was performed properly is to turn it into a product and bring that product to market. If the research was fraudulent, the product won’t work, and customer revolt will punish the fraudster directly or (more likely) indirectly.

    But without a paying customer for most academic research, the most powerful tool to keep researchers honest disappears, and any replacement will have this Achilles’ heel. So, yes, peer review is a very inherently flawed process, but it’s likely the best solution to a hopelessly flawed solution.

  12. Paul Erickson

    Is he now a respected climate scientist?

  13. John Walker

    There is also the phenomenon of “science by press release”.  In the current Nature Paul Steinhardt argues that the recent much-vaunted BICEP2 measurement indicating the presence of primordial gravitation radiation was flawed, not taking into account foreground effects which could mimic the small signal they were trying to measure.

    There is no assertion of fraud in this case: the BICEP2 scientists were diligent in analysing their data, but they chose to make them public and draw conclusions before other data were available, in particular from the Planck spacecraft, which could confirm or falsify their results.

    This just means that when reading accounts in the press of scientific discoveries, if you don’t wish to look under the hood to examine the evidence, it’s worth letting them age for a while to see if they stand up under scrutiny.

  14. John Walker

    Paul Erickson:

    Is he now a respected climate scientist?

    His Ph.D degree was revoked in 2004 by the University of Konstanz which issued it.  He appealed the ruling, but it was affirmed in 2009.  He then sued the university, and in 2010 the court held in his favour, restoring his doctorate.  The university then appealed the ruling, and in 2011 the state court found that the university was correct, and that his doctorate should be revoked.  A further appeal to the Federal Administrative Court affirmed the state court’s ruling than his degree should be revoked.

    That’s what happens when you fudge data in a hard science where others can replicate your experiments.  When you’re building computer models of coupled nonlinear systems with feedback and time delays where the initial conditions are poorly known, not so much.

    One might say that (former) Dr Schön chose poorly.

  15. John Walker

    Mendel:  On the other hand, I do think this is an indictment of Bell Labs. But Bell Labs isn’t part of the government-funded Big Science complex everyone is railing against here – it’s a private institution with a history of some of the greatest innovations of the last century.

    I can’t speak for everybody in the comments, but I’ve re-read my original article and I don’t think I once attributed the scientific malpractice to government funding.  (Which is not to say that I haven’t done so elsewhere, just not here.)  But Bell Labs is, I would argue, regardless of its funding, in the mainstream of Big Science, where prestige is evaluated by publications in high-impact journals and citations.  In that environment, the temptation at all levels to game the system exists.  It is inevitable that some people who master that arcane craft will, at least temporarily, rise to the top.

  16. Mendel

    I can’t speak for everybody in the comments, but I’ve re-read my original article and I don’t think I once attributed the scientific malpractice to government funding.

     Fair enough. I had just read the other thread ranting about science and probably conflated your criticisms with so many others I have read over the years on Ricochet.

    But Bell Labs is, I would argue, regardless of its funding, in the mainstream of Big Science, 

     While this is true, Bell Labs is also exceptional – for the most part, research at profit-seeking companies is not published in peer-reviewed journals.

    All of this was to comment on your point that a researcher’s management should be keeping track of him: for most peer-reviewed publications, there is no “management” – they are authored by tenured professors who are their own bosses. The only guard against publishing fraudulent research are the journals themselves, and they have deliberately decided to take a “trust, but verify if other complain after publication” approach.

  17. MJBubba

    “A great deal of experimental science is now guided by the expectations of theorists; it is difficult to obtain funding for an experimental program which looks for effects not anticipated by theory.”

    That is a great line.   Thanks.   I will quote it someday soon in a discussion of Evolution or the age of the earth.

  18. Nick Stuart

    Kinda long & technical. Apologies for not reading very closely and not really understanding.

    But it seems like the gist is that the science was settled, and the consensus was in; until it wasn’t.

  19. Israel P.

    How long could a more careful charlatan get away with it?

    Wrong question.

    Right question – How many more-careful charlatans haven’t been caught yet?

  20. Freesmith

    An interesting story. Too bad journalists like Janet Cooke, Stephen Glass and Jason Blair didn’t report it at the time.

    I guess they were too busy giving their bosses what the bosses wanted to see.

Want to comment on stories like these? Become a member today!

You'll have access to:

  • All Ricochet articles, posts and podcasts.
  • The conversation amongst our members.
  • The opportunity share your Ricochet experiences.

Join Today!

Already a Member? Sign In