Fail Fast, Fail Often

 

shutterstock_24567307There is a paradigm in software (and other engineering disciplines, I suspect) called “fail fast.” On first encounter, it sounds odd. Why fail fast? Don’t we want to delay failures? Of course, the context drives the design; and far too often, in software, one deals with a increasingly complex system that comprises of many moving pieces. So, sometimes, if something fails, it’s much better to know about it sooner than later, so it can be addressed. Jim Shore wrote the best article on the advantages of failing fast to my knowledge, here. He says:

Some people recommend making your software robust by working around problems automatically. This results in the software “failing slowly.” The program continues working right after an error but fails in strange ways later on. A system that fails fast does exactly the opposite: when a problem occurs, it fails immediately and visibly. Failing fast is a nonintuitive technique: “failing immediately and visibly” sounds like it would make your software more fragile, but it actually makes it more robust. Bugs are easier to find and fix, so fewer go into production.

I find parallels between my software experiences and my life. I often notice what works in one paradigm has uses in others and — in many cases — I want my experiences to be “fail fast.” In other words, I want to know sooner rather than later if a new product will be successful, if a new business relationship will work, if a new idea will manifest, etc.

If you want to succeed, double your failure rate. – Thomas J. Watson

In business, it is better to know of upcoming failures early in order to avoid the loss of putting time and money into ideas that are likely to fail. In dating it’s better to dump a guy up front if he doesn’t want kids and you do. I’d rather know this sooner than later. How does this translate to actions? To me, it translates to more encounters, more interactions, more ventures, more things to “start.” Not all ventures will succeed, but in order to increase successes, there must be a higher rate of corresponding failures. And if we have to have failures, why not get them over with quickly?

In order to do so, however, we need to rethink failure as a feedback mechanism. We often make failure out to be a stigma. While failure is not optimal in all situations, sometimes failures are nothing but lessons in disguise. They can be feedback mechanism for actions; they tell us how to improve on things, how not to do something. How else can you discover a new passion without exposing yourself to failure? How many times have we started a project, failed, but discovered a new way to do things? We have all heard stories of Edison failing to invent the light bulb 1,000 times.

So, my friends, fail fast, fail often.

There are 43 comments.

  1. 1
  2. 2
  1. Carey J. Inactive

    Barkha Herman: Fail Fast, Fail Often

    I think that’s the Obama administration’s motto.

    • #1
    • October 8, 2015, at 8:25 AM PDT
    • Like
  2. I Walton Member

    Taleb on anti fragility, and Megan McCardel on The success of Failure. We learn only from failure, we get strong by constant challenge and stress. A lot of political and policy lessons from these, with one giant exception, governments don’t learn from failure, because interests grow up around them and freeze them in place. So their failures are always slow and catastrophic.

    • #2
    • October 8, 2015, at 8:25 AM PDT
    • Like
  3. Barkha Herman Member
    Barkha Herman Post author

    John Penfold:Taleb on anti fragility, and Megan McCardel on The success of Failure. We learn only from failure, we get strong by constant challenge and stress. A lot of political and policy lessons from these, with one giant exception, governments don’t learn from failure, because interests grow up around them and freeze them in place. So their failures are always slow and catastrophic.

    Governments are monopolies; and monopolies always skew reality.

    Yes, Taleb and McCardel are good readings on this. Are we teaching our kids, though, that failure is not all “bad and wrong”?

    • #3
    • October 8, 2015, at 8:28 AM PDT
    • Like
  4. Carey J. Inactive

    Barkha Herman:

    John Penfold:Taleb on anti fragility, and Megan McCardel on The success of Failure. We learn only from failure, we get strong by constant challenge and stress. A lot of political and policy lessons from these, with one giant exception, governments don’t learn from failure, because interests grow up around them and freeze them in place. So their failures are always slow and catastrophic.

    Governments are monopolies; and monopolies always skew reality.

    Yes, Taleb and McCardel are good readings on this. Are we teaching our kids, though, that failure is not all “bad and wrong”?

    https://www.youtube.com/watch?v=hURajfBiJjQ

    • #4
    • October 8, 2015, at 8:41 AM PDT
    • Like
  5. Aaron Miller Member

    Barkha Herman: Governments are monopolies; and monopolies always skew reality.

    The problem with trying to apply this good advice to politics (which I don’t think you were doing) is that a government becomes equated with the community after a generation or two.

    When the Articles of Confederation didn’t work out, America’s founders quickly pushed forward to the Constitution we now admire. Today, citizens so take this government for granted that they can’t imagine the nation sticking together without it. Indeed, it has grown so wildly beyond its original boundaries that it is entrenched in nearly every aspect of American life. It’s too big to fail.

    Are there possible political applications of the “fast fail” principle to local politics? I don’t want legislators at any level of government rushing through bill after bill. It would be better if they focused on repealing laws at this point.

    But it’s good advice for non-political life. Thanks.

    • #5
    • October 8, 2015, at 8:54 AM PDT
    • Like
  6. Marion Evans Inactive

    Doubling your failure rate is a great idea if, like Tom Watson’s, your effort culminates in success. Otherwise you are just a bum whose wife left him because he failed too many times :)

    • #6
    • October 8, 2015, at 9:04 AM PDT
    • Like
  7. Yeah...ok. Inactive

    most of my comments are short, in the fail fast mode.

    Great post. Please post more.

    • #7
    • October 8, 2015, at 9:14 AM PDT
    • Like
  8. MarciN Member

    I agree with the premise in the original post completely.

    It’s insanity to keep pursuing some course of action that is not working. The sooner you know, the better.

    However, it’s also important to keep trying too sometimes. I used to do everything but stand on my head to get kids to keep trying to learn how to do something that was hard for them at first.

    • #8
    • October 8, 2015, at 9:16 AM PDT
    • Like
  9. MarciN Member

    [continued from comment 8]

    There is a great poem by Rudyard Kipling about keeping going in the face of great odds. “Don’t Quit” makes the point that you will never know how close to success you may have been if you quit. There is so much truth in that–it’s as if the fates tempt you to quit just as you are about to succeed. :)

    A friend of mine who was the principal of a middle school used to print this in the school newsletter at least once a year:

    Don’t Quit, by Rudyard Kipling

    When things go wrong as they sometimes will.

    When the road you’re drudging seems all uphill.

    When the funds are low, and the debts are high,

    and you want to smile but you have to sigh.

    When care is pressing you down a bit.

    Rest if you must but don’t you quit!

    Life is queer with its twists and turns,

    as everyone of us sometimes learns.

    And many a failure turns about.

    When he might have won had he stuck it out.

    Don’t give up though the pace seems slow –

    you may succeed with another blow.

    Success is failure turned inside out –

    the silver tint of the cloud of doubt.

    And you never can tell how close you are.

    It may be near when it seems so far.

    So stick to the fight when you’re hardest hit-

    it’s when things seem worst that you must not quit.

    • #9
    • October 8, 2015, at 9:17 AM PDT
    • Like
  10. Tuck Inactive

    I used this concept in my first big software project, one I’m still hammering away at years later (LOL).

    I had a robust debug routine built into the thing so that when I had a failure I’d know about it quickly and the users knew to leave the message on the screen so I could fix it immediately.

    I hadn’t started this way, but learned pretty quickly. But some of the old code didn’t have the error-handling. I had a default error message at the root of the thing that was never supposed to be hit, but would let me know that the error-handling hadn’t been implemented for the code that had failed. In a whimsical moment, I unfortunately used the phrase “Who the f— wrote this thing?”

    Of course years later, the day came when my phone rang and a user said, “I got a strange error message on my screen…” I’d long forgotten about it. Happily we’d been working together for a while, and he found it as amusing as I’d meant it to be.

    Government, of course, does what we call “eat errors”, and doesn’t react to them. Comparing the processes of legislatures to software developers explains virtually all the problems we’re now experiencing.

    The users of government are always getting that message, and the errors are rarely corrected.

    • #10
    • October 8, 2015, at 9:30 AM PDT
    • Like
  11. Dr. Strangelove Thatcher

    I cannot agree with this mode of software development more.

    While I agree that it better for a system to “fail gracefully” when in production (so as to avoid friction with the customer) it is far better to halt on failure in development so as to ensure that defects cannot be overlooked or ignored.

    • #11
    • October 8, 2015, at 9:31 AM PDT
    • Like
  12. Aaron Miller Member

    If it’s possible to explain to a layman, how do programmers arrange that? How do you ensure errors result in instant, epic failure rather than just a memory leak?

    • #12
    • October 8, 2015, at 9:43 AM PDT
    • Like
  13. donald todd Inactive

    There was an expression that some of my old confreres used to utter: There is not time to do it right, but there is time to do it over.

    • #13
    • October 8, 2015, at 9:56 AM PDT
    • Like
  14. Tuck Inactive

    Aaron Miller:If it’s possible to explain to a layman, how do programmers arrange that? How do you ensure errors result in instant, epic failure rather than just a memory leak?

    This is probably more than you’d ever want to know:

    Exception handling

    Memory leaks are typically more subtle programming errors that grab and then fail to release computer memory. Not all languages are subject to memory leaks, but C and C++, two of the more commonly-used languages are very susceptible to it.

    • #14
    • October 8, 2015, at 10:04 AM PDT
    • Like
  15. Barkha Herman Member
    Barkha Herman Post author

    Marion Evans:Doubling your failure rate is a great idea if, like Tom Watson’s, your effort culminates in success. Otherwise you are just a bum whose wife left him because he failed too many times :)

    Success / failure eventually comes down to where you stop. If after 999 tries, Edison had stopped, he would be a failure.

    So, see, tenacity has it’s place (a complete doody head like me can attest to this). The point is not stopping actions, it’s to learn from failure. Following that, you keep acting, but with prior failures in mind.

    This post is not about giving up, but moving forward and using failure as a motivator.

    • #15
    • October 8, 2015, at 10:04 AM PDT
    • Like
  16. Solon Inactive

    ‘Success is going from failure to failure without losing your enthusiasm.’ Lincoln

    • #16
    • October 8, 2015, at 10:04 AM PDT
    • Like
  17. Tuck Inactive

    LOL, of course “fail fast, fail often” isn’t always best:

    “For example, in 1996 the maiden flight of the Ariane 5 (Flight 501) ended in a catastrophic explosion due in part to the Ada programming language exception handling policy of aborting computation on arithmetic error, which in this case was a 64-bit floating point to 16-bit integer conversion overflow.[3] In the Ariane Flight 501 case, the programmers protected only four out of seven critical variables against overflow due to concerns about the computational constraints of the on-board computer and relied on what turned out to be incorrect assumptions about the possible range of values for the three unprotected variables because they reused code from the Ariane 4, for which their assumptions were correct.[5] According to William Kahan, the loss of Flight 501 would have been avoided if the IEEE 754 exception-handling policy of default substitution had been used because the overflowing 64-bit to 16-bit conversion that caused the software to abort occurred in a piece of code that turned out to be completely unnecessary on the Ariane 5.”

    • #17
    • October 8, 2015, at 10:09 AM PDT
    • Like
  18. Owen Findy Member

    donald todd:There was an expression that some of my old confreres used to utter: There is not time to do it right, but there is time to do it over.

    Yeah, but that was in resignation to what they considered a flaw of human nature.

    • #18
    • October 8, 2015, at 10:18 AM PDT
    • Like
  19. donald todd Inactive

    Owen Findy:

    donald todd:There was an expression that some of my old confreres used to utter: There is not time to do it right, but there is time to do it over.

    Yeah, but that was in resignation to what they considered a flaw of human nature.

    Actually it was about making communications (voice and data) work across nodes by making the nodes recognize and respond to each other by allocating channels and bandwidth to accommodate the voice or data call through its conclusion.

    • #19
    • October 8, 2015, at 10:21 AM PDT
    • Like
  20. Tom Meyer, Common Citizen Contributor

    41k7k0sGzmL._SX324_BO1,204,203,200_Also, there’s this:

    • #20
    • October 8, 2015, at 10:26 AM PDT
    • Like
  21. The Question Inactive

    If you’re going to be creative and innovative, it’s inevitable that you will have failures. Tried and true methods are superior to most innovative methods. That’s why they’re tried and true. Successful innovation requires creating an environment where failure can be tolerated, since you have to search through many innovative failures to find those few innovative ideas that are actually useful. I think that’s basically the purpose of a research and development laboratory. It’s a place where failures are small scale and contained so that they won’t do significant harm.

    I think that the value of universities and academics is that they produce the innovative ideas that we need. Their weakness is that they aren’t particularly good at determining which innovative ideas are valuable and which are not. Academics are good at creating new intellectual material, but given a selection of ideas, are not so good at picking the best ones. This is part of why progressives are wrong to give academics a privileged place in making political decisions. Academics are good at creating ideas, but not so good at determining which ideas are failures and which are not.

    • #21
    • October 8, 2015, at 10:28 AM PDT
    • Like
  22. Barkha Herman Member
    Barkha Herman Post author

    Aaron Miller:If it’s possible to explain to a layman, how do programmers arrange that? How do you ensure errors result in instant, epic failure rather than just a memory leak?

    Aaron – the article I linked really is a good read despite the use of code. One of the most concrete examples I use on a daily basis is continuous integration.

    I am currently working on a project that has several developers, designers etc. with the usually high level of complexity; and each one of us has the potential to check in a “breaking change” – i.e. something changed that breaks someone else’s use of it. Continuous integration is merely a fancy term for saying that every time code is checked in (to a repository), an automated build ensues (and tests run in our case) and let’s me know if I caused the build to fail or broke someone else’s code.

    The example in the linked article is about reading something from a file. Suppose it’s a tax rate. if there is an error in reading (someone fat fingered the input), one may initialize the tax rate to zero, costing the business money, not to mention IRS headaches.

    If G3 were to jump into his conversation, he would tell you how Scala barfs on any such erroneous assignments, having some built in fail fast concepts.

    • #22
    • October 8, 2015, at 10:47 AM PDT
    • Like
  23. oleneo65 Coolidge

    Carey J.:

    Barkha Herman: Fail Fast, Fail Often

    I think that’s the Obama administration’s motto.

    Unfortunately for us all, this admin does not admit to any failure. Hopefully there are enough voting citizens who see the failures and are open to a different course.

    • #23
    • October 8, 2015, at 11:16 AM PDT
    • Like
  24. Midget Faded Rattlesnake Moderator

    Barkha Herman:

    Marion Evans:Doubling your failure rate is a great idea if, like Tom Watson’s, your effort culminates in success. Otherwise you are just a bum whose wife left him because he failed too many times :)

    Success / failure eventually comes down to where you stop. If after 999 tries, Edison had stopped, he would be a failure.

    So, see, tenacity has it’s place (a complete doody head like me can attest to this). The point is not stopping actions, it’s to learn from failure. Following that, you keep acting, but with prior failures in mind.

    Sure, and every once in a while, a learner is going to get unlucky and learn the “wrong thing” from failure. That he really is a loser. That all his accumulated priors indicate that he is vanishingly unlikely to do anything but fail going forward.

    This post is not about giving up, but moving forward and using failure as a motivator.

    Big-picture, I like this life-lesson. I wish more people believed it. I wish I believed it more strongly myself :-)

    What I’d point out, though, is that commitment t0 this life-lesson sometimes does require a triumph of faith over experience. And where this faith in eventual success comes from (especially when all accumulated evidence quite reasonably predicts more failure) is something of a mystery.

    • #24
    • October 8, 2015, at 11:57 AM PDT
    • Like
  25. Barkha Herman Member
    Barkha Herman Post author

    Midget Faded Rattlesnake:

     And where this faith in eventual success comes from (especially when all accumulated evidence quite reasonably predicts more failure) is something of a mystery.

    All accumulated evidence is based on what worked in the past. All creativity (hence experimentation, innovation, etc.) requires new ideas.

    Yes, of course more failures exist than successes. That is precisely why one needs to move on from failures. Statistically speaking, you need failures to succeed.

    And yes, faith is elusive; and the key.

    • #25
    • October 8, 2015, at 12:10 PM PDT
    • Like
  26. I Walton Member

    Barkha Herman:

    Yes, Taleb and McCardel are good readings on this. Are we teaching our kids, though, that failure is not all “bad and wrong”?

    You mean everybody gets a trophy, all earn A’s and no rough play during recess? I’d guess not.

    It is not just that governments are monopolies, they aren’t part of the information system we call markets, have no prices, profits, no instant feed back to give rise to adjustments and if some person or office gets something right, there are no price signals to tell them, nor incentives for others to adopt the successful strategy. Moreover their monopoly is enforced at the point of a gun.

    • #26
    • October 8, 2015, at 12:24 PM PDT
    • Like
  27. Barkha Herman Member
    Barkha Herman Post author

    John Penfold:

    Barkha Herman:

    Yes, Taleb and McCardel are good readings on this. Are we teaching our kids, though, that failure is not all “bad and wrong”?

    You mean everybody gets a trophy, all earn A’s and no rough play during recess? I’d guess not.

    It is not just that governments are monopolies, they aren’t part of the information system we call markets, have no prices, profits, no instant feed back to give rise to adjustments and if some person or office gets something right, there are no price signals to tell them, nor incentives for others to adopt the successful strategy. Moreover their monopoly is enforced at the point of a gun.

    Exactly.

    • #27
    • October 8, 2015, at 12:27 PM PDT
    • Like
  28. Dr. Strangelove Thatcher

    Tuck: I hadn’t started this way, but learned pretty quickly. But some of the old code didn’t have the error-handling. I had a default error message at the root of the thing that was never supposed to be hit, but would let me know that the error-handling hadn’t been implemented for the code that had failed. In a whimsical moment, I unfortunately used the phrase “Who the f— wrote this thing?”

    Of course years later, the day came when my phone rang and a user said, “I got a strange error message on my screen…” I’d long forgotten about it. Happily we’d been working together for a while, and he found it as amusing as I’d meant it to be.

    Heh, I can top that! But I’d say this is more embarrassing.

    Sometime in the mid-90’s I was a consultant at a certain company. The group I was part of was mostly consultants and a few directs. They were continually hiring so at any time a few guys had only been there a couple of weeks.

    One of their “systems” (if you wanted to call a collection of bugs flying in formation a “system”) needed a touch of maintenance. I got the task to fix a couple of defects.

    I spent a couple of aggravating days of trying to decipher how that pile of spaghetti code functioned. It’s author was the sort who had no regard for compiler warnings; if the syntax didn’t cause the compiler to abort with an error then regarded that C code was fine. Nothing prepared me for the flood of compiler warnings that ensued when, as is my practice, I cranked the C compiler’s sensitivity for emitting a warning to its max. It was very deficient work-product that managed, in spite of its own problems, to produce serviceable output.

    Sometime during day three of this effort I couldn’t contain myself: I wrote a long scathing rant about this pile of junk as a long comment in the code! Then I somehow forgot to delete the rant.

    A few weeks later, one of the technical leads–a direct–decided to conduct a code walk-through on this same system as a sort of orientation for the team. A couple of computer monitors were set up on the table so everybody could see the code as he was paging through it in the editor. Of course, it happened: Somewhere during the code-walk-through he encountered my sarcastic rant in front of the entire team.

    He was a bit upset. The team was snickering at my choice of words. And he couldn’t get wise as to which jerk wrote these comments because they weren’t using source code control systems. So I got away with this bit of personal stupidity unscathed.

    • #28
    • October 8, 2015, at 1:12 PM PDT
    • Like
  29. Midget Faded Rattlesnake Moderator

    Barkha Herman:And yes, faith is elusive; and the key.

    Agreed.

    • #29
    • October 8, 2015, at 1:18 PM PDT
    • Like
  30. Barfly Member

    I always cringe at this kind of analogy, specifics details of one realm extended to draw completely unrelated conclusions. Fail-fast software operations have nothing at all to do with business planning. But at least you didn’t use the analogy to try to argue something obviously false in its own realm, so … ok.

    Since you’re committed to magical thinking anyway, here’s some feedstock: Fail-fast software operations are the norm for general purpose library modules, a judgement call for more integrated special purpose modules, and obviously inappropriate for main-loop or supervisory routines. You should be able to spin gold from that.

    • #30
    • October 8, 2015, at 1:28 PM PDT
    • Like
  1. 1
  2. 2