Saturday Night Science: Countdown to a Moon Launch

 

“Countdown to a Moon Launch” by Jonathan H. WardIn the companion volume, Rocket Ranch, the author describes the gargantuan and extraordinarily complex infrastructure which was built at the Kennedy Space Center (KSC) in Florida to assemble, check out, and launch the Apollo missions to the Moon and the Skylab space station. The present book explores how that hardware was actually used, following the “processing flow” of the Apollo 11 launch vehicle and spacecraft from the arrival of components at KSC to the moment of launch.

As intricate as the hardware was, it wouldn’t have worked, nor would it have been possible to launch flawless mission after flawless mission on time had it not been for the management tools employed to coordinate every detail of processing. Central to this was PERT (Program Evaluation and Review Technique), a methodology developed by the U.S. Navy in the 1950s to manage the Polaris submarine and missile systems. PERT breaks down the progress of a project into milestones connected by activities into a graph of dependencies. Each activity has an estimated time to completion. A milestone might be, say, the installation of the guidance system into a launch vehicle. That milestone would depend upon the assembly of the components of the guidance system (gyroscopes, sensors, electronics, structure, etc.), each of which would depend upon their own components. Downstream, integrated test of the launch vehicle would depend upon the installation of the guidance system. Many activities proceed in parallel and only come together when a milestone has them as its mutual dependencies. For example, the processing and installation of rocket engines is completely independent of work on the guidance system until they join at a milestone where an engine steering test is performed.

As a project progresses, the time estimates for the various activities will be confronted with reality: some will be completed ahead of schedule while other will slip due to unforeseen problems or over-optimistic initial forecasts. This, in turn, ripples downstream in the dependency graph, changing the time available for activities if the final completion milestone is to be met. For any given graph at a particular time, there will be a critical path of activities where a schedule slip of any one will delay the completion milestone. Each lower level milestone in the graph has its own critical path leading to it. As milestones are completed ahead or behind schedule, the overall critical path will shift. Knowing the critical path allows program managers to concentrate resources on items along the critical path to avoid, wherever possible, overall schedule slips (with the attendant extra costs).

Now all this sounds complicated, and in a project with the scope of Apollo, it is almost bewildering to contemplate. The Launch Control Center was built with four firing rooms. Three were outfitted with all of the consoles to check out and launch a mission, but the fourth cavernous room ended up being used to display and maintain the PERT charts for activities in progress. Three levels of charts were maintained. Level A was used by senior management and contained hundreds of major milestones and activities. Each of these was expanded out into a level B chart which, taken together, tracked in excess of 7000 milestones. These, in turn, were broken down into detail on level C charts, which tracked more than 40,000 activities. The level B and C charts were displayed on more than 400 square metres of wall space in the back room of firing room four. As these detailed milestones were completed on the level C charts, changes would propagate down that chart and those which affected its completion upward to the level A and B charts.

Now, here’s the most breathtaking thing about this: they did it all by hand! For most of the Apollo program, computer implementations of PERT were not available (or those that existed could not handle this level of detail). (Today, the PERT network for processing of an Apollo mission could be handled on a laptop computer.) There were dozens of analysts and clerks charged with updating the networks, with the processing flow displayed on an enormous board with magnetic strips which could be shifted around by people climbing up and down rolling staircases. Photographers would take pictures of the board which were printed and distributed to managers monitoring project status.

If PERT was essential to coordinating all of the parallel activities in preparing a spacecraft for launch, configuration control was critical to ensure than when the countdown reached T0, everything would work as expected. Just as there was a network of dependencies in the PERT chart, the individual components were tested, subassemblies were tested, assemblies of them were tested, all leading up to an integrated test of the assembled launcher and spacecraft. The successful completion of a test established a tested configuration for the item. Anything which changed that configuration in any way, for example unplugging a cable and plugging it back in, required re-testing to confirm that the original configuration had been restored. (One of the pins in the connector might not have made contact, for instance.) This was all documented by paperwork signed off by three witnesses. The mountain of paper was intimidating; there was even a slide rule calculator for estimating the cost of various kinds of paperwork.

With all of this management superstructure it may seem a miracle that anything got done at all. But, as the end of the decade approached, the level of activity at KSC was relentless (and took a toll upon the workforce, although many recall it as the most intense and rewarding part of their careers). Several missions were processed in parallel: Apollo 11 rolled out to the launch pad while Apollo 10 was still en route to the Moon, and Apollo 12 was being assembled and tested.

To illustrate how all of these systems and procedures came together, the author takes us through the processing of Apollo 11 in detail, starting around six months before launch when the Saturn V stages, and command, service, and lunar modules arrived independently from the contractors who built them or the NASA facilities where they had been individually tested. The original concept for KSC was that it would be an “operational spaceport” which would assemble pre-tested components into flight vehicles, run integrated system tests, and then launch them in an assembly-line fashion. In reality, the Apollo and Saturn programs never matured to this level, and were essentially development and test projects throughout. Components not only arrived at KSC with “some assembly required”; they often were subject to a blizzard of engineering change orders which required partially disassembling equipment to make modifications, then exhaustive re-tests to verify the previously tested configuration had been restored.

Apollo 11 encountered relatively few problems in processing, so experiences from other missions where problems arose are interleaved to illustrate how KSC coped with contingencies. While Apollo 16 was on the launch pad, a series of mistakes during the testing process damaged a propellant tank in the command module. The only way to repair this was to roll the entire stack back to the Vehicle Assembly Building, remove the command and service modules, return them to the spacecraft servicing building then de-mate them, pull the heat shield from the command module, change out the tank, then put everything back together, re-stack, and roll back to the launch pad. Imagine how many forms had to be filled out. The launch was delayed just one month.

The process of servicing the vehicle on the launch pad is described in detail. Many of the operations, such as filling tanks with toxic hypergolic fuel and oxidiser, which burn on contact, required evacuating the pad of all non-essential personnel and special precautions for those engaged in these hazardous tasks. As launch approached, the hurdles became higher: a Launch Readiness Review and the Countdown Demonstration Test, a full dress rehearsal of the countdown up to the moment before engine start, including fuelling all of the stages of the launch vehicle (and then de-fuelling them after conclusion of the test).

There is a wealth of detail here, including many obscure items I’ve never encountered before. Consider “Forward Observers”. When the Saturn V launched, most personnel and spectators were kept a safe distance of more than 5 km from the launch pad in case of calamity. But three teams of two volunteers each were stationed at sites just 2 km from the pad. They were charged with observing the first seconds of flight and, if they saw a catastrophic failure (engine explosion or cut-off, hard-over of an engine gimbal, or the rocket veering into the umbilical tower), they would signal the astronauts to fire the launch escape system and abort the mission. If this happened, the observers would then have to dive into crude shelters often frequented by rattlesnakes to ride out the fiery aftermath.

Did you know about the electrical glitch which almost brought the Skylab 2 mission to flaming catastrophe moments after launch? How lapses in handling of equipment and paperwork almost spelled doom for the crew of Apollo 13? The time an oxygen leak while fuelling a Saturn V booster caused cars parked near the launch pad to burst into flames? It’s all here, and much more. This is an essential book for those interested in the engineering details of the Apollo project and the management miracles which made its achievements possible.

Ward, Jonathan H. Countdown to a Moon Launch. Cham, Switzerland: Springer International, 2015. ISBN 978-3-319-17791-5.

Here is a NASA documentary about preparations for Apollo 4, the first unmanned launch of the Saturn V from KSC in 1967.

This film documents the flight of Apollo 4.

There are 13 comments.

  1. Darin Johnson Member

    “As intricate as the hardware was, it wouldn’t have worked, nor would it have been possible to launch flawless mission after flawless mission on time had it not been for the management tools employed to coordinate every detail of processing.” Flawless?

    • #1
    • January 23, 2016, at 1:24 PM PDT
    • Like
  2. John Walker Contributor
    John Walker Post author

    Darin Johnson:“As intricate as the hardware was, it wouldn’t have worked, nor would it have been possible to launch flawless mission after flawless mission on time had it not been for the management tools employed to coordinate every detail of processing.” Flawless?

    This is the perspective of KSC. Their responsibility for the mission officially ended when the spacecraft cleared the tower and effectively ended when the spacecraft separated from the Saturn booster. Apollo 6 (unmanned) had problems due to design and manufacturing flaws in the booster, but completed its primary mission objectives. Apollo 13 had a premature shutdown of the centre second stage engine due to a design flaw and an explosion of a service module oxygen tank en route to the Moon, but this cannot be traced entirely to processing at KSC.

    • #2
    • January 23, 2016, at 1:41 PM PDT
    • Like
  3. Gary McVey Contributor

    This is the kind of (literal!) process story I love to read. Thanks for bringing it to everyone’s attention, John. You are right to point out that the space program is and was an experimental flight test program, not a turn-key, can’t lose assembly line.

    Though, I have to wistfully say, it sure looked and felt like one for most of the Apollo-Saturn era. In “The French Connection”, when a trio of black women sang “Everybody Gets to Go to the Moon”, it was 1971, and although we knew that “everyone” wasn’t going, the word even on the street was positive: the future of every young person watching could be expected to be enriched by generations that would know the lunar exploration achievements started in our day, something to be proud of. And then we walked away.

    There were sensible sounding reasons why, and Vietnam exhausted us. But we walked away.

    • #3
    • January 23, 2016, at 2:40 PM PDT
    • Like
  4. John Walker Contributor
    John Walker Post author

    Gary McVey: Though, I have to wistfully say, it sure looked and felt like one for most of the Apollo-Saturn era. In “The French Connection”, when a trio of black women sang “Everybody Gets to Go to the Moon”, it was 1971, and although we knew that “everyone” wasn’t going, the word even on the street was positive: the future of every young person watching could be expected to be enriched by generations that would know the lunar exploration achievements started in our day, something to be proud of.

    Indeed….

    • #4
    • January 23, 2016, at 2:58 PM PDT
    • Like
  5. James Madison Member

    When in college I went to work as an intern in state government. I spent my summer updating PERT charts and kept tabs on them. We had a guy from NASA Huntsville who was a consultant. NASA had gotten the moon and was wandering. People were filtering out with skills and tools to sell.

    The charts were used inside different agencies and were considered the “hot” management tool to run very complex projects. We were automating and installing big iron to change the way government worked. IBM was involved, so it had to be good.

    The charts we had were on blueprint sized paper and consisted of rolls that went for 10 plus feet long. They were prepared by draftsmen and done by hand. When they were updated we simply made a new copy from the master PERT chart using the blueprint machine.

    We were so cutting edge.

    • #5
    • January 23, 2016, at 3:54 PM PDT
    • Like
  6. John Walker Contributor
    John Walker Post author

    James Madison:We were so cutting edge.

    Yes, it was crude, but it enabled us to do things we’d never done before.

    • Polaris.
    • Landing on the Moon.

    Today nobody would undertake any serious project without PERT/CPM methodology. Unlike many management fads, this is simply because it has been demonstrated to work for half a century. And, unlike when it was invented and people had to keep and update the charts by hand, it’s almost effortless with computer tools.

    • #6
    • January 23, 2016, at 4:13 PM PDT
    • Like
  7. James Madison Member

    John,

    It was fun to work with. That terminology CPM brings back smiles. And until you wrote this I had not thought of PERT in years. It is embedded in the software and procedures for project management.

    I noticed in the 1990’s our ability to find technologies (know-how, disciplines, procedures) and burn through them seemed endless, Activity Based Costing, Quality Process, Six Sigma, Fishbone diagrams, Decision Trees, Integrated Manufacturing, Competing on Time, Group Technology, and incredible simulating, modeling, and design to build, design to maintain. Hours spent making incremental improvements. Some of these were reheated from the German and American production techniques in the 20’s and 30’s.

    Is there a list somewhere that someone has compiled of all these waves of ideas that got embedded in how we did things?

    • #7
    • January 23, 2016, at 7:43 PM PDT
    • Like
  8. Percival Thatcher

    PERT charts are indeed valuable tools, but there have been projects where their primary benefit was to keep management occupied and away from the engineers so we could get some meaningful work done.

    • #8
    • January 23, 2016, at 8:11 PM PDT
    • Like
  9. Gary McVey Contributor

    Gotta hand it to old Kubrick. He used PERT techniques to manage the workflow of special effects shots for “2001” and was probably the first filmmaker to do so. Almost all of the live action in the film was shot in 1966; a half century ago now. I know it’s not everyone’s favorite (it is mine) but the shots still hold up, even after unimaginable revolution in the creation of film effects.

    • #9
    • January 23, 2016, at 8:43 PM PDT
    • Like
  10. Sabrdance Member

    I have almost finished a novel, set in 1548, about a military operation. In striving for realism, I pointed out that the most important officer in the army is not the commander, nor the strategist, but the logistician -who runs a clearinghouse of all the accounts owed to the victualers and all the mercenary companies, and who ensures that food, money, ammunition, and everything else is taken care of.

    In my research, I learned how important the people managing the supply train were, but I could not find how many men would be involved in such an office (my primary resource actually made a point that this is probably because no one at the time thought it necessary to specify, they would have just done it, which is why it required a book to try to reconstruct it). Figuring that each company would be largely self-sufficient, and that the number of companies came to 3 regiments, which would also have their own logisticians, I gave the army logistician 3 clerks, one per regiment.

    I think I have grossly underestimated the number of people required…

    Also, is it wrong that I read KSC as “Kerbal Space Center?”

    • #10
    • January 23, 2016, at 8:57 PM PDT
    • Like
  11. Jules PA Member

    Thanks for this. I would loved to have done the PERT Charts at NASA.
    As for the rattlesnake thing: there should have been a rattlesnake clearing detail for those 3 teams at the 2km site!

    • #11
    • January 23, 2016, at 9:15 PM PDT
    • Like
  12. Guruforhire Member

    You will probably enjoy this book quite a bit as well:

    http://www.amazon.com/Corvettes-Inside-Rebirth-American-Legend/dp/0671685015

    • #12
    • January 24, 2016, at 4:52 AM PDT
    • Like
  13. John Walker Contributor
    John Walker Post author

    Sabrdance:I have almost finished a novel, set in 1548, about a military operation. In striving for realism, I pointed out that the most important officer in the army is not the commander, nor the strategist, but the logistician -who runs a clearinghouse of all the accounts owed to the victualers and all the mercenary companies, and who ensures that food, money, ammunition, and everything else is taken care of.

    “Generals win battles. Logisticians win wars.”

    Also, is it wrong that I read KSC as “Kerbal Space Center?”

    It isn’t just you. Did you know that among all the people who have flown in space, not one has been named Jebediah?

    • #13
    • January 24, 2016, at 6:53 AM PDT
    • Like