Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
Working Oneself to Death
This little missive provoked a lot of likes in the running commentary we call the PIT at Ricochet.
I was poking fun at the nature of many of my fellow engineers, for whom the uncertainty of a steadfastly predictive universe meets at that boundary of surety. By nature, I would venture to submit that engineers are more comfortable with a Newtonian view of the universe, tolerates when it gets Einsteinian, and go bonkers when Heisenberg starts babbling on about cats in boxes.
I claim that I share that tribal instinct, and during the course of my career, I have witnessed just how far that can go. But it requires a bit of background.
As an upcoming engineer, I was tasked to support the development of a new generation of remote sensing instruments. There was considerable analytic uncertainty, which plagued all academic attempts to model our global climate. NASA was the logical government agency to determine the consensus of what data is needed, and develop orbital sensors to make what eventually boiled down to 17 critical data sets. I’m a mechanical engineer by training and probably also by mindset, with my expertise in thermodynamics and heat transfer, which is space is already a bit of boutique specialization. To be effective I had to understand other disciplines to be effective in my design decisions, and instrument development requires a lot of give-and-take between all of the disciplines to get something to work.
Part of my education (because we are always learning well after the formal college degree, right?) came with learning the nature of imaging things in wavelengths not in the visible spectrum. Lots of interesting physical processes that affect weather and climate occur in the infrared wavelengths, and the physics of photon sensitive materials requires that they be held at cold temperatures. So my job focus became, how do we get those imaging materials cold on orbit. Given that space is cold (the Universe’s background temperature, leftover from the big bang is around 4 degrees Kelvin—in Fahrenheit, a toasty 452 degrees below zero), how hard is it to tap into the infinite sink of cold to chill our detectors?
Water freezes at 273K, and I feel miserable when it is less than 293K (20C or ~70F). Most of the electronics we use operate at shirt-sleeve temperatures, and we need various electrical systems to extract the photons collected on the detector materials which are happy around 80K. Remote sensing is the counting of those photons, turning them into electrons, aligning them in time and space and producing an image of wavelength-specific recordings as the spacecraft carrying all of those instruments circles the earth. The heat from those electronics wants to conductively and radiatively migrate to a cooled detector, because all of these systems (detectors, their optical components, and their counting electronics) need to use minimal power, be packaged tightly, and weigh as little as possible for the trip to orbit.
I was not the first guy to come to this set of requirements, and there are two ways to cool a detector (colloquially called a Focal Plane Assembly, FPA). One is old school, using a radiant cooler (i.e., something that looks at the deep cold of space to exchange its heat) or the second way: more commonly we now use a mechanical refrigerator, which in the 1990s was in its infancy for space flight applications. One of my jobs (should you accept this assignment Mr. Phelps) was to develop an in-depth understanding of this radiant cooler technology and scale it up to the large multi-wavelength FPA’s that were envisioned for doing global measurements. Sounds straightforward and simple right? Well, certain physical aspects work against the idea of scaling up.
Everything that houses the instrument’s FPA radiator, which is pointing at deep space, has its own temperature and is going to be warmer than that radiator, and the design task is shunting unwanted heat away from the cooler’s coldest radiator and the FPA’s. Low earth orbits are needed for global monitoring. This avoids huge systems of optical elements and eliminates multiple heat sources. Beyond the internal instrument heat sources, is the sun (which is why we are warm and there is life on earth), the reflection of the sun off the top of the atmosphere, and the IR radiation from the surface of the earth (everything that has a temperature above absolute zero radiates heat) all heat the outside of the instrument. Most radiant cooler designs have a system of “shields” that keep direct impingement of those environmental sources off the radiator. The shields themselves have a temperature, warmer than the FPA’s radiator, and thus become a source of unwanted heat. A mirror-like shield surface can minimize that heat but it has to near perfect (99.9% specular is great, 99.5% is good, but 98% specular forget it, you’re done, too much returned energy). The FPA’s, and all of the radiant cooler’s shields have to be located and conductively isolated from the rest of the instrument. Isolating mounting materials tend to be exotic, brittle, and have very limited conductive measurement data.
At the time I was learning this technology only a few commercial or governmental sources had successfully developed and demonstrated these devices. In a pre-computer analytical modeling era, designing was done with line-of-sight drawings, hand calculators, and tables to determine the “view factors” for the radiant heat exchange between the various physical elements comprising the radiant cooler.
The typical engineering approach to designing anything new is to build in measures of conservative assumptions to deal with uncertainty. Unfortunately, the standard design approach to orbit thermal design philosophy would mean the radiant cooler could never be demonstrated on paper to work. The very small quantities of heat energy you are designing around are difficult to measure. Most instruments consume tens to perhaps a couple of hundred watts. An entire spacecraft will have perhaps a total budget of 2,000 watts. The radiant cooler is managing a budget of 0.040 to 0.060 watts for an FPA, and then absorbing and rejecting all the parasitic heat leaks from every source of heat outlined above.
Scaling up a radiant cooler increases those radiant heat loads exponentially. The growth in mass of the radiator makes all conductive isolation points get bigger in a non-linear fashion as well. Accurate simulation space on the ground is difficult (as well as expensive) and getting close to impossible when one is looking at demonstrating operational temperatures below 100K. The error bars involved with testing will affect the certainty of a cooler design.
Over the course of a year or two, I was immersed in this educational process while I was working with the engineers at a small firm in Southern California called SBRC. I was thick in the designing of substantial modifications for a design that they had demonstrated with reasonable success on the first few Landsat missions. By the mid-’90s the newer thermal analytic tools we were using gave us some confidence that we could run tighter into the “design box”, and add enhancements from the original radiant cooler they had, but it was seriously frustrating dealing with the physical unknowns.
To mitigate the risks, we built an engineering model which included our modification ideas, but did not include everything once the cooler was integrated into the instrument. Our test was fairly close to the modeling results we predicted, even a tad better than expected. However, when this engineering model was integrated with an FPA and installed into the engineering model of the whole instrument it failed… big time.
This became an intense situation. The engineer who was leading up the internal radiant cooler design work left the company after the engineering model test because a corporate merge would have relocated his young family to Los Angeles–not acceptable to him. I was left trying to explain to his management how to determine what our issues were, but it would take a few weeks of valuable (and expensive) thermal vacuum tank time. The Project Manager, an old Air Force veteran, did not like this bit of reality: The physics limitations of his instrument’s performance. He declared this was unacceptable and dressed me down for “playing research in his chamber”. I offered to bow out and let his team resolve this, which given the size and scope of this company was not possible. At this point, in front of the remaining mechanical engineers responsible for the radiant cooler, he dressed me down in four-letter words typical of certain military situations and stormed from the room. The poor souls whose professional lives would be miserable if I left asked me if I was serious about leaving. I said no, but I saw no other way to find out what changes were introduced from installing the radiant cooler into the instrument that substantially increase the unexpected heat energy. They quietly agreed to run my tests without informing the PM, as long as I did not pull a “Dana.”
What is a Dana?
Dana was the young, single engineer who originally designed the cooler we were modifying. He faced long hours over the course of a couple of years, all of the uncertainty I described with this class of thermal devices, and without the analysis tools we had to further limit our uncertainties. One night after instrument-level thermal vacuum testing, the ambiguity of the whole process and his personal investment in making this work was too much. He did not want to wait and see how it would perform when it reached orbit and the possible failure of his efforts. So after a late night at the office, he was driving to his home in the San Ynez Valley, stopped at the bridge over the Cold Spring Creek in the San Ynez mountains and jumped. The uncertainly literally ate away his ability to reason rationally.
As agitated as I was between not knowing what was happening to “My Cooler” which per the PM I inherited earlier that day, and the boiling pot of this instrument being behind schedule, as well as heading to over budget territory, I did not think I was going to pull the plug. Eventually, between some in-situ tests, and some forensics of what parts were introduced since the earlier successful assembly-level test, I was able to figure out what other folks introduced that increased the heat loads.
Six months later after the instrument was done and delivered, I was back in my pen at GSFC. My branch head asked me to sit in on a meeting with the corporate heads of A.D. Little, one of our vendor sources, who also builds the occasional radiant cooler. At the meeting, the VP and head of engineering wanted to know if we had any prospective business for their radiant cooler products. Given the nature of how we contract, and until a program has a need, we could not give any assurance, nor guarantee they would be selected.
They then informed us that it was a product line they were going to discontinue since their only thermal engineer who picked up the radiant cooler design mantle from the original designers from the 1960s hung himself in his living room. Seems the work dynamics of his specialty, and possibly some home issues, made him pull his plug.
There are probably fewer than ten active engineers in the US over the last twenty-five years who have design and development experience with this class of thermal device. Two out of less than a dozen seems a bit high for a self-inflicted mortality rate.
There is an air of gallows humor with those who come into this orbit when contemplating incorporating a radiant cooler into their instrument design. We need to remind ourselves not to focus on only the uncertainty the universe has for our prescriptive natures, and to chance a look at the cat that is in that box, and cross our fingers he is still alive.
I just started reading this post and I’m already loving it.
Got to run, have a sparkly gem of a Ricochet article to enjoy.
The textbook opening was very, very funny to those of us who like that sort of humor.
Its effect on the reader is extraordinary not for that reason alone, if he has spent much of his life-quota of avocational thinking time on science. It is deadly serious. The eager college students probably didn’t notice that at the time, especially since most of them were thoroughly Engineerish in their habits of thought.
Engineering is a quiet profession. The progress of our very comfortable lives has been because of the thinkers and risk takers. I followed the collapse of the Florida Bridge, that when watching is kind of heart wrenching. Engineers are not given enough credit when things go well, we take them for granted.
Thank you for this article. I only understood a part of it, especially the part where you wrote:
And the length of time to create these systems is under estimated by the public. Once a solution if found, it looks like it is right in front of you. The critics are quick to say it is simple. An example is brain teasers where, when you look up the solution, it makes sense. But someone had to spend a lot of time and thought, compared with some basic knowledge and understanding of the problem to come up with a solution, or solutions.
Thanks V man,
I really tried hard to avoid the trope of ask an Engineer what time it is and he tells you how to build a watch. However I was finding it difficult to convey how hard it is to design a system that has large and difficult to measure uncertainties, and you cannot just follow the typical “just over design script”. Ply on the inability to completely test it on the ground, with enough limiting of the measurement errors, and you get stomach churn.
Some of these sensors had half or more of their science hingeing on detectors being cold enough. No one want to be responsible for failing to meet that instrument’s requirements with few hundred million dollars worth of public space sensor. Then add in the type A personalities that seem to find a home in the engineering arena and you get a least a lot of sleepless nights when the test results don’t make sense.
Yeah good times all round.
An interesting story. My father could never understand suicide. He faced financial ruin through an out of control US attorney, but never turned away from life. Sad to hear that the pressures of design under uncertainty pushed the two engineers you mentioned over the edge.
And then there’s you, @gldiii, a brilliant engineer who took a sabbatical several years ago to build a private plane from scratch (just for fun!!) and who still lends your expertise to new high-tech projects when asked even though you’re ready to retire. You have doggedly persevered all these years, and because of your team’s efforts we now have reliable weather forecasting capabilities unheard of a few decades ago. But even more than that, you care for family members and step up at key moments to offer support and friendship to Ricochet members you barely know.
This March I seriously needed a break as a caregiver so I asked if anyone from Ricochet planned to attend the biennial National Review Institute Ideas Summit (several of us had attended the last summit two years ago). Nobody else planned to attend this year, but when @gldiii learned I was going to fly up, stay with my sister in Arlington, and hitch a ride with her each morning and evening so I could attend both days of the conference on my own, he bought a ticket too. He saved a seat for me at a good table, cheered me on when my education question was the first one asked of Education Secretary Betsy DeVos, and made it possible for me to not have to stand alone during breaks as we chatted with fascinating people from around the country.
That week I learned that @gldiii is a member of the Knights of Columbus and also has helped with his family’s caregiving needs. Is that generous spirit the secret to surviving for decades while working at the highest levels in the field of engineering? I don’t pretend to know. What I DO know is that because of GLD’s support, those two days in Washington buoyed the spirits of this tired caretaker tremendously, and I returned home re-energized and ready to make a difference again in my small corner of the world.
In its first decade Ricochet has become an online gathering place for very smart (and often brilliant) people to gather and know they are valued for more than just their accomplishments and/or failures. I want to thank you, GLDIII Temporarily Essential, for your decades of work that have made it possible for me to know it’s going to be cool in the mornings all week here in Florida and brutally hot by late afternoon. But thank you even more for being a gentleman and a friend this March when I really needed one. You are the very best of Ricochet—-so take good care of yourself as you wind down your current projects, and when it’s time to retire just do it!!
I am pleased to know, having switched to psychology from engineering, that some of you guys at least, art practicing art (as opposed to science) as much as I am.
It does happen. When you’re stuck in a problem and your team can’t or won’t help you climb out, and the situation persists and can’t be fixed, it affects your whole outlook.
But for the real exemplar of work that drives one mad, see Georg Cantor.
My goodness you are making me sound way to worthy ☺️
I figure you can look back on that wonderful instance of altruism the next time you’re being NOT so worthy, lol!
In your father’s case, he had an individual person who was responsible for his problems. Kudos to fighting back.
I’ve never contemplated suicide, but at the midpoint of my career (~ 36 years old) I went through a 2 year low spot, with anxiety followed by depression. The old saw about engineers being used up at 40 weighed heavily on my outlook. But when 41 years old, I had a major patent breakthrough.
I might not use my physics degree much these days, but I remember enough of it to know exactly how challenging a problem this is and to be suitably impressed with the solutions you’ve found. The fact that there is a solution at all is impressive!
That particular Radiant Cooler was launched in 1999. It is still up there, and still working, and still supplying measurements to the Global Climate Data Base (even if some of the scientists are not happy with the data not confirming run away global heating).
The original build to requirement was for 5 years with a goal of 7.
I posted about it’s birthday last December.
GLD III, just the topics you covered in the OP had me thinking about suck-starting my 9-mil.
what is that?
The way to worthy is paved with good intentions.
Our son (electrical engineer working in research and development in the United States Air Force) has risen to responsibilities I would not have predicted for him apparently in part because he understands and is willing to embrace more uncertainty than most of his engineering peers. He recently told me that he had to argue with the engineering team that wanted to build one failure-proof (and thus very expensive) prototype, while he thought it more informative for the eventual production roll-out of the program to build for the same amount of money several prototypes and see where the failures occurred. Many of the line engineers were really bothered by the idea that a prototype would fail. But he convinced them that was the purpose of the prototype.
This is why we’re develop “Engineering Models” of anything that is not of an acceptable level of technology readiness (we call them TRL’s, 1 to 9). The issue with this little slice of hades I was discussing is that the ability to accurately simulate the space environment. Without all of the reflections of energy happening inside a TVAC chamber, (yes there is re-reflections of energy in the infrared wavelengths). This is a real tail chasing endeavor.
This is why I fear for the folks who have been toiling away for more than 10 years trying to developed the JWST, which is the longer wavelength viewing replacement for the Hubble ST. They can not do a system level end to end test on earth. They have to wait until they reach L2 after dozens of mechanical deployments, and then see if it all cools down to around 40K to 50K on the instrument module. They have done dozens of small tests to gain confidence, but still ……. one can get burned.
We occasionally have engineering graduates who decide to go into medicine. Learning how to deal with the uncertainties of biological systems is sometimes a strain for engineers, who, as it was explained to me, are used to more precision, to accounting for all the variables, and so on. On the whole, engineers can contribute a great deal to medicine if they can do the work of adapting.
Doctors do a lot of things that can’t be fully justified with the available data, and that can drive a person trained to be an engineer crazy. But I never realized how crazy they might get.
It’s such a big issue that Texas A&M has decided to open a special medical school for engineering graduates. Learning how to prioritize data, discount information that’s irrelevant, and how to deal with unknowns is a big part of it, as I understand it.
I wonder, has there been an actuarial study of this field? Who wants to underwrite that whole life policy?
This post is part of our Group Writing Series under the March 2020 Group Writing Theme: “Working.” There are plenty of open days, so get busy and work it! Stop by and sign up now.
Interested in Group Writing topics that came before? See the handy compendium of monthly themes. Check out links in the Group Writing Group. You can also join the group to get a notification when a new monthly theme is posted.
Personally I prefer “suck start my shotgun” because of the alliteration, but I wouldn’t want to argue with the Boss Mongo on that one.
I sometimes wonder if confronting entropy in all its nihilistic futility might not be dangerous for the materialistic mind.
Thanks. I’m educated.
Now ask him to explain the one about the eyeball.