Bottom line

The Apollo 11 overload was not missed because NASA failed to test the guidance computer. The computer and its restart logic were tested extensively. What testing missed was a rare, analog electrical interaction between two independently powered subsystems that the software simulators did not model correctly.

More precisely, the underlying effect had actually been observed during earlier spacecraft testing, but it did not produce computer alarms because the computer was lightly loaded. The observation remained an obscure hardware-interface anomaly rather than being connected to the nearly saturated processor workload during lunar descent.

What produced the alarms

When the rendezvous-radar mode switch was in AUTO TRACK or SLEW, the radar antenna resolvers were excited by an 800-Hz power source in the Attitude and Translation Control Assembly. The radar coupling-data units, however, used another 800-Hz signal from the Primary Guidance and Navigation System as their reference.

The signals were locked to the same frequency, but they were not locked in phase. Under most phase relationships, this caused no serious problem. Near particular phase angles, approximately 90 or 270 degrees, the coupling-data units interpreted the resolver signals as an impossible or rapidly changing antenna position. They then ordered the guidance computer to increment or decrement its radar-angle counters at nearly the maximum rate, up to 6,400 operations per second on each of two channels. (ibiblio)

Those operations were implemented as involuntary memory-cycle steals. At the maximum combined rate of 12,800 increments per second, they consumed about 15 percent of the Lunar Guidance Computer’s processing time. The Apollo 11 mission report says the computer was already operating at about 90 percent of capacity during the busiest portions of powered descent. (ibiblio)

The overload sequence was therefore:

1. The radar interface silently stole roughly 15 percent of processor time.
2. Guidance and navigation jobs took longer to finish.
3. The Executive scheduler started new copies of periodic jobs before previous copies had completed.
4. Temporary work areas, called core sets and VAC areas, filled up.
5. Alarm 1202 indicated no available core set; alarm 1201 indicated no available VAC area.
6. The computer performed a software restart, discarded unfinished low-priority work and restored essential guidance functions. (ibiblio)

The computer did not truly crash in the modern sense. Its overload protection did exactly what it was designed to do.

Why testing did not expose the complete failure

1. The software simulator omitted the crucial electrical behavior

MIT’s primary software-verification environment was an all-digital simulation. It modeled the computer and flight software, but it did not reproduce the peculiar resolver and coupling-data-unit behavior created by two out-of-phase 800-Hz supplies.

George Cherry’s August 4, 1969 postflight memorandum states explicitly that, had the simulator generated the 6,400-pulse-per-second radar inputs when the radar switch was outside LGC mode, the overload would have been detected during software verification. (ibiblio)

This was the central testing hole: the software simulator received idealized radar data, not the electrical noise produced by the real interface.

2. The system test laboratories accidentally eliminated the fault

A later technical reconstruction by Apollo guidance engineer Fred Martin Blair reports that the ground laboratories used a single 800-Hz supply for the relevant simulated systems. That forced the phase difference to zero.

The flight Lunar Module had two independently phased sources. The test laboratory therefore simplified the design in a way that removed the very condition that caused the anomaly. Only when actual flight hardware was assembled at Kennedy Space Center could the independent phase relationship appear. (fredhaise.space) This is a classic hardware-in-the-loop failure: the test bench looked electrically cleaner than the spacecraft.

3. The bad condition was established randomly at power-up

The phase difference was established when the guidance system was powered and then remained essentially fixed. Only a narrow range of possible phase relationships caused maximum counter activity. Blair’s reconstruction estimates that the troublesome condition appeared in roughly one out of 70 LM-3 test runs. Random repetition was therefore a poor way to find it. Under a simplified one-in-70 probability model:

• After 70 independent power-ups, there would still be about a 37 percent chance of never seeing it.
• Approximately 209 power-ups would be needed for a 95 percent chance of observing it at least once.

A controlled phase sweep would have found it in one test campaign. (fredhaise.space)

4. The effect had been seen, but not under landing-level workload

The most important historical wrinkle is that the anomaly was not completely unknown. According to Blair and Don Eyles, abnormal radar-interface counter activity appeared during testing of LM-3, the Lunar Module later flown on Apollo 9. George Silver identified the phase relationship and documented it. But the computer was lightly loaded during those tests, so the result was disrupted telemetry rather than 1201 or 1202 alarms. (fredhaise.space) The engineers therefore saw: Unexplained radar-related computer activity. They didn’t see: Lunar-descent software fails its timing budget and repeatedly restarts. The symptom was filed in the hardware and telemetry world. The consequence belonged to the software-performance world.

5. The timing analysts and radar engineers did not connect their information

Cherry’s postflight memorandum is unusually candid. It says that some people understood the radar resolver interface, while others understood what a 15 percent loss of computer time would do to the landing program, but those groups did not bring the facts together. He consequently proposed joint hardware-software reviews, higher-fidelity simulator configuration control and hardware review of crew checklists. (ibiblio) This was an organizational interface failure mirroring the electrical one:

Group	What it knew	What it apparently did not connect
Radar and electrical engineers	Independent 800-Hz sources could produce strange CDU activity	The landing computer had almost no remaining timing margin
Software engineers	The descent program could tolerate roughly 10 percent time loss	The radar interface could steal roughly 15 percent
Simulation engineers	Their models reproduced nominal subsystem behavior	Their single-source model removed an important phase state
Crew-procedure planners	AUTO/SLEW kept the rendezvous radar ready for an abort	That switch state exposed the computer to meaningless CDU interrupts
Flight controllers	Restart alarms could be survivable if navigation remained valid	The precise physical cause of the overload

The incident was therefore not primarily a coding mistake. It was a systems-integration and information-routing failure.

6. Testing emphasized expected failures more than hostile combinations

NASA ran hundreds of mission simulations, and flight controllers were exceptionally well prepared to interpret alarms. Gene Kranz had required the computer specialists to document every possible alarm and its operational consequences, even when they couldn’t imagine how the alarm might occur. That preparation enabled Jack Garman and Steve Bales to recognize that the computer was recovering and that guidance remained valid. (NASA) But Garman later recalled that the actual processor-overload condition hadn’t, to his knowledge, been deliberately tested. Restart behavior had been exercised, including manually initiating restarts during simulations, but the exact combination of radar interrupts, flight software load and crew display activity hadn’t been reproduced. (NASA)

That distinction was:

• Recovery from a restart was tested.
• A sustained real-time overload caused by the radar interface wasn’t adequately tested.

The combination that had to occur

No individual condition was sufficient. The alarms required several pieces to click together:

• The two 800-Hz supplies had to start with a bad phase relationship.
• The rendezvous-radar mode switch had to be in AUTO TRACK or SLEW rather than LGC.
• The computer had to be running the high-load P63 or P64 descent software.
• Landing-radar processing and other periodic jobs had to be active.
• Crew-requested monitoring displays added further work.
• The accumulated workload had to exceed the Executive scheduler’s available time and temporary storage.

Most tests exercised these elements separately.

Could they have discovered it before flight?

Yes. Nothing required technology unavailable in 1969. Several practical tests would almost certainly have exposed it.

1. Sweep the relative phase of the two 800-Hz sources

The strongest test would have used two independent, phase-adjustable 800-Hz supplies and swept their relative phase from 0 through 360 degrees while monitoring radar CDU counter rates. That would have immediately exposed the narrow danger zones near 90 and 270 degrees, rather than waiting for a random power-up to land there.

2. Inject worst-case radar interrupts into the software simulator

MIT could have modified its digital simulator to inject:

• 6,400 positive or negative counter requests per second on the shaft channel;
• 6,400 per second on the trunnion channel;
• the exact P63 and P64 workload;
• the crew’s planned display commands.

Cherry’s postflight memo specifically says this simulator feature would have detected the overload. (ibiblio) This would now be called fault injection or timing-stress testing.

3. Repeat the exact flight checklist against the exact flight load

The test should have included the actual sequence:

1. Power the guidance system.
2. Put the rendezvous radar in the checklist-specified mode.
3. Run P63 and P64 in real time.
4. Activate landing-radar processing.
5. Enter the same monitoring verbs planned for the crew.
6. Measure remaining processor margin and core-set occupancy.

Cherry noted that the Apollo 11 checklist called for the radar switch to be placed in AUTO TRACK before P63 and that the software group had not understood the timing consequence. (ibiblio) A checklist is executable system configuration. It should be tested as though it were code.

4. Treat unexplained processor-time losses as flight hazards

The LM-3 anomaly should have triggered an end-to-end investigation:

• What physical condition causes the counter activity?
• What is its maximum possible rate?
• Can it occur with flight procedures?
• What is the worst-case software load at that time?
• What happens when the two are combined?
• Does the test equipment preserve the same electrical architecture?

The crucial missed calculation was almost embarrassingly simple once the facts were assembled:

90% nominal CPU load + 15% radar time loss > 100%

5. Add a pre-descent diagnostic

Blair proposes a plausible operational check using 1969 capabilities: briefly place the radar in the vulnerable mode after guidance-system power-up and have Houston inspect telemetry for excessive counter activity. If detected, the crew could recycle guidance-system power, producing a new random phase relationship, and test again. (fredhaise.space) That wouldn’t have been as elegant as correcting the interface, but it could have screened Eagle before powered descent.

6. Add a software interlock

The permanent solution for subsequent missions was straightforward. Luminary 1B was changed so that when the rendezvous radar was not in the computer-controlled LGC mode, the software zeroed the radar coupling-data units and suppressed the meaningless counter interrupts. The Apollo 11 mission report records this corrective action. (ibiblio) That fix showed the incident didn’t require a larger computer or a redesigned radar. It required the software to reject input that could not be operationally useful in that switch configuration.

Assessment

The Apollo 11 alarms are often presented as a triumph of brilliant fault-tolerant programming, which they were. But the fuller engineering lesson is less comfortable. The restart system was excellent but the test architecture was incomplete. NASA tested the computer, the radar, the spacecraft, the crew and Mission Control extensively, but its models did not preserve one obscure physical property of the complete system. An anomaly observed in real hardware was not connected to the flight software’s timing margin, and the exact operational configuration was never exercised under worst-case electrical and computational conditions. So the answer isn’t, “They could not have known.” They could have known. In fact, they possessed most of the clues. What they lacked was a test and anomaly-management process that forced those clues into the open.