WASHINGTON — A NASA investigation blamed millions of dollars in damage to one of its largest Deep Space Network (DSN) antennas on poor training and procedures.
NASA released this month a redacted version of the final report investigating an incident at its DSS-14 antenna, a 70-meter DSN antenna in Goldstone, California. That antenna has been offline since the Sept. 16 incident.
The antenna suffered damage when it over-rotated, stressing and breaking cables and hoses, including those carrying water for a fire-suppression system. That flooded the base of the antenna with more than 750,000 liters of water that also contained glycol.
The incident caused between $4.1 million and $4.6 million in damage to the antenna, according to the report, enough to classify the incident as a “Type A” mishap by NASA.
The investigation identified six critical events leading up to the mishap, although the public version of the report redacts nearly all details about those events. Those events started an unspecified period before the mishap when a hydraulic limit system for the antenna became inoperable.
There was an anomaly Sept. 15 during a communications session with the Juno spacecraft, according to the report’s timeline, leading to maintenance and troubleshooting. The antenna was then repeatedly driven into rotation limits, causing troubleshooting that led to the antenna being inadvertently over-rotated during another communications session with Juno on Sept. 16. There was additional damage when controllers tried to stow the antenna, causing further over-rotation.
The investigation identified several root causes for the mishap. They included personnel at the site who were not properly trained, procedures that were inadequate and a facility that was “overly reliant on undocumented behaviors and institutional knowledge.”
“The investigation cited some issues, workplace culture, inadequate procedures, undocumented practices and some unclear roles and responsibilities,” Greg Heckler, deputy program manager for capability development in NASA’s Space Communications and Navigation, or SCaN, program, said during a June 10 meeting of the Small Bodies Assessment Group.
The report highlighted a reliance on what it called “personal heroics” by personnel at the Goldstone site, described as being “willing to do whatever it takes to keep the antenna running.” The problem with that, the report concluded, is that it led them to do tasks outside their qualifications, work extended hours that led to fatigue and skip tests they felt would have delayed returning the antenna to operations.
Had the site’s personnel “acted with greater deliberation or shown more willingness to leave the antenna in a failed state at any point during the mishap,” the report concluded, “the undesired outcome likely would not have occurred.”
The report provided 20 recommendations, including a need to “incentivize technical rigor over personal heroics” as well as improve training and procedures.
“Our ops team in SCaN is looking across the entirety of the SCaN network, not just the Deep Space Network but also the Near Space Network, given these investigation findings to see if there’s similar behaviors,” Heckler said, “so they can develop a cohesive and comprehensive plan to address that.”
Earlier this year, it appeared the DSS-14 antenna would return to service in May after repairs before going offline again in August for major refurbishment work that would extend to October 2028. However, NASA now says the antenna will remain offline as it goes into that upgrade work.
DSS-14 is one of three 70-meter antennas in the DSN, the largest in the 14-antenna network, which also includes sites in Australia and Spain. Despite having DSS-14 out of service for an extended period, the agency says it has been able to manage the demand for communications services using the other antennas.
That included during the Artemis 2 mission in April, when DSN antennas provided communications while the Orion spacecraft made its trip around the moon. While the Artemis 1 mission in 2022 put a significant strain on the DSN, forcing science missions to give up hundreds of hours of time, Heckler said the DSN’s support for Artemis 2 was much smoother.
“We learned a lot on Artemis 1 and so we put some new processes in place ahead of Artemis 2, mostly focused around coordination and our scheduling processes with all the missions,” he said. “I think that worked well. We got some positive feedback from the Science Mission Directorate after the mission, and we’re continuing to evaluate that success through the post-mission assessment activities.”
