FAA stops installation of HPE Juniper routers in wake of ground stop investigation
In a statement to CRN, HPE said: “Based on what we know and have been told by others, our equipment was working as configured. The FAA is assessing the configuration and performance of all infrastructure involved to determine the cause.”
The Federal Aviation Administration (FAA) Tuesday confirmed that it has suspended the installation of HPE Juniper network routers as it moves forward with an investigation into the nationwide ground stop last Friday.
Transportation Secretary Sean Duffy told reporters Tuesday that it was “prudent” to pause the use of the routers in the wake of the investigation.
The pause comes as the FAA continues its investigation into the installation of an HPE Juniper network router in El Dorado, Arkansas, that flooded the network, leading to the shutdown.
A source close to the incident told CRN that no one has determined that the HPE Juniper device malfunctioned or that flooding from the device was the root cause of the FAA network issue.
Furthermore, the source said, the HPE Juniper device in question is not technically a router but rather is a communications device that converts copper to fibre with a multicast feature.
The source also noted that more than 60 similar HPE Juniper devices were deployed without incident.
The FAA said it addressed the initial cause of the disruption “by immediately disabling the functionality of the originating Juniper routers.”
What’s more, the FAA said, as a precaution it “quickly disabled the same functionality on more than 60 other Juniper routers in the system” and paused installing them.
CRN reached out to HPE Juniper for comment but had not heard back at press time.
But in a statement to CRN on Monday, HPE said: “Based on what we know and have been told by others, our equipment was working as configured. The FAA is assessing the configuration and performance of all infrastructure involved to determine the cause.”
Questions emerge over HPE network configuration
A source with knowledge of the incident said it appears the broadcasting feature on the HPE Juniper router should have been turned off because it was communicating with 20-year-old Cisco routers. “That capability should not have been turned on for a large legacy system,” said the source.
Duffy had earlier said that the problem with the HPE Juniper router occurred as part of a switchover from older Cisco equipment in El Dorado, Arkansas.
“What happened was as we did one of those switchovers in El Dorado, Arkansas, that server or that router started to flood our network,” said Duffy in an earlier press conference on the network upgrade from old copper wires to fibre.
In a press conference on Saturday, FAA Administrator Bryan Bedford told reporters that the FAA spends a billion dollars a year maintaining and sustaining its 1980s TDMI telco network, which he referred to as a 1G system. “We are constantly replacing 2,000 Cisco 600 routers with Juniper routers,” he said. “So, this is ordinary course-of-business stuff.”
Those Cisco 600 routers were manufactured between 2001 and 2006.
While the FAA’s infrastructure “badly” needs modernisation, that process has to be managed with extraordinary care, especially given the critical nature of the environment, according to one solution provider executive with federal government experience who asked not to be named.
“They were doing this [upgrade] in the middle of the day, on a Friday. Somebody should be asking some questions about that. Forget the technology under the hood. Who authorized going out and replacing this equipment in the middle of the day on a Friday? That’s a basic question to me,” the executive told CRN.
According to the FAA, maintenance at major air traffic control facilities is typically conducted at night when there’s less traffic. This particular disruption, however, was a result of a routine replacement at a remote facility.
“Based on what happened Friday, we will reevaluate when we do replacement/maintenance work,” the FAA said in a Q&A on the nationwide ground stop.
The router upgrade from Cisco routers to HPE Juniper equipment was being conducted by technology solution provider and federal contractor L3Harris Technologies, which under contract with the U.S. government runs the telecom network.
L3Harris Technologies, which has been working with the FAA for over 20 years, is now under scrutiny with regard to whether there was a misconfiguration issue regarding the HPE Juniper router as part of the network upgrade.
L3Harris Technologies referred CRN to its original statement issued the day after the ground stop.
In the statement, the company said: “L3Harris has been modernising the communications infrastructure supporting the National Airspace System in close coordination with the FAA. With safety and reliability as our highest priorities, L3Harris, together with its telco partners and the FAA, has completed more than 70 percent of the planned airspace communications conversion from legacy copper wires to a fiber optic system. We are supporting the FAA in its investigation to understand the conditions that led to the Nov. 13 ground stop of our nation’s air travel. L3Harris takes the nationwide ground stop and its impact on travellers, airlines and air traffic operations seriously.”
Duffy has said the FAA is conducting an investigation into the HPE Juniper routers at L3Harris Technologies Melbourne Florida facility. Duffy asked FBI Director Kash Patel to send agents to the Florida L3Harris Technologies lab to also participate in the review of the server to ensure it was not a cyberattack.
“So, everybody is congregated in Florida, and we are going to do an analysis on the server and the methods that were used on its installation,” said Duffy.
This article originally appeared on CRN UK sister websiteCRN US.