Telstra staff unaware of mass outage risk as critical software failure ‘rippled slowly across the network’ | Telstra

Telstra has blamed the lack of a software update for a key timekeeping system for the outage that caused chaos across the country last week; maintenance crews were also unaware of a design change that affected how the system was reset.

Telstra made a written submission explaining its reasoning before appearing before chief executive Vicki Brady at the Senate inquiry into the mobile phone outage on Friday. The company stated that there is no redundancy in its network, but this redundancy did not prevent the outage.

The notification confirmed a report that one of Telstra’s network time protocol (NTP) servers, designed to ensure systems receive the correct time, had been reset to “2006”.

Telstra said it has three NTP servers in Melbourne, Sydney and Perth.

During maintenance, the company shut down and restarted the server in Melbourne. It rebooted with the wrong date 2006 due to “underlying software configuration” on the server.

“Over the next few hours, the incorrect date slowly rippled through the network, causing authentication certificates on other servers to become invalid,” Telstra said.

“Customers were occasionally unable to authenticate to the network (“no service”), impacting their ability to make voice calls and use data over Telstra’s mobile network.”

Telstra said it made a deliberate design change to the equipment to fix an earlier fault but this was not properly documented. This meant maintenance workers who arrived at the Melbourne facility in the early hours of last Wednesday were unaware of how to reset the device.

Telstra said a software update was also not applied to the device and that if this had been done the outage might not have occurred.

The presentation also stated that when the Melbourne NTP server was disconnected for maintenance, the other two operated as backup and backup as expected.

“The failure mode here was not inherently related to the hardware, redundancy levels, or architecture of the system. our network,” Telstra said.

But when the Melbourne server turned back on and provided an incorrect date, “downstream systems used that date in security, authentication, session and policy control processes.”

“So the problem was not just the loss of one NTP server or redundancy in the configuration design of three NTP servers, but also the dissemination and acceptance of erroneous date information by interconnected systems that rely on timing as a trust and order reference.”

Telstra also said it took full responsibility for the outage.

“This is clearly unacceptable. If maintenance work triggers this type of outage, it means our controls are not good enough,” the company said.

“We are responsible for this and our investigation will address why the design change was not documented, why the software update was not completed and what needs to change in our controls to ensure known risks are caught, prioritized and closed before they impact customers.”

During the outage, Telstra said 58,835 calls were successfully connected to triple zero, while 604 encountered errors.

Telstra also operates the triple zero platform for all telcos and the company said the platform does not use NTP servers for synchronization, so it was not affected by the incident. Landline callers on NBN were also unaffected.

On Friday, Brady and several other executives will appear before a Senate inquiry into last year’s Optus outage to explain the outage that wreaked havoc on mobile services, transportation systems, retailers and electric vehicle charging.

Greens senator Sarah Hanson-Young, chair of the committee, said the inquiry would seek to establish the truth about the outage and “make sure we do not continue to accept a system that leaves Australians vulnerable”.