Telco admits software update and documentation failures caused national disruption
Telstra admitted in a submission to the Senate inquiry hours before executives were due to be grilled that the mass network outage could have been prevented if a software update had been applied to the triggering device and that its internal controls were “not good enough”.
The telecommunications company’s submission, made public ahead of Friday’s midday hearing, gives the most detailed account yet of the failure, which left hundreds of people leaving Triple Zero, trains being halted and Eftpos terminals collapsing nationally.
Telstra said a technician arrived at the Exhibition Street switchboard in Melbourne late on July 7 to replace a chassis housing a network timing server due to a faulty backup power supply. When power was restored at 3.38am the next morning, the GPS card inside the device “did not work as expected” and the server reset the date to 2006. The incorrect date then spread across the network, invalidating authentication certificates and knocking customers offline.
The company blames it on the fact that an earlier design change to the equipment was never properly documented, leaving the maintenance team unaware of how the device would behave when rebooted. This change was made to fix a previous bug.
“No software update was applied to the device,” the application states. “If the software update had been completed or the design change had been properly documented and reflected in the maintenance procedure, the outage might not have occurred.”
Telstra adds: “This is completely unacceptable. If maintenance work is triggering this type of outage it means our controls are not good enough.”
This imprint first revealed that the outage was caused by the device’s “time travel” to 2006. Telstra’s presentation does not disclose the model of server in question or whether it needs to be replaced.
It comes as executives including chief executive Vicki Brady, finance chief Michael Ackland and Triple Zero executive Jane Elkington were among senators at lunchtime.
David Swan is the technology editor of The Age and The Sydney Morning Herald. He was previously technology editor of The Australian newspaper.Connect with: X or email.
