Impact Summary At approximately 09:49 Monday 7th December 2020 AEDST, for up to 120 minutes, FireNet observed disruptions to a number of transit and peering connections within our New South Wales edge devices causing an internet outage for a majority of services.
Technical Brief FireNet engineers received alerts from internal systems for our edge devices notifying high cpu and memory usage. Upon investigation we noticed all BGP sessions in a hung state where they held current routes but were not updating. The devices were accepting commands but not applying. We were able to re-route traffic to other nodes and have now identified the issue that affected the nodes.
Moving Forward FireNet engineers have sent through logs of the affected devices to the vendor to identify the root cause. Once the logs have been analysed, we will work with the vendor to test and deploy changes to these nodes. FireNet will notify of any maintenance in due course.
Posted Dec 07, 2020 - 22:23 AEDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Dec 07, 2020 - 15:34 AEDT
Update
We are continuing to investigate this issue.
Posted Dec 07, 2020 - 11:42 AEDT
Investigating
We have identified that a number of IP transit and Internet services in NSW are experiencing degraded performance.
Our carrier is currently investigating the issue.
Further updates to follow.
Please contact the FireNet NOC should you have any questions, comments, or concerns. Email: support@firenet.com.au Phone one of the following numbers: Australia: 1300 636 636 International customers please call +61 2 9397 7050
Posted Dec 07, 2020 - 09:45 AEDT
This incident affected: Hosted Services (Co-location) and Network Services (IP Transit).