Last week a number of improvements were made that our carrier downgraded the fault to a resolved status, however the Sydney call collection area issue had not been resolved yet.
At this current time the upstream has applied a work around for the Sydney call collection areas to alleviate congestion and today we can confirm during peak period only a small handful of calls experienced congestion when dialing outbound, the upstream is still working with interconnect partners to bring on new links. As soon as those links come online the workarounds will be removed.
For now we will close this issue as resolved, if you have any further congestion issues please treat it as a separate issue and contact our support at support@firenet.com.au or call via 02 9397 7050
Posted Apr 16, 2020 - 12:49 AEST
Update
Dear Customers,
Things have improved greatly however intermittent congestion is still being experienced on outbound and inbound calls from certain regions/carriers.
Today we saw ~80 outbound calls receive a congestion code from the upstream, this will slowly decrease as the upstream adds greater capacity to specific carriers.
The core issue is still primarily Telstra call collection areas in Sydney and Melbourne regions. Our upstream is updating us daily on the progress of the new E1 interconnection expansion and we have been advised in addition to preliminary Sydney works this week the Melbourne CCA added 2511 additional channels.
At this time no ETR can be given.
Posted Apr 03, 2020 - 16:29 AEDT
Update
Dear Customers,
Today we had a total of 341 calls receive a congestion error code from the upstream when attempting to dial outbound.
This is COVID-19 related and we currently have been given no ETR.
The upstream carrier is continually working on solutions to solve their carrier interconnect issue and has expedited as many connections as they can.
At the moment a VAST majority of the congestion issues are due to the upstreams interconnection with Telstra. As the new links required use older technology the method to deploy these links in an extremely fast fashion is not possible.
Please note this is being experienced across every tier 1 carrier in Australia and we expect this to continue intermittently for at least this week.
Posted Mar 30, 2020 - 17:45 AEDT
Monitoring
Dear Customers,
Yesterday our upstream confirmed inbound traffic increased by 80% and is forecasted to double before COB Friday, the capacity increases and shifting of traffic is working and we have seen a consistent reduction in congestion codes from the carrier.
Today so far we have less than 1% of outbound calls congested, we will monitor this throughout the day but its likely safe to say the majority of the outbound issue has now been resolved and will continue to get better.
Inbound congestion from specific carriers and call collection areas we are not able to monitor as clearly as we can with outbound, for this reason and due to the fact the upstream is still working on interconnection links; we will keep this incident open until we receive full confirmation of the resolution from the upstream.
Thanks again for your patience during this period.
Posted Mar 26, 2020 - 11:19 AEDT
Update
Yesterday our upstream confirmed a further 20% increase in traffic and is expected to grow throughout the week. The unprecedented demand and stress placed on their systems is causing congestion for inbound and outbound calls via certain call collection areas. This is being experienced industry wide across multiple tier 1 providers nationally. This is attributed to the COVID-19 situation.
Works continue today to expedite carrier interconnection links, in addition to the already completed capacity increases the carrier has finished pre patching for severely affected call collection areas and exchanges in Sydney and Melbourne, these works will continue throughout the week.
Today so far we have seen a large drop in congestion during the heavy periods (under 3% of calls) which is a direct result of the speedy upgrades our upstream has applied.
Unfortunately, we are unable to provide an ETR at this stage however we can see massive improvements and expect further more to come this week.
Posted Mar 24, 2020 - 12:39 AEDT
Update
Dear Customers,
Since our last update the upstream carrier has been working extensively and has solved the following issues on their call collection areas (CCA's) regional and national during the heavy congestion periods (9am-1:30pm):
Vodafone CCA = Congestion resolved Optus CCA = Congestion resolved TPG/AAPT CCA = Congestion resolved Vocus CCA = Congestion resolved Other bi-lateral Interconnect Partners: Congestion resolved Telsra Other CCA = Congestion resolved
Telstra Sydney and Melbourne CCA = Congestion experienced
This means for inbound and outbound calls via the Telstra Sydney and Melbourne region users may experience call congestion during the heavy period.
We have altered this fault to 'Intermittent congestion' as this is not only affecting outbound calls.
Currently our upstream is focusing on bringing forward a number of interconnections between them and Telstra to alleviate these issues. The Telstra Melbourne CCA has work scheduled for 24/03/2020 which may solve the congestion period and their Sydney CCA work has also been brought forward as a matter of urgency however no ETR has been posted for this.
At this stage the upstream does not have an ETR.
Posted Mar 23, 2020 - 11:47 AEDT
Update
Here is an update on the ongoing major incident involving congestion issues being experienced on the carrier interconnect (CI) links between carriers.
After analysing the peak-time traffic patterns from today, our upstream carrier can confirm the congestion being experienced in Newcastle and Hobart has been relieved.
While not yet resolved, we have seen steady improvements as the restoration has progressed and can now see a 50% reduction in congestion issues.
The carrier team are continuing to work with our carrier partners to alleviate the congestion in other major cities, with our main focuses being Sydney and Melbourne. As mentioned in our previous update, given the magnitude of these network changes, and the number of parties involved, these changes will take time to complete.
FireNet apologises for the lengthy delays but are working hard with our industry carrier partners to drive resolution.
Posted Mar 19, 2020 - 16:31 AEDT
Update
Yesterday our upstream carrier in conjunction with other interconnect carriers identified the Coronavirus Hotline is the source of generating extreme traffic volumes in multiple call collection areas (CCA) resulting in the congestion, they are working with the end user of this service and intermediary carriers to have traffic redistributed to CCA's with spare capacity.
Whilst this is occurring other measures are taking place namely expanding the carrier interconnect capacity between networks. Our upstream has a expedited change request involving physical work within Telstra exchanges to increase capacity on carrier interconnect links, unfortunately the tentative delivery date for this is the end of next week. In addition to this our upstream alleviated congestion in Newcastle and Hobart by rerouting traffic to alternate paths and as a result call congestion has decreased in these areas.
This incident has the full attention of FireNet and the upstreams entire team, We thank for you patience while this issue is being resolved.
Posted Mar 19, 2020 - 09:26 AEDT
Identified
Intermittent call congestion issues have continued to occur for a second day during Morning to mid-afternoon, this has been attributed to the COVID-19 pandemic being experienced around the globe.
Unprecedented demand of calls have occurred due to many factors relating to the pandemic, this load is impacting our upstream carriers interconnection links to external carriers such as Vodafone, Telstra and Optus and unfortunately these interconnection links have not been able to sustain the vast increase of calls from an entire workforce shifting.
As of last night our upstream expedited and brought into production a number of new carrier interconnection links and has resolved a number of issues however some interconnection links still require capacity increases and some aren't as fast to deploy (such as E1).
These higher capacity links are being worked on as we speak and will be brought into production as soon as possible to alleviate congestion issues.
With that being said we expect during this week between 08:00 AEDT to 13:30 AEDT intermittent call congestion to occur.
FireNet apologizes for the late response on the congestion issues experienced today.
Posted Mar 17, 2020 - 17:22 AEDT
This incident affected: Voice Services (Voice trunking, Hosted voice).