Historical status reports for 2010
12:24 Our sydney servers are isolated from the rest of the network, resulting
in no DNS, auth or mail. A failed cisco 2950 switch has been identified
as the culprit, a tech is onsite replacing with a gigether switch now.
Some services back (13:40) everything else should be back shortly.
00:05 There will be an estimated 5 minute outage between 00:05 and 00:10 while
a core routing device is reloaded as part of a firmware upgrade.
The only affected customers will be ADSL2 and a few ADSL1 services.
00:06 Vendor applied patch to LNS firmware resulted in an exception and
reboot of the LNS, resulting in all tunnels to be cleared. Approx
8 minutes before authentication restarted. Affected most ADSL and
12:00 During the period 12:00 to 16:00 we will be testing additional backup
links, BGP routing functionality and general failover link performance.
We do not anticipate significant disruption during this time, however
there may be a few short (1 to 2 minute) periods of reduced connectivity
while links settle. (This work would normally be conducted around 2am
but for operational reasons these tests must be done in business hours.)
11:15 There is an auth issue - seems to be australia-wide affecting most if
not all optus wholesale users, not just ALI customers. We have confirmed
3G, dialup and some ADSL2 services australia-wide are down. Optus are
working on the problem.
19:00 Our wholesaler has a hardware failure on their radius authentication
system, as a result we are not seeing users auth requests. They have
technicians on the way to the data centre with replacement hardware.
Updates as they come to hand.
14:30 More tomfoolery at the wholesalers. The national dialin number is broken
yet again. No reason given, continuous stream of promises of it being
fixed in 20 minutes - still down 18 hours later with no ETR.