Historical status reports for 2011
20:31 Server power restored. All services back on-line.
19:41 Our primary sydney server has gone unresponsive. Engineer has been
called and is on the way to investigate. Mail, many users websites
and authentication affected. ETR unknown until problem identified.
17:35 A technician working on an adjacent machine in our rack appears to
have caught a cable as he was leaving at 16:27, taking our primary
authentication and mail server down. Friday afternoon, Sydney CBD,
takes time to get someone back, but all restored 17:35.
08:21 All adsl and dialup services dropped. 20% immediately reconnected
but the remaining services were not restored until 8:50. The cause
was Ericsson/Redback engineers performing upgrades to the main LNS
made an error resulting in no authentications being possible.
(They were not supposed to be working on it at this time anyway)
Investigations and preventive measures are underway.
17:30 We are expecting a 5-minute outage at 17:30 today for engineers to
install new core routing infrastructure at the datacentre.
10:17 There has been a major power problem at the GlobalSwitch data centre
resulting in thousands of servers, switches and routers losing power.
Partial restoration was achieved in 20 minutes, but it took until
11:07 for all our power to be restored and all services back online.
18:45 Between 18:45 and 18:47 our primary mail and authentication server had
a hard drive failure. Our primary onsite engineer left yesterday for
urgent work in Hong Kong, our secondary engineer was in Albury to see
me, it took 90 minutes for the next-in-line to get to the datacentre.
Services restored at 20:38. During the outage, no mail could be sent or
received, dialup users could not authenticate and various other systems
may have been slow or inoperable.
20:28 Shortly after 18:59:57 tonight our main authentication server locked up.
System admin was able to identify the problem remotely, but took until
20:18 to get an engineer to the data-centre to resolve the problem.
Dial-in users will have been unable to connect. All users will have been
unable collect email during this time.
04:00 About 80% of our adsl users lost connectivity between 04:00 and 04:23
this morning. Issue being investigated, but it appears to be un-notified
work being performed by telstra. (Issue affected several other ISPs but
at different times this morning)
19:56 A tech doing some routine maintenance work at the data centre in
Sydney inadvertently installed a faulty cable between two core routers.
It failed in a manner that was undetected by the automatic backup link
and required an engineer to attend the site. Services restored 21:20
(Remedial work on the failover to prevent a repeat being done 23-Feb)