Historical status reports for 2011

20:31	 Server power restored. All services back on-line.
19:41	 Our primary sydney server has gone unresponsive. Engineer has been
	 called and is on the way to investigate. Mail, many users websites
	 and authentication affected. ETR unknown until problem identified.
17:35	 A technician working on an adjacent machine in our rack appears to
	 have caught a cable as he was leaving at 16:27, taking our primary
	 authentication and mail server down. Friday afternoon, Sydney CBD,
	 takes time to get someone back, but all restored 17:35.

08:21	 All adsl and dialup services dropped. 20% immediately reconnected
	 but the remaining services were not restored until 8:50. The cause
	 was Ericsson/Redback engineers performing upgrades to the main LNS
	 made an error resulting in no authentications being possible.
	 (They were not supposed to be working on it at this time anyway)
	 Investigations and preventive measures are underway.

17:30    We are expecting a 5-minute outage at 17:30 today for engineers to
	 install new core routing infrastructure at the datacentre.
10:17	 There has been a major power problem at the GlobalSwitch data centre
	 resulting in thousands of servers, switches and routers losing power.
	 Partial restoration was achieved in 20 minutes, but it took until
	 11:07 for all our power to be restored and all services back online.

18:45    Between 18:45 and 18:47 our primary mail and authentication server had
	 a hard drive failure. Our primary onsite engineer left yesterday for
         urgent work in Hong Kong, our secondary engineer was in Albury to see
         me, it took 90 minutes for the next-in-line to get to the datacentre.
         Services restored at 20:38. During the outage, no mail could be sent or
         received, dialup users could not authenticate and various other systems
         may have been slow or inoperable.
20:28    Shortly after 18:59:57 tonight our main authentication server locked up.
	        System admin was able to identify the problem remotely, but took until
         20:18 to get an engineer to the data-centre to resolve the problem.
         Dial-in users will have been unable to connect. All users will have been
         unable collect email during this time.
04:00   About 80% of our adsl users lost connectivity between 04:00 and 04:23 
	this morning. Issue being investigated, but it appears to be un-notified
	work being performed by telstra. (Issue affected several other ISPs but
        at different times this morning)
19:56   A tech doing some routine maintenance work at the data centre in
	Sydney inadvertently installed a faulty cable between two core routers.
	It failed in a manner that was undetected by the automatic backup link
	and required an engineer to attend the site. Services restored 21:20
	(Remedial work on the failover to prevent a repeat being done 23-Feb)