Historical status reports for 2009


22-Dec-2009
13:02    Somewhere between 13:02 and 13:05 our primary server stopped responding.
         Engineering performed basic tests, all other servers were alive. The
         machine in question was reset and came back online at 13:22.
         Many services were affected during this time including news, mail some
         virtual domains, authentication. Engineering are reviewing logs to see
         if a cause can be identified.
         
13-Dec-2009
00:23	 Two powerfails at the datacentre have taken out both primary and
	 secondary routers (corrupted FLASH). Engineers are on site trying
	 to work a temporary fix. (Restored 12:30, although there are MTU
	 issues affecting some customers, particularly in Victoria. Fix
	 expected soon)

17-Nov-2009
11:30   Approx 11:30 today, M2 disabled a link in error which resulted in no
	auths from our dialup pool. After hours of yelling, someone took the
	issue seriously and restored the link (that they thought was not in
	use). Dialup services restored late afternoon.

03-Sep-2009
20:29    At 20:29:43 tonight we lost all adsl services for approx 35 seconds.
         The fault was caused by a contractor briefly disconnecting and then
         reconnecting a cable from the wrong core switch fabric at our Equinox
         data centre in Sydney.
         
04-Aug-2009
12:20    At approx 12:20 we lost most or all adsl services. While chasing optus
         faults people they all returned (approx 12:30). At this stage we do not
         know the cause or extent, will update when information comes to hand.
         
13-Mar-2009
9:33     Approx 9:33am we lost connectivity between core servers, LNS and all
         external links. While the guys were running new fibre at the datacentre
         someone has knocked out the power to the core switch. Restored within
         about 10 minutes.
         
22-Jan-2009
18:20    Following a fierce storm that hit the area about 16:20, our Norris Park
         office lost power and has been running on UPS. Without notice, it shut
         down at 18:20, taking the accounts server offline until a genset was run up.
         Machine recovery time meant accounts access was unavailable until approx
         18:49. No data was lost, however access to current and historical
         billing information was unavailable during this time, approx 30 mins.