Historical status reports for 2009
13:02 Somewhere between 13:02 and 13:05 our primary server stopped responding.
Engineering performed basic tests, all other servers were alive. The
machine in question was reset and came back online at 13:22.
Many services were affected during this time including news, mail some
virtual domains, authentication. Engineering are reviewing logs to see
if a cause can be identified.
00:23 Two powerfails at the datacentre have taken out both primary and
secondary routers (corrupted FLASH). Engineers are on site trying
to work a temporary fix. (Restored 12:30, although there are MTU
issues affecting some customers, particularly in Victoria. Fix
11:30 Approx 11:30 today, M2 disabled a link in error which resulted in no
auths from our dialup pool. After hours of yelling, someone took the
issue seriously and restored the link (that they thought was not in
use). Dialup services restored late afternoon.
20:29 At 20:29:43 tonight we lost all adsl services for approx 35 seconds.
The fault was caused by a contractor briefly disconnecting and then
reconnecting a cable from the wrong core switch fabric at our Equinox
data centre in Sydney.
12:20 At approx 12:20 we lost most or all adsl services. While chasing optus
faults people they all returned (approx 12:30). At this stage we do not
know the cause or extent, will update when information comes to hand.
9:33 Approx 9:33am we lost connectivity between core servers, LNS and all
external links. While the guys were running new fibre at the datacentre
someone has knocked out the power to the core switch. Restored within
about 10 minutes.
18:20 Following a fierce storm that hit the area about 16:20, our Norris Park
office lost power and has been running on UPS. Without notice, it shut
down at 18:20, taking the accounts server offline until a genset was run up.
Machine recovery time meant accounts access was unavailable until approx
18:49. No data was lost, however access to current and historical
billing information was unavailable during this time, approx 30 mins.