Historical status reports for 2012
18:22 Localised routing issue. Upstream BGP flapping, causing loss
of some routes to certain IP ranges. Affected sites vary.
Upstream BGP server rebooted, problem cleared 19:04.
08:15 Outage identified, engineers rebuilding affected networks.
Customers comming back on-line from 08:15.
06:00 Widespread major telstra outage affecting all services in and
around Albury. They are working on it but no ETR yet.
14:34 Main mail and authentication server became non-responsive
at 14:34. System restarted at 14:47 and back online 14:50
Rack temperatures are excessive, talking with datacentre.
17:03 Last of the links have been restored. BGP issues have
been affecting users all over the country from lots of
15:45 Loss of connectivity again. Affecting different users
in different ways.
11:25 Links returned. Seems to be BGP issue, possibly as a
result of connectivity issues to charter in the USA.
11:10 Intermittent loss of connectivity to multiple networks,
22:04 Our main mail and authentication server became non-responsive
at 22:04. System administrator restarted the box at 22:26.
Investigating the cause.
06:42 New system built, tested, all data migrated from old system.
Old system mail system stopped, all mail moved to new system.
Old system taken out of service, new system brought online.
System back up 06:49am. 8 minutes downtime sunday morning.
16:00 Estimated time only. Hoping to have hardware ready to replace
failing drive subsystem. Hoping for less than 1hr downtime.
Will advise more when details available/confirmed.
18:00 Authentication and mail server stopped responding again.
Cause is a failing drive array. Replacement parts being prepared.
06:18 Our authentication and mail server stopped responding.
Remote restart has the system back online. Cause being investigated.
22:00 We're moving the last 3 servers from the old rack at GS, to the
new racks from 10pm, 23-Jan-2012. Downtime is not expected to be
more than an hour for each server, and should be considerably
less. During this time, various functions will be unavailable.
Authentication, mail, websites, DNS etc. Most operations should
failover to other servers (like DNS) or only be down for fairly
short times (hopefully less than 30 minutes).
syd3: down 21:38 up 22:25 (47 mins)
syd2: down 22:29 up 22:52 (23 mins)
syd1: down 23:22 up 23:39 (17 mins)