Historical status reports for 2014


12-Dec-2014
14:30	 Primary authentication server has become non-responsive.
	 System restarted, back online 14:42. Investigations under way.
	 Mail, authentication and some websites will have been affected.

28-Oct-2014
14:50	 Intermittently since 01:50 this morning we've suffered sporadic but brief
	 connectivity outages and brief periods of packet loss, attributed to a
	 DDoS attack against another network in the same datacentre. At 14:50 this
	 afternoon, our upstream corrupted a router config while attempting to
	 mitigate the DDoS, resulting in loss of connectivity to our racks. This
	 was rectified at 15:20, but the replacement device failed within 15 mins
	 and a replacement replacement was unable to be sourced for almost an hour.
	 Services restored 16:20. During this time, websites, mail and DNS were
	 all unavailable. Customers whos DSL services were connected before the
	 outage retained external connectivity. Steps are being taken to reduce
	 our exposure to a similar incident in future, including relocating some
	 of our equipment to another datacentre, and replicating the servers at
	 a third, interstate, datacentre.

07-Sep-2014
21:02    Loss of ADSL services. Appears to be a VPN/Router failure, engineers are
	        on their way to investigate.
         
         21:47. Services restored. Waiting on complete report, initial indication
         is that it was a BGP failure.
         
28-Aug-2014
03:00	 Work from 26-Aug-2014 rescheduled to this timeslot.

26-Aug-2014
06:45	 Between 06:45 and 07:00 we will be changing our core router.
	 We anticipate an outage of under 10 minutes during the changeover.
	 All services will be affected.  (Work rescheduled for 3am, 28-Aug-2014)

17-Aug-2014
06:50	 All appears to have completed smoothly. Downtime 100 seconds.
	 Testing indicates everything operating properly.

17-Aug-2014
06:45	 Between 06:45 and 07:45 we will be changing some critical routing
	 in our core routers. We expect up to 30 minutes during which all
	 our servers will be unreachable. (This period could be as short
	 as 30 seconds, but we're warning of a 30-minute outage for safety)

27-Feb-2014
16:30	 Two fiber optic cables have been cut, affecting all our outbound
	 domestic and international connectivity. Engineers have been sent
	 to investigate and route around the issue. More as it comes to hand.

11-Feb-2014
18:10	 Major Distributed Denial-of-Service attack is under way. Many of
	 our servers, all of our inbound and outbound links affected.
	 Seeing traffic in excess of 2 gigabytes/minute directed at some
	 servers. Working with our upstream providers to mitigate the
	 attack.
	 Services restored to normal around 20:20 although the attack is
	 still underway. Three teams are constantly monitoring as the
	 vector changes, and responding. Mitigation and improvement of
	 automatic mitigation techniques is continuing.
	 Services affected: basically everything, due to extremely high
	 latency on all interfaces and over all links.

01-Feb-2014
01:27	 Total loss of connectivity.
	 All links are up, all servers are running. Upstream has lost BGP
	 routing and no routes are being announced or accepted. They are
	 investigating the failure mode. Services restored 4:45 but further
	 work being conducted.

17-Jan-2014
21:00	 Loss of connectivity to many (but not all) national and international
	 destinations. Was upstream of our router. Call logged with carrier,
	 but fault restored about 21:08. Waiting on information.