Historical status reports for 2008

18:10    An unknown issue with connectivity between our wholesale provider and
         upstream has left us completely isolated from the network. Techs have
         been recalled to the datacentre to investigate and remedy. Resolution
         times unknown until the cause is identified.
         Finally all services restored 22:33, still waiting on full details.
         All services and all customers affected.
03:00    We have been notified by our wholesaler that there is to be a 5-10
         minute interruption due to a software defect in one of their core
         routers which requires reloading of the firmware and rebooting.
18:00    We attempted to relocate one of our servers (ntp and accounts), but have
         encountered problems with a misbehaving router at the new site. Accounts
         and time are currently down. If we can't get the router working tonight
         we will return the router to our albury datacentre and review the new
         site before trying again.
00:01	 We have been advised of an emergency "planned outage" at our Sydney
         datacentre. We expect an interruption of up to 20 minutes within the
         window of midnight to 6am. This will affect all customers, and all
         services. ADSL, Dialup, hosting, DNS, email etc because the whole
         section of the datacentre will be isolated during this work.

14:11    We are upgrading our online billing server this afternoon and expect
         approx 1 hour interruption to users trying to access their bills
         on-line, change their plans, or use customer support.
         (Works completed, server back on-line)
16:20    Approx 16:20 there was a widespread loss of connectivity to most (if not
         all) of our Victorian users. Ticket has been lodged with upstream.
         Update: 16:55, services are comming back now, no update yet as to cause.
17:00    Unknown cause: loss of connectivity beyond Sydney. Engineers are
	 working on it, no cause or ETA yet.
	 Fault cleared 17:25 but no answer yet as to cause.

02:00	 Scheduled maintenance for router and code infrastructure upgrade
	 at the datacentre. Outage expected to be for 10 minutes.
	 (Event unfortunately took 30 minutes)
	 (two subsequent (also planned) interruptions to complete the works
	 which happened at 4:35 and 5:40)

17:14    We have lost connectivity to all ADSL, Dialup and fixed services. Fault
	 logged but cause (and restoration times) unknown.
         Update: 17:20 - cause found, restoration expected 20-30 minutes
         Update: 17:46 - all services restored.
         Problem identified as a failed optical fibre patch within the carriers
         network, explaining why we had lost all connectivity.
00:40    Widespread powerfail at the DataCentre resulted in temporary loss of all
	 services. Unfortunately, due to a misconfiguration, a critical process
         failed to restart resulting in no ADSL services comming back online
         until engineers were able to attend. A few transitional services (like
         lightning tracking, weather monitoring) were not back until about 7am
10:02    At 9:23 our main office link failed. Customer links were not affected.
	        On-line accounts enquiries, weather data and webcams were offline for
         between 20 and 30 minutes. We are waiting on explanation from upstream.
10:45    Intermittently, some ADSL users connections are dropping out and then
	        immediately re-connecting. After substantial investigative work, it has
         been identified as a flakey OMC. Telstra are being organised to do an
         urgent swapover. Most users won't even notice the issue as it happens
         almost instantly.
16:35    There is an unknown issue in Sydney, affecting our national dialup
	        number. Calls are not being authenticated. Technicians are working on
         the problem and expect resolution soon.
         (Problem rectified 17:49)
08:30    A widespread, multi-carrier fault is disrupting adsl and some mobile
	        services in the Tawonga/Mt Beauty area. Multiple carriers are affected
         not just ALI. The cause is currently unknown, as is restoration time.
06:30    Mail, authentication and most web services interrupted for 2 hrs due
	        to unforseen difficulties with work on our Sydney server.