Historical status reports for 2008
18:10 An unknown issue with connectivity between our wholesale provider and
upstream has left us completely isolated from the network. Techs have
been recalled to the datacentre to investigate and remedy. Resolution
times unknown until the cause is identified.
Finally all services restored 22:33, still waiting on full details.
All services and all customers affected.
03:00 We have been notified by our wholesaler that there is to be a 5-10
minute interruption due to a software defect in one of their core
routers which requires reloading of the firmware and rebooting.
18:00 We attempted to relocate one of our servers (ntp and accounts), but have
encountered problems with a misbehaving router at the new site. Accounts
and time are currently down. If we can't get the router working tonight
we will return the router to our albury datacentre and review the new
site before trying again.
00:01 We have been advised of an emergency "planned outage" at our Sydney
datacentre. We expect an interruption of up to 20 minutes within the
window of midnight to 6am. This will affect all customers, and all
services. ADSL, Dialup, hosting, DNS, email etc because the whole
section of the datacentre will be isolated during this work.
14:11 We are upgrading our online billing server this afternoon and expect
approx 1 hour interruption to users trying to access their bills
on-line, change their plans, or use customer support.
(Works completed, server back on-line)
16:20 Approx 16:20 there was a widespread loss of connectivity to most (if not
all) of our Victorian users. Ticket has been lodged with upstream.
Update: 16:55, services are comming back now, no update yet as to cause.
17:00 Unknown cause: loss of connectivity beyond Sydney. Engineers are
working on it, no cause or ETA yet.
Fault cleared 17:25 but no answer yet as to cause.
02:00 Scheduled maintenance for router and code infrastructure upgrade
at the datacentre. Outage expected to be for 10 minutes.
(Event unfortunately took 30 minutes)
(two subsequent (also planned) interruptions to complete the works
which happened at 4:35 and 5:40)
17:14 We have lost connectivity to all ADSL, Dialup and fixed services. Fault
logged but cause (and restoration times) unknown.
Update: 17:20 - cause found, restoration expected 20-30 minutes
Update: 17:46 - all services restored.
Problem identified as a failed optical fibre patch within the carriers
network, explaining why we had lost all connectivity.
00:40 Widespread powerfail at the DataCentre resulted in temporary loss of all
services. Unfortunately, due to a misconfiguration, a critical process
failed to restart resulting in no ADSL services comming back online
until engineers were able to attend. A few transitional services (like
lightning tracking, weather monitoring) were not back until about 7am
10:02 At 9:23 our main office link failed. Customer links were not affected.
On-line accounts enquiries, weather data and webcams were offline for
between 20 and 30 minutes. We are waiting on explanation from upstream.
10:45 Intermittently, some ADSL users connections are dropping out and then
immediately re-connecting. After substantial investigative work, it has
been identified as a flakey OMC. Telstra are being organised to do an
urgent swapover. Most users won't even notice the issue as it happens
16:35 There is an unknown issue in Sydney, affecting our national dialup
number. Calls are not being authenticated. Technicians are working on
the problem and expect resolution soon.
(Problem rectified 17:49)
08:30 A widespread, multi-carrier fault is disrupting adsl and some mobile
services in the Tawonga/Mt Beauty area. Multiple carriers are affected
not just ALI. The cause is currently unknown, as is restoration time.
06:30 Mail, authentication and most web services interrupted for 2 hrs due
to unforseen difficulties with work on our Sydney server.