Historical status reports for 1998
23/Dec/98
12:30 The link appears to be ok now, although I still have not heard from telstra.
23/Dec/98
10:50 All international links through Telstra are horribly slow and have
3424226 massive packet loss. Telstra have been advised but are (were) unaware
of the problem. (Has been happening since about 08:45)
19/Dec/98
10:50 Both Sydney and Melbourne international links through Telstra are
4323993 experiencing connectivity problems. Telstra have been advised.19/Dec/98
18/Dec/98
11:00 We are upgrading our Corowa, Culcairn and Yackandandah modem pools.
We don't expect there to be any interruption to service, although users
calling from out of town should see faster connects after the upgrade
is completed later today.
17/Dec/98
16:10 The fault at MtBeauty has been found and fixed.
It seems the guys from Complex Sound were doing some work at that
site for another company and a cable that carries half the modem
lines to our server became disconnected.
17/Dec/98
11:59 Half our lines in MtBeauty are off air.
15/Dec/98
22:18 The problem with the mail pop server has been identified and
corrected. inetd had stopped thus all services run out of inetd
were down on that server, including ftp and pop.
Investigations are continuing as to the cause of this failure.
(It happened once before, 22 months ago, but left no indication
then, hopefully the additional logging since implemented will help)
15/Dec/98
21:58 POP mail server has become inaccessable.
09/Dec/98
09:10 Proxy server upgrade completed on time and without incident.
09/Dec/98
09:00 Scheduled upgrade of our main proxy server to add a further 128Mb RAM
Shutdown expected to be 10-15 minutes duration.
01/Dec/98
13:45 We are adding incomming bandwidth to our links. This is being done
in such a way that we do not expect there to be any interruption to
service, although it is possible it will require a reload of our core
router which would cause a 2-3 minute interruption of all incomming
web, ftp and e-mail. This is an informational posting only.
20/Nov/98
12:45 World connectivity has been restored.
20/Nov/98
10:55 Sydney Padington exchange apparantly has lost one of their high speed
3422097 links. This is causing significant problems with communications to the
USA. Telstra are working on the problem, as it is affecting most of
Australia connected via Telstra. They have been unable to advise any
estimated repair time.
07/Nov/98
16:11 Widespread Power fail due apparantly to trees down over power
lines in Lavington/North Albury, combined with a UPS fail at
our backup site has us operating our backup site on Generator.
Our primary site still has utility power although there was a
21 second outage which the main UPS there handled without a hitch.
04/Nov/98
09:30 Engineer/Admin has restored service to all Corryong, Tumbarumba and
surrounding areas. The problem source has been identified and is not
expected to cause any further problems in the future. Additional
monitoring is being installed on all our remote POPs to give early
warning of this failure mode so it can be prevented in future.
03/Nov/98
17:40 Despite getting someone on-site, the problem is more complex than
initially anticipated. An engineer will be on-site tomorrow morning.
03/Nov/98
17:07 Our Corryong server has become uncontactable.
Technicians have been despatched, we have no understanding of
the failure mode at this stage. All systems were nominal before
the server stopped responding.
20/Oct/98
15:38 Advice from Telstra about a serious network fault as follows.
There are a number of 565 Mbps Services Down in the Sydney
Area. Restoration is at this time unknown.
The problem is larger than first estimated and is believed to be
caused by construction in the Sydney area. A quantity of 565Mb
and 140Mb services have been effected. Technicians have been
working non stop through the night and it is expected that most
services should be restored by 10:00am AEST.
16/Oct/98
12:03 The nameserver has been repaired and is back on-line.
An upgrade of this machine has been scheduled and should
be swapped into service within 24 hours.
16/Oct/98
09:51 The disk on the nameserver has started to give errors again.
This has not caused inconvenience to any of our subscribers as
the secondary DNS is a backup service.
16/Oct/98
05:30 Temporary repairs effected, name server etc back on-line.
16/Oct/98
01:30 Our Secondary Nameserver has suffered a serious hard disk failure.
This has not caused inconvenience to any of our subscribers as
the secondary DNS is a backup service.
29/Jul/98
12:40 The Telstra router at the exchange has failed yet again.
3415844 We've reported the fault to telstra engineers, and are
still waiting to hear back.
20/Jul/98
17:00 The attack has finally been repelled.
We have taken steps and measures to prevent any similar
attacks in the future, and reported the attack to the
relevant international authorities for action.
Telstra have been singularly USELESS in this matter,
until the attack has been sustained for 24 hours, they
are not prepared to assist. Sheesh!
20/Jul/98
16:21 We are under attack from an outside source.
3415438 The origin is not yet identified, because their packets
appear to have forged IP addresses.
8/Jul/98
We are adding another 30 modems to our Digital (56K) modem pool.
Whilst the work being conducted should not affect the existing
modems, it is possible there will be a brief interruption to the
digital modem pool sometime Wednesday or Thursday. Please bear
with us through this upgrade.
25/Jun/98
09:30 Thanks Telstra. Due to a various problems, the upgrade did NOT
go smoothly, and telstra were unable to get our services back
on-line until 9:30am.
25/Jun/98
07:00 We have a scheduled outage, expected to be up to 1 hour, for all
in-dial subscribers calling our facilities North and West of
Albury, extending as far as Holbrook and Yarrawonga.
This is necessary while we commission a further 36 modems for our
out-of-town facilities.
The outage will NOT affect anyone directly calling our Albury,
Wangaratta, Killawarra, Milawa, MtBeauty, Corryong or Jingellic
facilities.
19/May/98
11:08 The main core router in Melbourne had failed, taking with it
virtually all of Victoria and Tasmania. Fault has been rectified.
19/May/98
10:24 Telstra main core router link to Lonsdale has failed.
3412996 Telstra are unable to contact the router and I believe have
despatched an engineer to attend to the problem. No restoration
time has been advised.
18/May/98
15:10 The fault has been traced to a failed ethernet hub and rectified.
18/May/98
14:45 Our Camera and Weather servers are unreachable.
11/May/98
14:01 Server has been re-started, we're investigating the cause.
11/May/98
13:54 Our main Albury terminal server crashed.
20/Apr/98
17:00 Routers have been re-loaded and the problem seems to have gone away.
Telstra have marked this as a "long term" problem and will be keeping
an eye on the routing issues as they are intermittent and difficult
to isolate and/or resolve.
20/Apr/98
12:45 Some weird routing problems to the USA are causing slow response
3412075 from some sites. Seems to be tied up with the pad16 router in Sydney.
31/Mar/98
11:45 Service restored.
An official complaint is going to Telstra. The fault was caused
by a telstra technician performing a hot-swap of a network component
at the Lonsdale exchange, which resulted in a significant number of
services state-wide being disrupted.
31/Mar/98
11:20 Link problem. Appears to be the Lonsdale exchange router.
3411463
17/Mar/98
11:50 Service restored. Problem has been escalated yet again.
17/Mar/98
11:30 Telstra router failure yet again. Engineers are working on the problem
Resolution is expected within 20 minutes.
18/Feb/98
18:10 Telstra News server is back. No report yet as to what happened.
18/Feb/98
18:00 Telstra News server seems to be down. I've reported it and am awaiting
3410120 a response from Telstra engineers.
12/Feb/98
08:40 Wangaratta ISDN link restored. Telstra are still investigating the cause.
No users were affected during the failure thanks to a backup link.
12/Feb/98
01:07 Wangaratta ISDN link down. Reported to Telstra
S9255908
12/Feb/98
14:35 Router re-loaded and re-started. Operational again.
We await information on a permanent cure for the fault.
12/Feb/98
14:00 Telstra router failed yet again. Cisco are now working on the fault
in conjunction with Telstra to actually identify and resolve the
on-going problems. Restoration expected in about 30-40 mins.
08/Feb/98
04:50 Corryong server back on-line.
07/Feb/98
20:35 Corryong server is running, but the router seems to have locked up.
No-one has access to the strong room it's located in tonight, so
we aim to restore service ASAP - assuming we can regain control of
the router. It's possible storm damage has taken it out of service
in which case we'll affect replacement on Monday/Tuesday.
07/Jan/98
10:05 Link restored. I'm sending an official complaint to TIS about this
router and it's performance (or lack thereof!).
07/Jan/98
09:40 Telstra router again failed. Faults centre not accepting calls to
be able to even report the fault. I've called the TIS engineering
group directly and the fault is being worked on now.
