Historical status reports for 1998

12:30	The link appears to be ok now, although I still have not heard from telstra.

10:50	All international links through Telstra are horribly slow and have
3424226	massive packet loss. Telstra have been advised but are (were) unaware
	of the problem. (Has been happening since about 08:45)

10:50	Both Sydney and Melbourne international links through Telstra are
4323993	experiencing connectivity problems. Telstra have been advised.19/Dec/98

11:00	We are upgrading our Corowa, Culcairn and Yackandandah modem pools.
	We don't expect there to be any interruption to service, although users
	calling from out of town should see faster connects after the upgrade
	is completed later today.

16:10	The fault at MtBeauty has been found and fixed.
	It seems the guys from Complex Sound were doing some work at that
	site for another company and a cable that carries half the modem
	lines to our server became disconnected.

11:59	Half our lines in MtBeauty are off air.

22:18	The problem with the mail pop server has been identified and	
	corrected. inetd had stopped thus all services run out of inetd
	were down on that server, including ftp and pop. 
	Investigations are continuing as to the cause of this failure.
	(It happened once before, 22 months ago, but left no indication
	then, hopefully the additional logging since implemented will help)

21:58	POP mail server has become inaccessable.

09:10	Proxy server upgrade completed on time and without incident.

09:00	Scheduled upgrade of our main proxy server to add a further 128Mb RAM
	Shutdown expected to be 10-15 minutes duration.

13:45	We are adding incomming bandwidth to our links. This is being done
	in such a way that we do not expect there to be any interruption to
	service, although it is possible it will require a reload of our core
	router which would cause a 2-3 minute interruption of all incomming
	web, ftp and e-mail. This is an informational posting only.

12:45	World connectivity has been restored.

10:55	Sydney Padington exchange apparantly has lost one of their high speed
3422097	links. This is causing significant problems with communications to the
	USA. Telstra are working on the problem, as it is affecting most of
	Australia connected via Telstra. They have been unable to advise any
	estimated repair time.

16:11	Widespread Power fail due apparantly to trees down over power
	lines in Lavington/North Albury, combined with a UPS fail at
	our backup site has us operating our backup site on Generator.
	Our primary site still has utility power although there was a
	21 second outage which the main UPS there handled without a hitch.

09:30	Engineer/Admin has restored service to all Corryong, Tumbarumba and
	surrounding areas. The problem source has been identified and is not
	expected to cause any further problems in the future. Additional
	monitoring is being installed on all our remote POPs to give early
	warning of this failure mode so it can be prevented in future.

17:40	Despite getting someone on-site, the problem is more complex than
	initially anticipated. An engineer will be on-site tomorrow morning.

17:07	Our Corryong server has become uncontactable.
	Technicians have been despatched, we have no understanding of
	the failure mode at this stage. All systems were nominal before
	the server stopped responding.

15:38	Advice from Telstra about a serious network fault as follows.
        There are a number of 565 Mbps Services Down in the Sydney
        Area. Restoration is at this time unknown.
        The problem is larger than first estimated and is believed to be 
        caused by construction in the Sydney area. A quantity of 565Mb
        and 140Mb services have been effected. Technicians have been
        working non stop through the night and it is expected that most
        services should be restored by 10:00am AEST.

12:03	The nameserver has been repaired and is back on-line.
	An upgrade of this machine has been scheduled and should
	be swapped into service within 24 hours.

09:51	The disk on the nameserver has started to give errors again.
	This has not caused inconvenience to any of our subscribers as
	the secondary DNS is a backup service.

05:30	Temporary repairs effected, name server etc back on-line.

01:30	Our Secondary Nameserver has suffered a serious hard disk failure.
	This has not caused inconvenience to any of our subscribers as
	the secondary DNS is a backup service.

12:40	The Telstra router at the exchange has failed yet again.
3415844	We've reported the fault to telstra engineers, and are
	still waiting to hear back.

17:00	The attack has finally been repelled.
	We have taken steps and measures to prevent any similar 
	attacks in the future, and reported the attack to the
	relevant international authorities for action.
	Telstra have been singularly USELESS in this matter,
	until the attack has been sustained for 24 hours, they
	are not prepared to assist. Sheesh!

16:21	We are under attack from an outside source. 
3415438	The origin is not yet identified, because their packets
	appear to have forged IP addresses.

	We are adding another 30 modems to our Digital (56K) modem pool.
	Whilst the work being conducted should not affect the existing
	modems, it is possible there will be a brief interruption to the
	digital modem pool sometime Wednesday or Thursday. Please bear
	with us through this upgrade.

09:30	Thanks Telstra. Due to a various problems, the upgrade did NOT
	go smoothly, and telstra were unable to get our services back
	on-line until 9:30am.

07:00	We have a scheduled outage, expected to be up to 1 hour, for all
	in-dial subscribers calling our facilities North and West of
	Albury, extending as far as Holbrook and Yarrawonga.
	This is necessary while we commission a further 36 modems for our
	out-of-town facilities.
	The outage will NOT affect anyone directly calling our Albury,
	Wangaratta, Killawarra, Milawa, MtBeauty, Corryong or Jingellic

11:08	The main core router in Melbourne had failed, taking with it
	virtually all of Victoria and Tasmania. Fault has been rectified.

10:24	Telstra main core router link to Lonsdale has failed.
3412996	Telstra are unable to contact the router and I believe have
	despatched an engineer to attend to the problem. No restoration
	time has been advised.

15:10	The fault has been traced to a failed ethernet hub and rectified.

14:45	Our Camera and Weather servers are unreachable.

14:01	Server has been re-started, we're investigating the cause.

13:54	Our main Albury terminal server crashed.

17:00	Routers have been re-loaded and the problem seems to have gone away.
	Telstra have marked this as a "long term" problem and will be keeping
	an eye on the routing issues as they are intermittent and difficult
	to isolate and/or resolve.

12:45	Some weird routing problems to the USA are causing slow response
3412075	from some sites. Seems to be tied up with the pad16 router in Sydney.

11:45	Service restored.
	An official complaint is going to Telstra. The fault was caused
	by a telstra technician performing a hot-swap of a network component
	at the Lonsdale exchange, which resulted in a significant number of
	services state-wide being disrupted.

11:20	Link problem. Appears to be the Lonsdale exchange router.

11:50	Service restored. Problem has been escalated yet again.

11:30	Telstra router failure yet again. Engineers are working on the problem
	Resolution is expected within 20 minutes.

18:10	Telstra News server is back. No report yet as to what happened.

18:00	Telstra News server seems to be down. I've reported it and am awaiting
3410120	a response from Telstra engineers.

08:40	Wangaratta ISDN link restored. Telstra are still investigating the cause.
	No users were affected during the failure thanks to a backup link.
01:07	Wangaratta ISDN link down. Reported to Telstra

14:35	Router re-loaded and re-started. Operational again.
	We await information on a permanent cure for the fault.

14:00	Telstra router failed yet again. Cisco are now working on the fault
	in conjunction with Telstra to actually identify and resolve the
	on-going problems. Restoration expected in about 30-40 mins.

04:50	Corryong server back on-line.

20:35	Corryong server is running, but the router seems to have locked up.
	No-one has access to the strong room it's located in tonight, so
	we aim to restore service ASAP - assuming we can regain control of
	the router. It's possible storm damage has taken it out of service
	in which case we'll affect replacement on Monday/Tuesday.

10:05	Link restored. I'm sending an official complaint to TIS about this
	router and it's performance (or lack thereof!).

09:40	Telstra router again failed. Faults centre not accepting calls to
	be able to even report the fault. I've called the TIS engineering
	group directly and the fault is being worked on now.