Historical status reports for 2001

17:08	 Merimbula Power restored finally !  Country Link Energy could
	 only tell me that someone had taken out a power pole in the
	 CBD which knocked out all Power in Merimbula. 
	 Tech also called on site to check servers.  Systems checked
	 remotely and all services back to normal.
16:00	 Merimbula Completely DOWN, Major Power fault in Merimbula
	 causing all of the CBD to be taken out.  UPS went as far as
	 it could, but had to give up sooner or later.
	 Waiting for Country Energy to call back with ETA for Power
	 to be restored.
09:05	 When the revolution comes, this damn annex will be the first
	 one up against the wall. Failing to respond, not pinging.
	 Tech called to site to fix. This one is earmarked for replacement
	 early in the new year.
14:30	 Annex had a slight hickup.  Called on-site to rectify.
	 Once again, systems fully operational
06:58	 Arghh! The Annex was responding, but not well. High pings, no
	 DTR asserted, a few users logged in but not doing much.
	 Tech attended site, graceful restart has cured things for now.
16:48	 Ping times from Perth to USA way to high 1000ms to 2000ms
13556666 Telstra called, it's on their network, so we will wait and
	 see how long it takes before someone can address it
11:03	 At long bloody last. I've determined that our dialup pools were
13546453 able to route ok ( but not our main server pool
	 on William from TIS has added a new agregate to alb1, which is a higher preference than the
	 damped /24, thus restoring connectivity, will remove it in an
	 hour or so and things should be ok.
	 This is just unacceptable.....
10:26	 Called back, it's now like 2 HOURS, still no word from telstra.
13546453 Evan advises they "had a major BGP fault, and they have about
	 50 faults open at the moment", they thought they had the problem
	 fixed, but evidently not. Still waiting on information.
08:15	 Wierd routing problems yet again between alb1 and alb3 at
13546453 telstra, cannot get onto 1800-066-594 (rang out repeatedly)
	 and the so-called ISP fault centre 1800-624-512 go thorough
	 to corporate faults who have ZERO CLUE about TCP/IP. 
	 These INCOMPETENT FOOLS have no idea how to run a network OR
	 a fault centre. 
	 traceroute to irc.sux.com (, 30 hops max, 40 byte packets
	  1  albury-core.albury.NET.AU (  5.074 ms
	  2  Serial4.alb1.Albury.telstra.net (  13.736 ms
	  3  *
	  4  *
	  5  GigabitEthernet3-0.lon-core3.Melbourne.telstra.net (  41.074 ms
	  6  *
	  7  *
	 Cannot even *RESOLVE* most domain names, traceroutes to our
	 USA servers same as above, die.
	 FINALLY got someone to answer the phones at 9:23!
15:20	 Alan at telstra is looking into this fault, re slow ping times
13537371 to the USA via perth, and to perth directly. International
	 RTT getting progressively worse. Seems to have started at or
	 about 4:50am this morning when something changed in sydney,
	 but didn't really get worse until about 11:30am
06:45	 Finally back on-air. Not bad for a maximum 30minute outage!

06:30	 Despite a known outage (20011210-0600-01) declared to be from
	 0600hrs AEDT on 10/12/2001 for 30 minutes, affecting alb3,
	 this activity appears to have commenced about 05:40, yet at
	 06:35 there is STILL no connectivity. Called in as a fault,
	 seems their engineer is still working on it, will notify as
	 soon as they know when it will be fixed.
22:14	 Telstra have just advised (Sat, 8 Dec 2001 21:42:23 +1100)
	 that there will be a 30-minute interruption to our service:
	 "The router alb3.Albury will be rebooted at 0600hrs AEDT on
	 10/12/2001 in order to rectify a fault."
	 I believe this actually translates to "we will be switching
	 over to the new 16 megabit ATM bearer instead of the 3x2M
	 circuits currently feeding this router".
17:20	 Telstra have confirmed the earlier outage was caused by them
13524343 stuffing about with a new ATM link, the installation of which
	 screwed up the albury-melbourne link. They have more playing
	 about to do, and are expecting at least one more outage of up
	 to 30 minutes duration sometime between now and 1am tomorrow.
16:20	 Reported fault at telstras albury POP. Trying to get someone
13524343 to investigate and fix it promptly.
	  1  3.824 ms  6.371 ms  6.477 ms
	  2  6.255 ms  6.486 ms  8.095 ms
	  3  * * *
	  4  * *  5.107 ms !H
	  5  * * *
	  6  * * *
	  7  8.019 ms !H * *
	  8  * * *
	  9  * * *
	 Fault seems to be intermittent cleared 16:26, re-ocurred 16:33
08:36	 While telstra have still been unable to confirm everything is
13513050 fixed, things APPEAR to be back to normal (whatever normal is!)
	 Hopefully there will be no subsequent problems, but rest assured
	 if there are, we'll be onto it quick smart.
20:58	 For goodness sake.... finally got onto hel$tra again and they
	 can only advise that this is a major, serious break (it is also
	 affecting Adelaide aparantly), and is as a result of fibre 
	 optic cable number 565, broken due to lightning strike (??)
	 23Km south of Yass. Still no ETA on completion of repairs.
	 (Funny the lightning tracking system shows NO lightning activity
	 in that area anywhere near the time in question!!)
17:16	 Clive from telstra has advised that there is indeed a MAJOR
	 network fault, and Albury, Wodonga and Wagga are all isolated
	 from the rest of Australia. 
	 They have "lost" 4 x 565 transmission bearers and have no
	 estimated time for restoration of services.
17:07	 Telstra have said there are no faults, then decided its the
13513050 storms in NSW (not withstanding they are 1000Km north of us!)
	 and in general seem completely unable to comprehend that there
	 might even be a problem. Still unable to contact telstra
	 countrywide here, mobile services still down.
16:27	 There are MASSIVE phone problems in Albury area, most of our
	 ISDN and phone lines are down, we cannot call telstra faults
	 on any of the fixed service lines, nor can we get out on the
	 telstra mobile network. Managed to get through to telstra on
	 an Optus mobile phone 16:40, NO information to hand yet.
04:15	 Albury Annex is showing very slow network response (300+mS).
	 Tech was unable to correct it remotely, attended the site
	 at 06:15 and has restored nominal operation.
08:01	 Severe packet loss and excessive latency in the USA affecting
	 some parts of USA and possibly other international connectivity.
	 15  p1-0.lsanca2-cr1.bbnplanet.net (  193.947 ms  205.726 ms  192.208 ms
	 16  * * *
	 17  * * p4-1-3-0.r00.lsanca01.us.bb.verio.net (  412.153 ms
	 Reported to telstra with traceroutes in both directions.
16:37	 Telstra Proxy in BAD shape.  Very slow.  Telstra called
13459475 twice to finally get some action.
	 2001/11/16 15:39:40| Detected DEAD Parent: proxy.cache.telstra.net/3128/3130
08:41	 Power restored, Corryong site back on line.

07:40	 Extensive power fail in the Corryong area, no indication on
	 when power may be restored. Our Corryong site is currently
	 off air (UPS exhausted at 8:07)
15:10	 Again (2nd time in as many weeks) mobilenet.telstra.net is
13440455 unreachable, thus no SMS messages getting out. Reported to 
	 telstra through the ISP faults centre but got someone without
	 ANY CLUE who is going to try and track someone down (perhaps).
	 bash-2.03# traceroute -q1 tmd-liv1.mobilenet.telstra.net
	  3  FastEthernet0-0.alb3.Albury.telstra.net (  5.803 ms
	  4  Serial5-1-7-1.lon-core2.Melbourne.telstra.net (  18.400 ms
	  5  FastEthernet1-0.lon29.Melbourne.telstra.net (  20.658 ms
	  6  *
	  7  *
12:00	 terran, one of the servers which hosts our users virtual websites,
	 reported 8 transient disk errors overnight. The source has been
	 traced to one 9Gb Quantum Viking SCSI drive. Whilst the errors
	 were recoverable, admins have removed the drive from service and
	 all content is now running from a Seagate Barracuda 9Gb LVD SCSI3
	 drive. Unfortunately, this required interruption to some virtual
	 services for a period of approximately 90 seconds. No data lost.
18:36	 Bega site restored.  Work on building STILL being done.
	 Looks like someone has cut power during work.
18:05	 Lost Power to Bega POP, calling out tech to see to problem

8:04	 Bega Pop failed to Come alive after work on power at Bega
	 overnight.  Tech sent to Bega site to restore systems.
	 Operation back to normal.
17:02	 Yackandandah indial number ringing unanswered. Reported to
13409720 Chris at ISP faults. He's checked and can't see anything wrong
	 so has called it in as a fault.
11:06	 Bega POP Scheduled outage.  8.30pm to 9.30pm Tuesday 30 Oct
	 2001.  Work being completed to Power in building at Bega 
13:50	 Corryong tigris has been spitting this message out about every 22 seconds:
13404788 *** TRAP from local agent at 29-Oct-2001 12:54:55 uptime 3 Days, 03:10:37
	 *** DM: Datalink error (unsolicited supervisory response), J7.3
	 Has been doing it for at least the last 3 hours. Reporte to
	 Anthea at the ISP faults centre. 1Hr response quoted.
08:50	 Telstra (again) have significant delays between lon-core2 
13391657 and lon-core3 routers in melbourne (a gigabit ethernet!)
	 (Anthea took the call, seemed somewhat unconcerned about it,
	 assured me there were no "known problems" in melbourne.
05:00	 There will be a brief interruption to connectivity at about
13378026 5:00am, Sunday morning, when telstra replace a faulty card
	 in their Albury router. Should only take 10 minutes, but we
	 are allowing for a 1-hour outage for this work.
14:50	 Link has been slow between telstras albury and melbourne routers.
13378026 3 FastEthernet0-0.alb3.Albury.telstra.net  5.477 ms
	 4 Serial5-1-7-1.lon-core2.Melbourne.telstra.net  396.481 ms
	 Called in as a fault, ISP fault centre not answering phone,
	 called 066594 and "tony" got all bent out of shape because
	 I didn't call the ISP number. On hold 20 mins in the end but
	 no real effort to get it fixed.
21:38	 Truly amazing. NOW they admit there is actually a problem,
13364417 after denying it for hours. They've sent a technician out to
	 investigate/fix. No ETA, but who cares?
20:32	 Called telstra back, spoke to Cliff. Still NO WORD from anyone
13364417 about what is (or is not) happening or being done. He has now
	 PROMISED to call me back shortly with an update.
	 Strewth, these guys are HOPELESS!
16:49	 Sydney and International connectivity getting progressively
13364417 worse over the last 2 hours. Called Telstra (Andrew). They 
	 were quite unaware of any problem, have taken a fault report
	 and should call back soon.
16:22	 We have had no advice from telstra, but from the significantly
	 improved performance over the last 12 hours or so, presume
	 the replacement bandwidth has finally come on-line.
21:50	 Sydney ken-pad latency issue seems fixed, although as usual
13313756 no word from telstra to say if it's permanent, transient,
	 really fixed, a fluke or of course, what caused it. Perhaps
	 they'll dane to tell us tomorrow.
21:12	 Useless GITS at telstra still can't tell me what is going on
	 with the problems at sydney, even though several other ISPs
	 are all reporting via IRC the same problems. Re-reported the
	 fault, perhaps someone will look at it in the next month!
18:40	 Reported fault to Nicole @ telstra faults, major delays on
13313756 the gigabit ethernet between ken-core4 in sydney and pad-core3
	 in the order of 200mS additional delay.
19:05	 Darren from Telstra just called.  Onramp service had problems
	 with Earth.  Telstra on site and restored problem.
	 Moruya Batemans Bay service now back on line
16:59	 Maro from Telstra called re Moruya link.  Telstra found
13308537 errors running on the On Ramp service.  Looking into the
	 fault and will advise.  At this stage until Telstra rectify
	 the fault, dial up services at Moruya Batemans Bay will NOT
	 be available
16:05	 Moruya Link down, Telstra contacted, suspect issues with
13308537 On Ramp service.  Waiting for Telstra to call back.
11:30	 Well, contrary to telstras earlier notice, there are more
	 problems with international capacity. A submarine cable
	 between Australia and the USA has been badly damaged near
	 China, killing 60% of telstras international capacity. They
	 are tying to rush an extra 622Mbps of capacity through SCC,
	 but are not expecting to have it on line until next week.
	 We're sorry for any inconvenience, but it's beyond our control.
00:00	 International traffic situation APPEARS resolved.

19:46	 Still can't get any connectivity past into the
13305671 USA. Reported as a fault (not just conjestion). Telstra are
	 still saying they are trying to get alternate capacity to the
	 USA until the singapore-USA cable cut can be fixed.
17:00	 Yes, all connectivity to international sites is WAY slow at
	 the moment. Telstra have a problem in Perth, and are re-routing
	 all traffic via Sydney, which has brought the whole network
	 to its knees. They are aware of the problem and are working
	 on it. No restoration time yet available.
08:56	 DDoS has returned as soon as the filters were removed.
13291117 Seeing 617 pkts/sec, 440922 bytes/sec of UDP traffic.
	 Tony is creating a UDP filter instead of a total block.
	 ACL169 on lon core2. Applied to all interfaces to albury.
	 Tony is still working on tracking the packets (9:10am). Will
	 call me back shortly with progress updates.
08:52	 After several more calls, Tony Van Reit is working on and
	 says he has fixed the filter on lon-core2, which should fix
	 the problems with connectivity on giroc. He also said that 
	 the "ACLs" on albury have nearly brought that router to its
	 knees with additional CPU.
07:45	 Seems last nights "block" by telstra was over-zealous. They
	 have totally blocked to ALL packets at
	 FastEthernet1-1-0.lon-core2.Melbourne.telstra.net (
	 Asked for urgent recall to correct the filter.
01:41	 Telstra finally called back to advise they have completed the
	 filters and to check things are all working ok now.
00:27	 Called Steve @ telstra yet again, he's calling to find out what
	 progress is. After 8 mins on the phone to Steve he still can't
	 tell me anything more. It appears to be not only a DDoS attack,
	 but something else as well. They have called in everyone they
	 can, but can't give any idea of ETR.
23:58	 There's a glimmer of light at the end of the tunnel. No more
	 DDoS observed for a couple of minutes, external connectivity
	 is back as far as melbourne now.
	  1  albury-core.albury.NET.AU (  4.748 ms
	  2  Serial4.alb1.Albury.telstra.net (  47.987 ms
	  3  FastEthernet0-0.alb3.Albury.telstra.net (  260.799 ms
	  4  Serial5-1-7-1.lon-core2.Melbourne.telstra.net (  269.162 ms
	  5  *
	  6  *
	 Still no word back from telstra yet.
	 Drat. Spoke too soon. Just as I'm about to post this, looks
	 like routing and connectivity are screwed again!
23:10	 Looks like telstra may have restored their router, but we
	 are now under a massive DDoS, in excess of 350Kbytes/Sec
	 via UDP directed at one of our hosts (giroc). Telstra tech
	 should be calling me shortly.
22:48	 Called telstra back again (Steve) for update. They are (or are
	 supposed to be) in the process of applying filters.
	 Been on hold 10 mins, no update yet (still)!
	 They've had to call in one of the remote techs.
	 Now estimating a further 15 minutes. (ie, 23:07)
22:36	 Reported to another party (AT, Sydney) by John at telstra,
	 that there is a ping-flood on the albury telstra router, and
	 telstra are putting filters on there now. ETR: 10 mins.
22:15	 We seem to be disconnected from the internet due to a fault
13291117 at the telstra albury router.
	 traceroute to dn.albury.net.au (, 30 hops max, 40 byte packets
	 1  albury-core.albury.NET.AU (  2.123 ms
	 2  Serial4.alb1.Albury.telstra.net (  6.452 ms
	 3  * * *
	 4  * * *
	 5  * * *
14:54	 Modem Card on Corryong site overheating causing
	 issues with dial up access.  Admin restored
	 dial up services promptly.
20:36 De-routed to reduce SOME of the codered crud.

06:20	 Annex stopped responding. Admin recalled to site to fix.

16:00	 Excessive delays to USA via Sydney (700mS).
13234482 Called in to Dale at telstra - problem appears to have gone
	 away now (evident from 15:30 to 16:00 approx)
16:44	 Merimbula Link reset and services restored to Normal

15:00	 We are seeing rather large ping times to across to Merimbula 
13232958 from Albury causing some delay across the link.  Report made
	 to Mat at Telstra to investigate.  The issue appeared at the
	 same time Telstra worked on our On-ramps at the Albury Exchange
	 Thursday morning 23.8.2001.  Also experiencing some packet loss.
08:36	 Albury Annex required a reset. Some services affected from
	 08:00 to 08:35 this morning.
11:15	 Corryong chassis replaced, all services back on-line, checked
	 and all services nominal.
10:30	 We will have a brief service outage at our Corryong POP for
	 approx 15 minutes on Thursday morning to facilitate the
	 exchange of a faulty server chassis.

10:08	 Found nearly 100 sendmail processes running on secondary mail
	 server - mostly for junkmail from "howareyoutoday.org" to
	 bunge.com (whos mail server was off-line). Deleted all the
	 junk, killed all the sendmails, everything cool. Also added
	 howareyoutoday.org to the access file to prevent a repeat.
09:22	 Connectivity problems from albury to melbourne, started 8am.

17:21	 Corryong tigris modem card has developed a fault.
	 Replaced 17:20
06:17	 Albury Annex stopped responding. Sysadmin paged, attended and
	 reset it, everything operational again.
09:11	 (Originally fault number 13165198) - Spoke to Trevor and they
13182625 changed ports on the albury AXE at 07-Aug-2001 19:22, we have
	 had NO FAULTS on the line since then. Appears that the real
	 cause has been the faulty port, but that there were also bad
	 or marginal (a) radio link (changed to fibre), (b) NT1 (replaced)
	 (c) optomux port (swapped), (d) Krone block (replaced). 
13:30	 News is restored.

12:09	 Telstra have advised that their "upgrade" of the victorian
	 news server have not gone to plan, and news is currently down.
	 They say only: "Services connected to this server will be 
	 interrupted for approximately 3 hours while this is being 
20:20	 Re fault 13168359. Reported to Chris @ telstra faults, who is
	 going to talk to William who will in turn contact Wayne to get
	 the low-down on the fault and its history etc. (Sheesh, no
	 wonder nothing ever gets done there!) (Between wel-core3 and
	 wel-gw1). Called back by William 21:10 - fault "disappeared" 
	 again, without resolution still. (He accidently took down the
	 interface and when he brought it back up, the problem had 
14:14	 Telstra techs have replaced the NT1 card at the corryong site
	 as there were still hits on the line since changing over the
	 bearer from radio to fibre.
13:16	 Telstra NDC techs have changed the bearer from microwave link
	 from corryong exchange back to albury, onto a fibre link, in
	 a bid to fix the clock-slips that we've been experiencing.
19:30	 Further reference to fault 13168359 - so much for "making it
	 easy" or "world class". What a joke! It's taken an HOUR to
	 get someone to answer the after hours fault number, and then
	 nobody is interested in DOING anything about the fault. Bad
	 latency to perth AGAIN (up from 70 to 280+ milliseconds).
	 I'm PROMISED a call back from a recall person shortly.
17:50	 Word back from telstra is that they believe they have found
13165198  a faulty krone block at the corryong exchange, which is being
	 changed out, along with moving our 2 megabit bearer from the
	 radio link to fibre (as it should have been from day 1!)
	 We expect a 5-minute interruption tomorrow while changes are made.
13:20	 Have spoken to Andrew Cotterill, general manager, albury,
13165198 countrywide, who has undertaken to find out what the hell is
	 going on, and when it will be fixed and what the delays have
	 been (with reference to the Corryong problems)
10:48	 Called telstra yet again (about the 12th time since this issue
13165198 started) spoke to Anthea who PROMISED a return call within an
	 hour. Still no reply by midday, called Matt at midday, he is
	 reading the original notes from Mick. He's calling the wideband
	 group again now to find whats going on.

20:38	 Last nights fault was reported to Wayne who asked for me to
13171950 let him know if it happened again. Called in tonights fault
	 but they don't want to send it to Wayne and are investigating
	 the fault themselves. Initially, only USA via Syd affected,
	 but by 20:55, perth domestic and international showing the
	 same effect as before.
11:10	 Excessive delays international via perth, reported to telstra
13168359 who are investigating it now.
11:10	 Office phones, all lines, again dead. Finally got some back,
13166135 called telstra corporate faults - but they're overloaded and
	 have stuck us on hold!
08:39	 Ongoing errors at the new Corryong OptoMux
13165198 Started 3:01am.
14:10	 Users of our Corryong facilities should change phone number
	 FROM the old number: 763000 to 763300 IMMEDIATELY.
11:40	 DoS traffic appears to have been directed at the albury annex,
	 taking it off-line. Admin recalled to site to correct, all
	 back online and operational 10:58.
17:33	 Well, telstra still have not confirmed they have (or have NOT)
13139044  fixed the problem, but all lines seem to have been operational
	 from about 30 minutes after the problem was reported.
	 No INDIAL lines were affected, although various customers may
	 have had THEIR phone lines down too.
09:55	 Talking with telstra, seems half our office phone lines are
13139044 down. They have confirmed a major exchange problem and are 
	 working on it. (Two fault nos, 13139014 data, 13139044 voice)
14:01	 Out-of-town modem pool (analogue only) is not holding calls.
	 It's only one pool, seems to be the CentelPlus groups, called
	 telstra who are testing. (But then, while testing it, it has
	 come good! Damn!) Problem definately appears to be in the
	 telstra network as other calls on the same NAS were fine.
16:15	 News services restored by telstra.

14:15	 Telstra NSWPULL news server is refusing connections.
13093986 Reported to Clive after it's been down an hour, for attention.
13:10	 Telstra have confirmed they've identified and fixed the fault.
13068156 Seems one of the Packet-over-SONET links from the USA to AUS
	 that was ACTIVE, was put into loopback test in error. Strange
	 they didn't notice or discover it until I reported the fault.
	 *sigh*. World class carriers indeed.
11:45	 Reported to Bruce Mccoll @ telstra. Admits there have been
13068156 problems with sydney routers all this week. Is checking now.
	  6  GigabitEthernet4-0.win-core1.Melbourne.telstra.net (  25.888 ms
	  7  Pos2-0.ken-core4.Sydney.telstra.net (  40.231 ms
	  8  GigabitEthernet3-1.pad-core3.Sydney.telstra.net (  38.760 ms
	  9  Pos2-3.pad-core3.Sydney.telstra.net (  309.558 ms
	 10  Pos2-3.pad-core3.Sydney.telstra.net (  803.851 ms
	 11  Pos2-3.pad-core3.Sydney.telstra.net (  1331.665 ms
	 12  Pos2-3.pad-core3.Sydney.telstra.net (  1879.476 ms
	 13  Pos2-3.pad-core3.Sydney.telstra.net (  2398.007 ms
	 14  Pos2-3.pad-core3.Sydney.telstra.net (  2935.739 ms
09:14	 Advised Telstra Outage - Telstra Albury Router
	 The router ALB3 has a faulty card being replaced 
	 at 19:45 hrs AEST on 15/06/2001.
	 Services connected to this router will be interrupted
	 for approximately 5 minutes during this activity.
08:00	 We plan to shutdown the Albury Tigris for approx 5 minutes
	 on 15-Jun-2001 at approx 8am in order to install an extra
	 60 port digital modem card. Should cause minimal disruption
15:55	 Albury Annex Dropped DTR.  On Site - immediate reset
	 Reset and back to normal 16:03
18:03	 Slow response to the USA - Telstra are working on it (since
13025368 17:40) - sudden increase in latency to the USA about 1700mS
20:30	 Albury Annex has dropped DTR on all modems again. Arghh!
	 Staff recalled to site, server operational again 20:50.
17:41	 Albury Annex has dropped DTR on all modems.
	 Staff were still on site, reset immediately, operational
	 again 17:46.
10:34	 Maro from ISP Faults centre has done a complete system reset
13014543 of the NT1 and interface to force a reset of the OR30 system.
	 Now running no errors, which is a big improvement.
10:00	 Still running slip errors. (Badly....)
13014543 Spoke to Hans, then on to Maro who has done some testing,
	 confirmed it's out of Albury Exchange (ALBG88). Layer 1 up.
	 Layer 2. Telstra will send a tech.
11:30	 Chris called back, seems that fault cause was "Unknown" and
13014543 no further action will be taken. I've STRONGLY requested they
	 investigate further tomorrow to actually identify a cause
	 BEFORE they close the fault.
10:36	 Cate Valon, Corp faults, Sydney totally useless GIMP, had
13014543 absolutely no idea about anything, finally transferred me to
	 the 1800-624-512 number we're NOT allowed to call as that is
	 for bigpong customers, WE have to call the ISP fault centre
	 (which is what I did, and got this ignorant cow in the 1st place!)
	 Chris at big pond says he can see from the logs their end
	 SES errors (section 1, ie: within telstra network)
	 Dip fault (power feed) Confirmed failure 10:13am
	 Is investigating and will call me back.
	 Fault is that ALL lines to alb-tig1 disconnected ALL users.
17:10	 Telstras news server is refusing connections again (and has
	 been since 16:50). Of course, it's a "non critical" system
	 and won't be attended to until next week. I've reported it
	 so at least they are aware of it. Who knows, perhaps they 
	 will fix it soon anyway?

02:50	 Telstra advises:
20010506-0250-01 There is a problem with the trunk network circuit from Sydney
	 to Tokyo, Sydney to Taipei, Sydney to China, Perth to Palo 
	 Alto (USA). This may be causing degraded performance to 
	 Internet traffic between these regions.
	 Telstra have not given any restoration time, it is possible
	 that the slow international response still being experienced
	 (from 9am onwards) is a direct result.
16:10	 Finally, our phones seem to be fixed. Seems some GIMP at
	 telstra had pulled some wrong jumpers in the exchange!
11:10	 Reported to Maro - our primary office numer (60402692) is
12933544 dead again - side tone but no dial tone. Customers reporting
	 phone ringing out.
14:15	 We're down from 5 phone lines into the office to just ONE.
12931262 Have called fault in to telstra who seem to think it's a
	 line fault and they will have to get a liney out. Could be
	 a couple of days before someone can get here. I'm escalating
	 it as far as I can to expedite it.
10:00	 Bandwidth to our Wangaratta POP was doubled at 10am today.

01:00	 Telstra have advised there will be 5-minute interruption to
P36204V service for all our MtBeauty and Wangaratta facilities on
	 15-May somewhere between 00:01-03:30.
13:00	 The time has come - our old Corrying server has been turned
	 OFF now and has been de-commissioned. 
11:50	 News server is back up.

08:55	 Tony from Telstra reports that yes, the Sydney news server
	 is indeed down. It's a non-critical service so no one will
	 look at the problem until Tuesday.

18:55	 Telstra's Sydney NNTP (Usenet News) server is down.
12884843 Reported to Frank.

19:55	 Alb-anx1 stopped responding. Tech recalled to site 20:00,
	 system restarted 20:03, back on-line 20:08 and taking calls.
17:00	 Our new PoP in Corryong is ready for users to start testing.
	 56K class connections have finally made it to Corryong. For
	 full information, users affected by this change, have been
	 sent details by e-mail.
21:40	 Repeat of last nights abysmal performance (1400mS) to first
12818319 hop to the USA via Sydney. (Fault from last night still open)
21:50	 Meiling returned my call. All telstras capacity through sydney
	 is at or near 100%. No capacity via perth due to the (still)
	 cut china cables. She will call David Woodgate now to see if
	 they can bring on additional capacity, or loadbalance better.
	 Otherwise, there's nothing they can do. Expect more of the 
	 same for the next 3 weeks or so until china cable is fixed.
21:08	 Excessive ping times to the USA (>1500mS). Reported to John
12818319 telstra faults. Been like this for the last 35 minutes.
14:20	 Power finally restored, although major works are required
	 next week to replace the main switchboard and various of
	 the control, generator-changeover and phase-fail sensors.
	 At least we're back off generator after 13 hours.
09:30	 Secondary site still running on generator. Widespread damage
	 from this mornings storm. Power board is blown off the side
	 of the building, electricians are in attendance, but power
	 is not expect back for many hours. Quite a few bits of our
	 equipment have sustained damage, it appears that it was either
	 a direct lightning strike, or the 22KV power has been brought
	 down into contact with the LV supply.
01:25	 Secondary site power fail. Whole area is without power, we
	 are running on generator.
01:20	 This is unbelievable. Both Mt Beauty and Wangaratta gone for
	 a 3rd time. Heads will surely have to roll....
01:06	 Not before damn time, looks like tel$ux have finally got the
	 lines back up after re-resetting the exchange or lines.
	 Still no idea what the hell caused the problems and basically
	 denying any liability.
00:04	 Both Mt Beauty and Wangaratta gone AGAIN! RE-reported.

23:36	 Both Wang and MtBeauty servers back, although no advice from
	 telstra. Nicole (tester, melbourne) advised 10 mins ago that
	 it was a major fault at the wang exchange, ETR 30-60 mins.
	 Compensation paperwork to be sent from Customer fault 
	 management tomorrow re all the lost user records.
22:50	 Wangaratta completely disappeared.
12802112 Reported to corporate faults.
22:50	 Mt Beauty completely disappeared.
12802111 Reported to corporate faults.
13:30	 Replacement equipment has been sourced, both ends of our
	 microwave link have been replaced with borrowed equipment.
	 Chartered aircraft will have replacements here by 4pm.
11:30	 One Microwave unit has failed. Attempting to affect repair
	 or replacement ASAP. Secondary DNS, camera and fax gateway
	 are unreachable.
19:13	 John at telstra advised massive delays at Sydney AGAIN for 
12789016 all international traffic. Traceroute to microsoft, or any
	 USA site I've been able to check is the same:
	  8  GigabitEthernet3-1.pad-core3.Sydney.telstra.net (  48.102 ms  38.217 ms  44.713 ms
	  9 (  1722.288 ms  1680.338 ms  1718.552 ms
	 10  f0.pao1.verio.net (  1665.194 ms  1713.789 ms  1701.922 ms
10:55	 5 minute interruption to service to re-install new UPS for
	 our Wangaratta facilities. All customers connecting to our
	 Wangaratta POP will have experienced a 5-minute outage.
09:00	 Telstra news is unreachable since 5:25am, dies at 
12748122 FastEthernet1-0.pad-service1.Sydney.telstra.net
20:15	 Chris at bigpond has officially recognized (eventually) that
12744629 there is indeed a major routing problem australia wide, has
	 gone to report and get action.
19:56	 Damn telstra to the 7th level of hell! They have a broken
	 router in albury has died, yet none of the faults numbers
	 are answering.
13:23	 Traffic DEAD yet again at 
12742933 GigabitEthernet3-1.pad-core3.Sydney.telstra.net
	 Reported to telstra, seems we're the first to report it.
12:45	 News server up again.

12:20	 Telstra NNTP (Usenet) server down. Reported to Darren.

19:45	 The Annex terminal server has stopped responding.
	 Technician recalled to site, operational again 19:57
13:02	 Most national and all international connectivity failed at
12731333 sydney. Reported to Larry @ telstra. Confirmed fault, they
	 are working on it now.
15:59	 Wangaratta site sync lost on OR30 yet again. Reported to 
12684785 telstra. Terry. Seems the 4th-Feb-2001 fault HAS been left
	 open on QMR (Quality Monitoring), recalling someone now.
	 Link restored 40 seconds later.

17:20	 Power restored at Corryong, site back up and running.

16:05	 UPS and remote site shutdown to preserve batteries.
15:35	 Power to Corryong has been lost. TXO advises of a widespread
	 failure affecting the whole area, restoration not expected
	 until 18:00 tonight.
15:35	 Power to Corryong has been lost. TXO advises of a widespread
 failure affecting the whole area, restoration not expected
	 until 18:00 tonight.
15:43	 General national routing instability, reported to Peter.
12705602 "Complete network avalanche failure" - the core links having
	 failed seems to have just screwed the entire national network
	 and they currently have no idea on where or why, or when it
	 will all be fixed.
14:20	 A sand dredging ship has cut a submarine cable off
	 Singapore. No ETA on fix yet.
	 Performance to sites outside Australia may be poor.

10:30	 Very high latency Perth -> Palo Alto. In excess of 1500ms.
12703455 International network performance sub optimal. Reported
	 to Darren.

16:30	 High latency PER -> SFO.
12693844 Telstra advised egress links at 96% capacity, new capacity
	 coming online "soon".
13:14	 Wangaratta site has returned. No fault at our equipment,
12684785 logged yet another fault with telstra. Sooner or later
	 I hope somebody will give up and actually FIX it!
13:10	 Wangaratta site has disappeared. Suspecting storm/power
	 problems affecting telstra, calling them now.
15:35	 A small technical "glitch" during software testing resulted
	 in interrupted core connectivity for a few minutes. No data
	 loss or loss of customer connectivity resulted.
17:10	 Telstra have been unable to find any faults with the OR30 in
	 Wang and closed the fault. If it does happen again, they'll
	 put an analyser on the line and monitor it more closely.
09:38	 Wangaratta OR30 failed yet again. Fault reported to telstra.
12658655 We'll see if they actually FIX it this time.

18:40	 Telstra technician reports ALL onramp services in wangaratta
	 have been flapping up and down from the same time period,
	 they have no evidence of what the exact problem is/was, but
	 have blamed it on the number and severity of local storms.

17:43	 Wangaratta OnRamp service seems to have become faulty.
12658333 Reported to telstra, they are recalling a tech to test/fix
	 it, but no ETA at this stage. The Onramp seems to be bringing
	 the D channel up and dropping it every 30-90 seconds :(

21:45	 Some residual impact from the server failures earlier today, some
	 people with only one DNS configured were unable to resolve some
	 names. Tracked to being a perculiar issue with named not binding
	 to an aliased port, full operation should now be restored.

18:00	 We've sucessfully managed to return the main UPS to service
	 with zero downtime due to some "creative" cabling solutions.

16:01	 Our APC UPS last reserve for our main servers went BANG without
	 any notice, took all our servers out and spiked our primary UPS
	 which went into a latched bypass mode. We have returned all servers
	 to service after approx 7 minutes, however a later shutdown MAY be
	 required to return our main UPS to normal operation. We are still
	 investigating zero-downtime alternatives to bringing back normal

	 Our games server has a developing hardware failure. While the server
	 is currently operational, we are planning a maintenance shutdown
	 for tomorrow to replace it, and will take the oportunity to also
	 perform some other upgrades to the system.

00:30	 Telstra advise an exchange software update will be done on our
	 exchange somewhere between midnight and 1am today, which should
	 create approximately 5-minutes of downtime.
	 This will affect all Corryong, Wangaratta and MtBeauty users,
	 and those Albury users calling our digital numbers.

15:44	 Appears there are problems with Telstra services in Corryong.
12615310 Many customers reporting inability to get connected, along with
"Harry"	 general faults involving PSTN services (NOT only Internet facilities).
	 A fault has been lodged with Telstra.

02:30	 Telstra advise "Urgent Transmission Work" is to take place today
	 sometime between midnight and 5am. An outage of approximately 2
	 minutes is expected. Customers calling our Mt Beauty, Corryong
	 and Albury digital facilities may be affected.

15:00	 A brief outage at our Wangaratta POP (approx 10mins) will occur
	 to upgrade power supply systems.

05:15	 Wangaratta power restored.

04:10	 Power outage in Wangaratta. Seems fairly widespread. Customers in
	 Wangaratta, Killawarra, Milawa etc affected.

16:20	 An unscheduled reboot of our Annex terminal server was required
	 to correct an operational issue (routes not being added).
	 Approximately 5 minutes downtime was experienced by users from
	 Corowa/Rutherglen, Culcairn, Holbrook and surrounding areas.