Historical status reports for 2001
17:08 Merimbula Power restored finally ! Country Link Energy could
only tell me that someone had taken out a power pole in the
CBD which knocked out all Power in Merimbula.
Tech also called on site to check servers. Systems checked
remotely and all services back to normal.
16:00 Merimbula Completely DOWN, Major Power fault in Merimbula
causing all of the CBD to be taken out. UPS went as far as
it could, but had to give up sooner or later.
Waiting for Country Energy to call back with ETA for Power
to be restored.
09:05 When the revolution comes, this damn annex will be the first
one up against the wall. Failing to respond, not pinging.
Tech called to site to fix. This one is earmarked for replacement
early in the new year.
14:30 Annex had a slight hickup. Called on-site to rectify.
Once again, systems fully operational
06:58 Arghh! The Annex was responding, but not well. High pings, no
DTR asserted, a few users logged in but not doing much.
Tech attended site, graceful restart has cured things for now.
16:48 Ping times from Perth to USA way to high 1000ms to 2000ms
13556666 Telstra called, it's on their network, so we will wait and
see how long it takes before someone can address it
11:03 At long bloody last. I've determined that our dialup pools were
13546453 able to route ok (22.214.171.124/23) but not our main server pool
on 126.96.36.199/24. William from TIS has added a new agregate
188.8.131.52/23 to alb1, which is a higher preference than the
damped /24, thus restoring connectivity, will remove it in an
hour or so and things should be ok.
This is just unacceptable.....
10:26 Called back, it's now like 2 HOURS, still no word from telstra.
13546453 Evan advises they "had a major BGP fault, and they have about
50 faults open at the moment", they thought they had the problem
fixed, but evidently not. Still waiting on information.
08:15 Wierd routing problems yet again between alb1 and alb3 at
13546453 telstra, cannot get onto 1800-066-594 (rang out repeatedly)
and the so-called ISP fault centre 1800-624-512 go thorough
to corporate faults who have ZERO CLUE about TCP/IP.
These INCOMPETENT FOOLS have no idea how to run a network OR
a fault centre.
traceroute to irc.sux.com (184.108.40.206), 30 hops max, 40 byte packets
1 albury-core.albury.NET.AU (220.127.116.11) 5.074 ms
2 Serial4.alb1.Albury.telstra.net (18.104.22.168) 13.736 ms
5 GigabitEthernet3-0.lon-core3.Melbourne.telstra.net (22.214.171.124) 41.074 ms
Cannot even *RESOLVE* most domain names, traceroutes to our
USA servers same as above, die.
FINALLY got someone to answer the phones at 9:23!
15:20 Alan at telstra is looking into this fault, re slow ping times
13537371 to the USA via perth, and to perth directly. International
RTT getting progressively worse. Seems to have started at or
about 4:50am this morning when something changed in sydney,
but didn't really get worse until about 11:30am
06:45 Finally back on-air. Not bad for a maximum 30minute outage!
06:30 Despite a known outage (20011210-0600-01) declared to be from
0600hrs AEDT on 10/12/2001 for 30 minutes, affecting alb3,
this activity appears to have commenced about 05:40, yet at
06:35 there is STILL no connectivity. Called in as a fault,
seems their engineer is still working on it, will notify as
soon as they know when it will be fixed.
22:14 Telstra have just advised (Sat, 8 Dec 2001 21:42:23 +1100)
that there will be a 30-minute interruption to our service:
"The router alb3.Albury will be rebooted at 0600hrs AEDT on
10/12/2001 in order to rectify a fault."
I believe this actually translates to "we will be switching
over to the new 16 megabit ATM bearer instead of the 3x2M
circuits currently feeding this router".
17:20 Telstra have confirmed the earlier outage was caused by them
13524343 stuffing about with a new ATM link, the installation of which
screwed up the albury-melbourne link. They have more playing
about to do, and are expecting at least one more outage of up
to 30 minutes duration sometime between now and 1am tomorrow.
16:20 Reported fault at telstras albury POP. Trying to get someone
13524343 to investigate and fix it promptly.
1 126.96.36.199 3.824 ms 6.371 ms 6.477 ms
2 188.8.131.52 6.255 ms 6.486 ms 8.095 ms
3 * * *
4 * * 184.108.40.206 5.107 ms !H
5 * * *
6 * * *
7 220.127.116.11 8.019 ms !H * *
8 * * *
9 * * *
Fault seems to be intermittent cleared 16:26, re-ocurred 16:33
08:36 While telstra have still been unable to confirm everything is
13513050 fixed, things APPEAR to be back to normal (whatever normal is!)
Hopefully there will be no subsequent problems, but rest assured
if there are, we'll be onto it quick smart.
20:58 For goodness sake.... finally got onto hel$tra again and they
can only advise that this is a major, serious break (it is also
affecting Adelaide aparantly), and is as a result of fibre
optic cable number 565, broken due to lightning strike (??)
23Km south of Yass. Still no ETA on completion of repairs.
(Funny the lightning tracking system shows NO lightning activity
in that area anywhere near the time in question!!)
17:16 Clive from telstra has advised that there is indeed a MAJOR
network fault, and Albury, Wodonga and Wagga are all isolated
from the rest of Australia.
They have "lost" 4 x 565 transmission bearers and have no
estimated time for restoration of services.
17:07 Telstra have said there are no faults, then decided its the
13513050 storms in NSW (not withstanding they are 1000Km north of us!)
and in general seem completely unable to comprehend that there
might even be a problem. Still unable to contact telstra
countrywide here, mobile services still down.
16:27 There are MASSIVE phone problems in Albury area, most of our
ISDN and phone lines are down, we cannot call telstra faults
on any of the fixed service lines, nor can we get out on the
telstra mobile network. Managed to get through to telstra on
an Optus mobile phone 16:40, NO information to hand yet.
04:15 Albury Annex is showing very slow network response (300+mS).
Tech was unable to correct it remotely, attended the site
at 06:15 and has restored nominal operation.
08:01 Severe packet loss and excessive latency in the USA affecting
some parts of USA and possibly other international connectivity.
15 p1-0.lsanca2-cr1.bbnplanet.net (18.104.22.168) 193.947 ms 205.726 ms 192.208 ms
16 * * *
17 * * p4-1-3-0.r00.lsanca01.us.bb.verio.net (22.214.171.124) 412.153 ms
Reported to telstra with traceroutes in both directions.
16:37 Telstra Proxy in BAD shape. Very slow. Telstra called
13459475 twice to finally get some action.
2001/11/16 15:39:40| Detected DEAD Parent: proxy.cache.telstra.net/3128/3130
08:41 Power restored, Corryong site back on line.
07:40 Extensive power fail in the Corryong area, no indication on
when power may be restored. Our Corryong site is currently
off air (UPS exhausted at 8:07)
15:10 Again (2nd time in as many weeks) mobilenet.telstra.net is
13440455 unreachable, thus no SMS messages getting out. Reported to
telstra through the ISP faults centre but got someone without
ANY CLUE who is going to try and track someone down (perhaps).
bash-2.03# traceroute -q1 tmd-liv1.mobilenet.telstra.net
3 FastEthernet0-0.alb3.Albury.telstra.net (126.96.36.199) 5.803 ms
4 Serial5-1-7-1.lon-core2.Melbourne.telstra.net (188.8.131.52) 18.400 ms
5 FastEthernet1-0.lon29.Melbourne.telstra.net (184.108.40.206) 20.658 ms
12:00 terran, one of the servers which hosts our users virtual websites,
reported 8 transient disk errors overnight. The source has been
traced to one 9Gb Quantum Viking SCSI drive. Whilst the errors
were recoverable, admins have removed the drive from service and
all content is now running from a Seagate Barracuda 9Gb LVD SCSI3
drive. Unfortunately, this required interruption to some virtual
services for a period of approximately 90 seconds. No data lost.
18:36 Bega site restored. Work on building STILL being done.
Looks like someone has cut power during work.
18:05 Lost Power to Bega POP, calling out tech to see to problem
8:04 Bega Pop failed to Come alive after work on power at Bega
overnight. Tech sent to Bega site to restore systems.
Operation back to normal.
17:02 Yackandandah indial number ringing unanswered. Reported to
13409720 Chris at ISP faults. He's checked and can't see anything wrong
so has called it in as a fault.
11:06 Bega POP Scheduled outage. 8.30pm to 9.30pm Tuesday 30 Oct
2001. Work being completed to Power in building at Bega
13:50 Corryong tigris has been spitting this message out about every 22 seconds:
13404788 *** TRAP from local agent at 29-Oct-2001 12:54:55 uptime 3 Days, 03:10:37
*** DM: Datalink error (unsolicited supervisory response), J7.3
Has been doing it for at least the last 3 hours. Reporte to
Anthea at the ISP faults centre. 1Hr response quoted.
08:50 Telstra (again) have significant delays between lon-core2
13391657 and lon-core3 routers in melbourne (a gigabit ethernet!)
(Anthea took the call, seemed somewhat unconcerned about it,
assured me there were no "known problems" in melbourne.
05:00 There will be a brief interruption to connectivity at about
13378026 5:00am, Sunday morning, when telstra replace a faulty card
in their Albury router. Should only take 10 minutes, but we
are allowing for a 1-hour outage for this work.
14:50 Link has been slow between telstras albury and melbourne routers.
13378026 3 FastEthernet0-0.alb3.Albury.telstra.net 5.477 ms
4 Serial5-1-7-1.lon-core2.Melbourne.telstra.net 396.481 ms
Called in as a fault, ISP fault centre not answering phone,
called 066594 and "tony" got all bent out of shape because
I didn't call the ISP number. On hold 20 mins in the end but
no real effort to get it fixed.
21:38 Truly amazing. NOW they admit there is actually a problem,
13364417 after denying it for hours. They've sent a technician out to
investigate/fix. No ETA, but who cares?
20:32 Called telstra back, spoke to Cliff. Still NO WORD from anyone
13364417 about what is (or is not) happening or being done. He has now
PROMISED to call me back shortly with an update.
Strewth, these guys are HOPELESS!
16:49 Sydney and International connectivity getting progressively
13364417 worse over the last 2 hours. Called Telstra (Andrew). They
were quite unaware of any problem, have taken a fault report
and should call back soon.
16:22 We have had no advice from telstra, but from the significantly
improved performance over the last 12 hours or so, presume
the replacement bandwidth has finally come on-line.
21:50 Sydney ken-pad latency issue seems fixed, although as usual
13313756 no word from telstra to say if it's permanent, transient,
really fixed, a fluke or of course, what caused it. Perhaps
they'll dane to tell us tomorrow.
21:12 Useless GITS at telstra still can't tell me what is going on
with the problems at sydney, even though several other ISPs
are all reporting via IRC the same problems. Re-reported the
fault, perhaps someone will look at it in the next month!
18:40 Reported fault to Nicole @ telstra faults, major delays on
13313756 the gigabit ethernet between ken-core4 in sydney and pad-core3
in the order of 200mS additional delay.
19:05 Darren from Telstra just called. Onramp service had problems
with Earth. Telstra on site and restored problem.
Moruya Batemans Bay service now back on line
16:59 Maro from Telstra called re Moruya link. Telstra found
13308537 errors running on the On Ramp service. Looking into the
fault and will advise. At this stage until Telstra rectify
the fault, dial up services at Moruya Batemans Bay will NOT
16:05 Moruya Link down, Telstra contacted, suspect issues with
13308537 On Ramp service. Waiting for Telstra to call back.
11:30 Well, contrary to telstras earlier notice, there are more
problems with international capacity. A submarine cable
between Australia and the USA has been badly damaged near
China, killing 60% of telstras international capacity. They
are tying to rush an extra 622Mbps of capacity through SCC,
but are not expecting to have it on line until next week.
We're sorry for any inconvenience, but it's beyond our control.
00:00 International traffic situation APPEARS resolved.
19:46 Still can't get any connectivity past 220.127.116.11 into the
13305671 USA. Reported as a fault (not just conjestion). Telstra are
still saying they are trying to get alternate capacity to the
USA until the singapore-USA cable cut can be fixed.
17:00 Yes, all connectivity to international sites is WAY slow at
the moment. Telstra have a problem in Perth, and are re-routing
all traffic via Sydney, which has brought the whole network
to its knees. They are aware of the problem and are working
on it. No restoration time yet available.
08:56 DDoS has returned as soon as the filters were removed.
13291117 Seeing 617 pkts/sec, 440922 bytes/sec of UDP traffic.
Tony is creating a UDP filter instead of a total block.
ACL169 on lon core2. Applied to all interfaces to albury.
Tony is still working on tracking the packets (9:10am). Will
call me back shortly with progress updates.
08:52 After several more calls, Tony Van Reit is working on and
says he has fixed the filter on lon-core2, which should fix
the problems with connectivity on giroc. He also said that
the "ACLs" on albury have nearly brought that router to its
knees with additional CPU.
07:45 Seems last nights "block" by telstra was over-zealous. They
have totally blocked 18.104.22.168/32 to ALL packets at
Asked for urgent recall to correct the filter.
01:41 Telstra finally called back to advise they have completed the
filters and to check things are all working ok now.
00:27 Called Steve @ telstra yet again, he's calling to find out what
progress is. After 8 mins on the phone to Steve he still can't
tell me anything more. It appears to be not only a DDoS attack,
but something else as well. They have called in everyone they
can, but can't give any idea of ETR.
23:58 There's a glimmer of light at the end of the tunnel. No more
DDoS observed for a couple of minutes, external connectivity
is back as far as melbourne now.
1 albury-core.albury.NET.AU (22.214.171.124) 4.748 ms
2 Serial4.alb1.Albury.telstra.net (126.96.36.199) 47.987 ms
3 FastEthernet0-0.alb3.Albury.telstra.net (188.8.131.52) 260.799 ms
4 Serial5-1-7-1.lon-core2.Melbourne.telstra.net (184.108.40.206) 269.162 ms
Still no word back from telstra yet.
Drat. Spoke too soon. Just as I'm about to post this, looks
like routing and connectivity are screwed again!
23:10 Looks like telstra may have restored their router, but we
are now under a massive DDoS, in excess of 350Kbytes/Sec
via UDP directed at one of our hosts (giroc). Telstra tech
should be calling me shortly.
22:48 Called telstra back again (Steve) for update. They are (or are
supposed to be) in the process of applying filters.
Been on hold 10 mins, no update yet (still)!
They've had to call in one of the remote techs.
Now estimating a further 15 minutes. (ie, 23:07)
22:36 Reported to another party (AT, Sydney) by John at telstra,
that there is a ping-flood on the albury telstra router, and
telstra are putting filters on there now. ETR: 10 mins.
22:15 We seem to be disconnected from the internet due to a fault
13291117 at the telstra albury router.
traceroute to dn.albury.net.au (220.127.116.11), 30 hops max, 40 byte packets
1 albury-core.albury.NET.AU (18.104.22.168) 2.123 ms
2 Serial4.alb1.Albury.telstra.net (22.214.171.124) 6.452 ms
3 * * *
4 * * *
5 * * *
14:54 Modem Card on Corryong site overheating causing
issues with dial up access. Admin restored
dial up services promptly.
20:36 126.96.36.199/24 De-routed to reduce SOME of the codered crud.
06:20 Annex stopped responding. Admin recalled to site to fix.
16:00 Excessive delays to USA via Sydney (700mS).
13234482 Called in to Dale at telstra - problem appears to have gone
away now (evident from 15:30 to 16:00 approx)
16:44 Merimbula Link reset and services restored to Normal
15:00 We are seeing rather large ping times to across to Merimbula
13232958 from Albury causing some delay across the link. Report made
to Mat at Telstra to investigate. The issue appeared at the
same time Telstra worked on our On-ramps at the Albury Exchange
Thursday morning 23.8.2001. Also experiencing some packet loss.
08:36 Albury Annex required a reset. Some services affected from
08:00 to 08:35 this morning.
11:15 Corryong chassis replaced, all services back on-line, checked
and all services nominal.
10:30 We will have a brief service outage at our Corryong POP for
approx 15 minutes on Thursday morning to facilitate the
exchange of a faulty server chassis.
10:08 Found nearly 100 sendmail processes running on secondary mail
server - mostly for junkmail from "howareyoutoday.org" to
bunge.com (whos mail server was off-line). Deleted all the
junk, killed all the sendmails, everything cool. Also added
howareyoutoday.org to the access file to prevent a repeat.
09:22 Connectivity problems from albury to melbourne, started 8am.
17:21 Corryong tigris modem card has developed a fault.
06:17 Albury Annex stopped responding. Sysadmin paged, attended and
reset it, everything operational again.
09:11 (Originally fault number 13165198) - Spoke to Trevor and they
13182625 changed ports on the albury AXE at 07-Aug-2001 19:22, we have
had NO FAULTS on the line since then. Appears that the real
cause has been the faulty port, but that there were also bad
or marginal (a) radio link (changed to fibre), (b) NT1 (replaced)
(c) optomux port (swapped), (d) Krone block (replaced).
13:30 News is restored.
12:09 Telstra have advised that their "upgrade" of the victorian
news server have not gone to plan, and news is currently down.
They say only: "Services connected to this server will be
interrupted for approximately 3 hours while this is being
20:20 Re fault 13168359. Reported to Chris @ telstra faults, who is
going to talk to William who will in turn contact Wayne to get
the low-down on the fault and its history etc. (Sheesh, no
wonder nothing ever gets done there!) (Between wel-core3 and
wel-gw1). Called back by William 21:10 - fault "disappeared"
again, without resolution still. (He accidently took down the
interface and when he brought it back up, the problem had
14:14 Telstra techs have replaced the NT1 card at the corryong site
as there were still hits on the line since changing over the
bearer from radio to fibre.
13:16 Telstra NDC techs have changed the bearer from microwave link
from corryong exchange back to albury, onto a fibre link, in
a bid to fix the clock-slips that we've been experiencing.
19:30 Further reference to fault 13168359 - so much for "making it
easy" or "world class". What a joke! It's taken an HOUR to
get someone to answer the after hours fault number, and then
nobody is interested in DOING anything about the fault. Bad
latency to perth AGAIN (up from 70 to 280+ milliseconds).
I'm PROMISED a call back from a recall person shortly.
17:50 Word back from telstra is that they believe they have found
13165198 a faulty krone block at the corryong exchange, which is being
changed out, along with moving our 2 megabit bearer from the
radio link to fibre (as it should have been from day 1!)
We expect a 5-minute interruption tomorrow while changes are made.
13:20 Have spoken to Andrew Cotterill, general manager, albury,
13165198 countrywide, who has undertaken to find out what the hell is
going on, and when it will be fixed and what the delays have
been (with reference to the Corryong problems)
10:48 Called telstra yet again (about the 12th time since this issue
13165198 started) spoke to Anthea who PROMISED a return call within an
hour. Still no reply by midday, called Matt at midday, he is
reading the original notes from Mick. He's calling the wideband
group again now to find whats going on.
20:38 Last nights fault was reported to Wayne who asked for me to
13171950 let him know if it happened again. Called in tonights fault
but they don't want to send it to Wayne and are investigating
the fault themselves. Initially, only USA via Syd affected,
but by 20:55, perth domestic and international showing the
same effect as before.
11:10 Excessive delays international via perth, reported to telstra
13168359 who are investigating it now.
11:10 Office phones, all lines, again dead. Finally got some back,
13166135 called telstra corporate faults - but they're overloaded and
have stuck us on hold!
08:39 Ongoing errors at the new Corryong OptoMux
13165198 Started 3:01am.
14:10 Users of our Corryong facilities should change phone number
FROM the old number: 763000 to 763300 IMMEDIATELY.
11:40 DoS traffic appears to have been directed at the albury annex,
taking it off-line. Admin recalled to site to correct, all
back online and operational 10:58.
17:33 Well, telstra still have not confirmed they have (or have NOT)
13139044 fixed the problem, but all lines seem to have been operational
from about 30 minutes after the problem was reported.
No INDIAL lines were affected, although various customers may
have had THEIR phone lines down too.
09:55 Talking with telstra, seems half our office phone lines are
13139044 down. They have confirmed a major exchange problem and are
working on it. (Two fault nos, 13139014 data, 13139044 voice)
14:01 Out-of-town modem pool (analogue only) is not holding calls.
It's only one pool, seems to be the CentelPlus groups, called
telstra who are testing. (But then, while testing it, it has
come good! Damn!) Problem definately appears to be in the
telstra network as other calls on the same NAS were fine.
16:15 News services restored by telstra.
14:15 Telstra NSWPULL news server is refusing connections.
13093986 Reported to Clive after it's been down an hour, for attention.
13:10 Telstra have confirmed they've identified and fixed the fault.
13068156 Seems one of the Packet-over-SONET links from the USA to AUS
that was ACTIVE, was put into loopback test in error. Strange
they didn't notice or discover it until I reported the fault.
*sigh*. World class carriers indeed.
11:45 Reported to Bruce Mccoll @ telstra. Admits there have been
13068156 problems with sydney routers all this week. Is checking now.
6 GigabitEthernet4-0.win-core1.Melbourne.telstra.net (188.8.131.52) 25.888 ms
7 Pos2-0.ken-core4.Sydney.telstra.net (184.108.40.206) 40.231 ms
8 GigabitEthernet3-1.pad-core3.Sydney.telstra.net (220.127.116.11) 38.760 ms
9 Pos2-3.pad-core3.Sydney.telstra.net (18.104.22.168) 309.558 ms
10 Pos2-3.pad-core3.Sydney.telstra.net (22.214.171.124) 803.851 ms
11 Pos2-3.pad-core3.Sydney.telstra.net (126.96.36.199) 1331.665 ms
12 Pos2-3.pad-core3.Sydney.telstra.net (188.8.131.52) 1879.476 ms
13 Pos2-3.pad-core3.Sydney.telstra.net (184.108.40.206) 2398.007 ms
14 Pos2-3.pad-core3.Sydney.telstra.net (220.127.116.11) 2935.739 ms
09:14 Advised Telstra Outage - Telstra Albury Router
The router ALB3 has a faulty card being replaced
at 19:45 hrs AEST on 15/06/2001.
Services connected to this router will be interrupted
for approximately 5 minutes during this activity.
08:00 We plan to shutdown the Albury Tigris for approx 5 minutes
on 15-Jun-2001 at approx 8am in order to install an extra
60 port digital modem card. Should cause minimal disruption
15:55 Albury Annex Dropped DTR. On Site - immediate reset
Reset and back to normal 16:03
18:03 Slow response to the USA - Telstra are working on it (since
13025368 17:40) - sudden increase in latency to the USA about 1700mS
20:30 Albury Annex has dropped DTR on all modems again. Arghh!
Staff recalled to site, server operational again 20:50.
17:41 Albury Annex has dropped DTR on all modems.
Staff were still on site, reset immediately, operational
10:34 Maro from ISP Faults centre has done a complete system reset
13014543 of the NT1 and interface to force a reset of the OR30 system.
Now running no errors, which is a big improvement.
10:00 Still running slip errors. (Badly....)
13014543 Spoke to Hans, then on to Maro who has done some testing,
confirmed it's out of Albury Exchange (ALBG88). Layer 1 up.
Layer 2. Telstra will send a tech.
11:30 Chris called back, seems that fault cause was "Unknown" and
13014543 no further action will be taken. I've STRONGLY requested they
investigate further tomorrow to actually identify a cause
BEFORE they close the fault.
10:36 Cate Valon, Corp faults, Sydney totally useless GIMP, had
13014543 absolutely no idea about anything, finally transferred me to
the 1800-624-512 number we're NOT allowed to call as that is
for bigpong customers, WE have to call the ISP fault centre
(which is what I did, and got this ignorant cow in the 1st place!)
Chris at big pond says he can see from the logs their end
SES errors (section 1, ie: within telstra network)
Dip fault (power feed) Confirmed failure 10:13am
Is investigating and will call me back.
Fault is that ALL lines to alb-tig1 disconnected ALL users.
17:10 Telstras news server is refusing connections again (and has
been since 16:50). Of course, it's a "non critical" system
and won't be attended to until next week. I've reported it
so at least they are aware of it. Who knows, perhaps they
will fix it soon anyway?
02:50 Telstra advises:
20010506-0250-01 There is a problem with the trunk network circuit from Sydney
to Tokyo, Sydney to Taipei, Sydney to China, Perth to Palo
Alto (USA). This may be causing degraded performance to
Internet traffic between these regions.
Telstra have not given any restoration time, it is possible
that the slow international response still being experienced
(from 9am onwards) is a direct result.
16:10 Finally, our phones seem to be fixed. Seems some GIMP at
telstra had pulled some wrong jumpers in the exchange!
11:10 Reported to Maro - our primary office numer (60402692) is
12933544 dead again - side tone but no dial tone. Customers reporting
phone ringing out.
14:15 We're down from 5 phone lines into the office to just ONE.
12931262 Have called fault in to telstra who seem to think it's a
line fault and they will have to get a liney out. Could be
a couple of days before someone can get here. I'm escalating
it as far as I can to expedite it.
10:00 Bandwidth to our Wangaratta POP was doubled at 10am today.
01:00 Telstra have advised there will be 5-minute interruption to
P36204V service for all our MtBeauty and Wangaratta facilities on
15-May somewhere between 00:01-03:30.
13:00 The time has come - our old Corrying server has been turned
OFF now and has been de-commissioned.
11:50 News server is back up.
08:55 Tony from Telstra reports that yes, the Sydney news server
is indeed down. It's a non-critical service so no one will
look at the problem until Tuesday.
18:55 Telstra's Sydney NNTP (Usenet News) server is down.
12884843 Reported to Frank.
19:55 Alb-anx1 stopped responding. Tech recalled to site 20:00,
system restarted 20:03, back on-line 20:08 and taking calls.
17:00 Our new PoP in Corryong is ready for users to start testing.
56K class connections have finally made it to Corryong. For
full information, users affected by this change, have been
sent details by e-mail.
21:40 Repeat of last nights abysmal performance (1400mS) to first
12818319 hop to the USA via Sydney. (Fault from last night still open)
21:50 Meiling returned my call. All telstras capacity through sydney
is at or near 100%. No capacity via perth due to the (still)
cut china cables. She will call David Woodgate now to see if
they can bring on additional capacity, or loadbalance better.
Otherwise, there's nothing they can do. Expect more of the
same for the next 3 weeks or so until china cable is fixed.
21:08 Excessive ping times to the USA (>1500mS). Reported to John
12818319 telstra faults. Been like this for the last 35 minutes.
14:20 Power finally restored, although major works are required
next week to replace the main switchboard and various of
the control, generator-changeover and phase-fail sensors.
At least we're back off generator after 13 hours.
09:30 Secondary site still running on generator. Widespread damage
from this mornings storm. Power board is blown off the side
of the building, electricians are in attendance, but power
is not expect back for many hours. Quite a few bits of our
equipment have sustained damage, it appears that it was either
a direct lightning strike, or the 22KV power has been brought
down into contact with the LV supply.
01:25 Secondary site power fail. Whole area is without power, we
are running on generator.
01:20 This is unbelievable. Both Mt Beauty and Wangaratta gone for
a 3rd time. Heads will surely have to roll....
01:06 Not before damn time, looks like tel$ux have finally got the
lines back up after re-resetting the exchange or lines.
Still no idea what the hell caused the problems and basically
denying any liability.
00:04 Both Mt Beauty and Wangaratta gone AGAIN! RE-reported.
23:36 Both Wang and MtBeauty servers back, although no advice from
telstra. Nicole (tester, melbourne) advised 10 mins ago that
it was a major fault at the wang exchange, ETR 30-60 mins.
Compensation paperwork to be sent from Customer fault
management tomorrow re all the lost user records.
22:50 Wangaratta completely disappeared.
12802112 Reported to corporate faults.
22:50 Mt Beauty completely disappeared.
12802111 Reported to corporate faults.
13:30 Replacement equipment has been sourced, both ends of our
microwave link have been replaced with borrowed equipment.
Chartered aircraft will have replacements here by 4pm.
11:30 One Microwave unit has failed. Attempting to affect repair
or replacement ASAP. Secondary DNS, camera and fax gateway
19:13 John at telstra advised massive delays at Sydney AGAIN for
12789016 all international traffic. Traceroute to microsoft, or any
USA site I've been able to check is the same:
8 GigabitEthernet3-1.pad-core3.Sydney.telstra.net (18.104.22.168) 48.102 ms 38.217 ms 44.713 ms
9 22.214.171.124 (126.96.36.199) 1722.288 ms 1680.338 ms 1718.552 ms
10 f0.pao1.verio.net (188.8.131.52) 1665.194 ms 1713.789 ms 1701.922 ms
10:55 5 minute interruption to service to re-install new UPS for
our Wangaratta facilities. All customers connecting to our
Wangaratta POP will have experienced a 5-minute outage.
09:00 Telstra news is unreachable since 5:25am, dies at
20:15 Chris at bigpond has officially recognized (eventually) that
12744629 there is indeed a major routing problem australia wide, has
gone to report and get action.
19:56 Damn telstra to the 7th level of hell! They have a broken
router in albury has died, yet none of the faults numbers
13:23 Traffic DEAD yet again at
Reported to telstra, seems we're the first to report it.
12:45 News server up again.
12:20 Telstra NNTP (Usenet) server down. Reported to Darren.
19:45 The Annex terminal server has stopped responding.
Technician recalled to site, operational again 19:57
13:02 Most national and all international connectivity failed at
12731333 sydney. Reported to Larry @ telstra. Confirmed fault, they
are working on it now.
15:59 Wangaratta site sync lost on OR30 yet again. Reported to
12684785 telstra. Terry. Seems the 4th-Feb-2001 fault HAS been left
open on QMR (Quality Monitoring), recalling someone now.
Link restored 40 seconds later.
17:20 Power restored at Corryong, site back up and running.
16:05 UPS and remote site shutdown to preserve batteries.
15:35 Power to Corryong has been lost. TXO advises of a widespread
failure affecting the whole area, restoration not expected
until 18:00 tonight.
15:35 Power to Corryong has been lost. TXO advises of a widespread
failure affecting the whole area, restoration not expected
until 18:00 tonight.
15:43 General national routing instability, reported to Peter.
12705602 "Complete network avalanche failure" - the core links having
failed seems to have just screwed the entire national network
and they currently have no idea on where or why, or when it
will all be fixed.
14:20 A sand dredging ship has cut a submarine cable off
Singapore. No ETA on fix yet.
Performance to sites outside Australia may be poor.
10:30 Very high latency Perth -> Palo Alto. In excess of 1500ms.
12703455 International network performance sub optimal. Reported
16:30 High latency PER -> SFO.
12693844 Telstra advised egress links at 96% capacity, new capacity
coming online "soon".
13:14 Wangaratta site has returned. No fault at our equipment,
12684785 logged yet another fault with telstra. Sooner or later
I hope somebody will give up and actually FIX it!
13:10 Wangaratta site has disappeared. Suspecting storm/power
problems affecting telstra, calling them now.
15:35 A small technical "glitch" during software testing resulted
in interrupted core connectivity for a few minutes. No data
loss or loss of customer connectivity resulted.
17:10 Telstra have been unable to find any faults with the OR30 in
Wang and closed the fault. If it does happen again, they'll
put an analyser on the line and monitor it more closely.
09:38 Wangaratta OR30 failed yet again. Fault reported to telstra.
12658655 We'll see if they actually FIX it this time.
18:40 Telstra technician reports ALL onramp services in wangaratta
have been flapping up and down from the same time period,
they have no evidence of what the exact problem is/was, but
have blamed it on the number and severity of local storms.
17:43 Wangaratta OnRamp service seems to have become faulty.
12658333 Reported to telstra, they are recalling a tech to test/fix
it, but no ETA at this stage. The Onramp seems to be bringing
the D channel up and dropping it every 30-90 seconds :(
21:45 Some residual impact from the server failures earlier today, some
people with only one DNS configured were unable to resolve some
names. Tracked to being a perculiar issue with named not binding
to an aliased port, full operation should now be restored.
18:00 We've sucessfully managed to return the main UPS to service
with zero downtime due to some "creative" cabling solutions.
16:01 Our APC UPS last reserve for our main servers went BANG without
any notice, took all our servers out and spiked our primary UPS
which went into a latched bypass mode. We have returned all servers
to service after approx 7 minutes, however a later shutdown MAY be
required to return our main UPS to normal operation. We are still
investigating zero-downtime alternatives to bringing back normal
Our games server has a developing hardware failure. While the server
is currently operational, we are planning a maintenance shutdown
for tomorrow to replace it, and will take the oportunity to also
perform some other upgrades to the system.
00:30 Telstra advise an exchange software update will be done on our
exchange somewhere between midnight and 1am today, which should
create approximately 5-minutes of downtime.
This will affect all Corryong, Wangaratta and MtBeauty users,
and those Albury users calling our digital numbers.
15:44 Appears there are problems with Telstra services in Corryong.
12615310 Many customers reporting inability to get connected, along with
"Harry" general faults involving PSTN services (NOT only Internet facilities).
A fault has been lodged with Telstra.
02:30 Telstra advise "Urgent Transmission Work" is to take place today
sometime between midnight and 5am. An outage of approximately 2
minutes is expected. Customers calling our Mt Beauty, Corryong
and Albury digital facilities may be affected.
15:00 A brief outage at our Wangaratta POP (approx 10mins) will occur
to upgrade power supply systems.
05:15 Wangaratta power restored.
04:10 Power outage in Wangaratta. Seems fairly widespread. Customers in
Wangaratta, Killawarra, Milawa etc affected.
16:20 An unscheduled reboot of our Annex terminal server was required
to correct an operational issue (routes not being added).
Approximately 5 minutes downtime was experienced by users from
Corowa/Rutherglen, Culcairn, Holbrook and surrounding areas.