Historical status reports for 2001
26-Dec-2001 17:08 Merimbula Power restored finally ! Country Link Energy could only tell me that someone had taken out a power pole in the CBD which knocked out all Power in Merimbula. Tech also called on site to check servers. Systems checked remotely and all services back to normal. 26-Dec-2001 16:00 Merimbula Completely DOWN, Major Power fault in Merimbula causing all of the CBD to be taken out. UPS went as far as it could, but had to give up sooner or later. Waiting for Country Energy to call back with ETA for Power to be restored. 26-Dec-2001 09:05 When the revolution comes, this damn annex will be the first one up against the wall. Failing to respond, not pinging. Tech called to site to fix. This one is earmarked for replacement early in the new year. 23-Dec-2001 14:30 Annex had a slight hickup. Called on-site to rectify. Once again, systems fully operational 22-Dec-2001 06:58 Arghh! The Annex was responding, but not well. High pings, no DTR asserted, a few users logged in but not doing much. Tech attended site, graceful restart has cured things for now. 19-Dec-2001 16:48 Ping times from Perth to USA way to high 1000ms to 2000ms 13556666 Telstra called, it's on their network, so we will wait and see how long it takes before someone can address it 16-Dec-2001 11:03 At long bloody last. I've determined that our dialup pools were 13546453 able to route ok (203.17.234.0/23) but not our main server pool on 203.15.244.0/24. William from TIS has added a new agregate 203.15.244.0/23 to alb1, which is a higher preference than the damped /24, thus restoring connectivity, will remove it in an hour or so and things should be ok. This is just unacceptable..... 16-Dec-2001 10:26 Called back, it's now like 2 HOURS, still no word from telstra. 13546453 Evan advises they "had a major BGP fault, and they have about 50 faults open at the moment", they thought they had the problem fixed, but evidently not. Still waiting on information. 16-Dec-2001 08:15 Wierd routing problems yet again between alb1 and alb3 at 13546453 telstra, cannot get onto 1800-066-594 (rang out repeatedly) and the so-called ISP fault centre 1800-624-512 go thorough to corporate faults who have ZERO CLUE about TCP/IP. These INCOMPETENT FOOLS have no idea how to run a network OR a fault centre. Evan. traceroute to irc.sux.com (203.31.238.2), 30 hops max, 40 byte packets 1 albury-core.albury.NET.AU (203.15.244.2) 5.074 ms 2 Serial4.alb1.Albury.telstra.net (139.130.63.29) 13.736 ms 3 * 4 * 5 GigabitEthernet3-0.lon-core3.Melbourne.telstra.net (203.50.80.1) 41.074 ms 6 * 7 * Cannot even *RESOLVE* most domain names, traceroutes to our USA servers same as above, die. FINALLY got someone to answer the phones at 9:23! 12-Dec-2001 15:20 Alan at telstra is looking into this fault, re slow ping times 13537371 to the USA via perth, and to perth directly. International RTT getting progressively worse. Seems to have started at or about 4:50am this morning when something changed in sydney, but didn't really get worse until about 11:30am 10-Dec-2001 06:45 Finally back on-air. Not bad for a maximum 30minute outage! 10-Dec-2001 06:30 Despite a known outage (20011210-0600-01) declared to be from 0600hrs AEDT on 10/12/2001 for 30 minutes, affecting alb3, this activity appears to have commenced about 05:40, yet at 06:35 there is STILL no connectivity. Called in as a fault, seems their engineer is still working on it, will notify as soon as they know when it will be fixed. 08-Dec-2001 22:14 Telstra have just advised (Sat, 8 Dec 2001 21:42:23 +1100) that there will be a 30-minute interruption to our service: "The router alb3.Albury will be rebooted at 0600hrs AEDT on 10/12/2001 in order to rectify a fault." I believe this actually translates to "we will be switching over to the new 16 megabit ATM bearer instead of the 3x2M circuits currently feeding this router". 07-Dec-2001 17:20 Telstra have confirmed the earlier outage was caused by them 13524343 stuffing about with a new ATM link, the installation of which screwed up the albury-melbourne link. They have more playing about to do, and are expecting at least one more outage of up to 30 minutes duration sometime between now and 1am tomorrow. 07-Dec-2001 16:20 Reported fault at telstras albury POP. Trying to get someone 13524343 to investigate and fix it promptly. 1 203.15.244.2 3.824 ms 6.371 ms 6.477 ms 2 139.130.63.29 6.255 ms 6.486 ms 8.095 ms 3 * * * 4 * * 139.130.63.29 5.107 ms !H 5 * * * 6 * * * 7 139.130.63.29 8.019 ms !H * * 8 * * * 9 * * * Fault seems to be intermittent cleared 16:26, re-ocurred 16:33 05-Dec-2001 08:36 While telstra have still been unable to confirm everything is 13513050 fixed, things APPEAR to be back to normal (whatever normal is!) Hopefully there will be no subsequent problems, but rest assured if there are, we'll be onto it quick smart. 04-Dec-2001 20:58 For goodness sake.... finally got onto hel$tra again and they can only advise that this is a major, serious break (it is also affecting Adelaide aparantly), and is as a result of fibre optic cable number 565, broken due to lightning strike (??) 23Km south of Yass. Still no ETA on completion of repairs. (Funny the lightning tracking system shows NO lightning activity in that area anywhere near the time in question!!) 04-Dec-2001 17:16 Clive from telstra has advised that there is indeed a MAJOR network fault, and Albury, Wodonga and Wagga are all isolated from the rest of Australia. They have "lost" 4 x 565 transmission bearers and have no estimated time for restoration of services. 04-Dec-2001 17:07 Telstra have said there are no faults, then decided its the 13513050 storms in NSW (not withstanding they are 1000Km north of us!) and in general seem completely unable to comprehend that there might even be a problem. Still unable to contact telstra countrywide here, mobile services still down. 04-Dec-2001 16:27 There are MASSIVE phone problems in Albury area, most of our ISDN and phone lines are down, we cannot call telstra faults on any of the fixed service lines, nor can we get out on the telstra mobile network. Managed to get through to telstra on an Optus mobile phone 16:40, NO information to hand yet. 03-Dec-2001 04:15 Albury Annex is showing very slow network response (300+mS). Tech was unable to correct it remotely, attended the site at 06:15 and has restored nominal operation. 22-Nov-2001 08:01 Severe packet loss and excessive latency in the USA affecting some parts of USA and possibly other international connectivity. 15 p1-0.lsanca2-cr1.bbnplanet.net (4.25.111.1) 193.947 ms 205.726 ms 192.208 ms 16 * * * 17 * * p4-1-3-0.r00.lsanca01.us.bb.verio.net (129.250.2.205) 412.153 ms Reported to telstra with traceroutes in both directions. 16-Nov-2001 16:37 Telstra Proxy in BAD shape. Very slow. Telstra called 13459475 twice to finally get some action. 2001/11/16 15:39:40| Detected DEAD Parent: proxy.cache.telstra.net/3128/3130 12-Nov-2001 08:41 Power restored, Corryong site back on line. 12-Nov-2001 07:40 Extensive power fail in the Corryong area, no indication on when power may be restored. Our Corryong site is currently off air (UPS exhausted at 8:07) 10-Nov-2001 15:10 Again (2nd time in as many weeks) mobilenet.telstra.net is 13440455 unreachable, thus no SMS messages getting out. Reported to telstra through the ISP faults centre but got someone without ANY CLUE who is going to try and track someone down (perhaps). bash-2.03# traceroute -q1 tmd-liv1.mobilenet.telstra.net 3 FastEthernet0-0.alb3.Albury.telstra.net (139.130.240.36) 5.803 ms 4 Serial5-1-7-1.lon-core2.Melbourne.telstra.net (203.50.76.85) 18.400 ms 5 FastEthernet1-0.lon29.Melbourne.telstra.net (203.50.80.39) 20.658 ms 6 * 7 * 09-Nov-2001 12:00 terran, one of the servers which hosts our users virtual websites, reported 8 transient disk errors overnight. The source has been traced to one 9Gb Quantum Viking SCSI drive. Whilst the errors were recoverable, admins have removed the drive from service and all content is now running from a Seagate Barracuda 9Gb LVD SCSI3 drive. Unfortunately, this required interruption to some virtual services for a period of approximately 90 seconds. No data lost. 03-Nov-2001 18:36 Bega site restored. Work on building STILL being done. Looks like someone has cut power during work. 03-Nov-2001 18:05 Lost Power to Bega POP, calling out tech to see to problem 31-Oct-2001 8:04 Bega Pop failed to Come alive after work on power at Bega overnight. Tech sent to Bega site to restore systems. Operation back to normal. 30-Oct-2001 17:02 Yackandandah indial number ringing unanswered. Reported to 13409720 Chris at ISP faults. He's checked and can't see anything wrong so has called it in as a fault. 30-Oct-2001 11:06 Bega POP Scheduled outage. 8.30pm to 9.30pm Tuesday 30 Oct 2001. Work being completed to Power in building at Bega site. 29-Oct-2001 13:50 Corryong tigris has been spitting this message out about every 22 seconds: 13404788 *** TRAP from local agent at 29-Oct-2001 12:54:55 uptime 3 Days, 03:10:37 *** DM: Datalink error (unsolicited supervisory response), J7.3 Has been doing it for at least the last 3 hours. Reporte to Anthea at the ISP faults centre. 1Hr response quoted. 24-Oct-2001 08:50 Telstra (again) have significant delays between lon-core2 13391657 and lon-core3 routers in melbourne (a gigabit ethernet!) (Anthea took the call, seemed somewhat unconcerned about it, assured me there were no "known problems" in melbourne. 21-Oct-2001 05:00 There will be a brief interruption to connectivity at about 13378026 5:00am, Sunday morning, when telstra replace a faulty card in their Albury router. Should only take 10 minutes, but we are allowing for a 1-hour outage for this work. 18-Oct-2001 14:50 Link has been slow between telstras albury and melbourne routers. 13378026 3 FastEthernet0-0.alb3.Albury.telstra.net 5.477 ms 4 Serial5-1-7-1.lon-core2.Melbourne.telstra.net 396.481 ms Called in as a fault, ISP fault centre not answering phone, called 066594 and "tony" got all bent out of shape because I didn't call the ISP number. On hold 20 mins in the end but no real effort to get it fixed. 14-Oct-2001 21:38 Truly amazing. NOW they admit there is actually a problem, 13364417 after denying it for hours. They've sent a technician out to investigate/fix. No ETA, but who cares? 14-Oct-2001 20:32 Called telstra back, spoke to Cliff. Still NO WORD from anyone 13364417 about what is (or is not) happening or being done. He has now PROMISED to call me back shortly with an update. Strewth, these guys are HOPELESS! 14-Oct-2001 16:49 Sydney and International connectivity getting progressively 13364417 worse over the last 2 hours. Called Telstra (Andrew). They were quite unaware of any problem, have taken a fault report and should call back soon. 26-Sep-2001 16:22 We have had no advice from telstra, but from the significantly improved performance over the last 12 hours or so, presume the replacement bandwidth has finally come on-line. 24-Sep-2001 21:50 Sydney ken-pad latency issue seems fixed, although as usual 13313756 no word from telstra to say if it's permanent, transient, really fixed, a fluke or of course, what caused it. Perhaps they'll dane to tell us tomorrow. 24-Sep-2001 21:12 Useless GITS at telstra still can't tell me what is going on with the problems at sydney, even though several other ISPs are all reporting via IRC the same problems. Re-reported the fault, perhaps someone will look at it in the next month! 24-Sep-2001 18:40 Reported fault to Nicole @ telstra faults, major delays on 13313756 the gigabit ethernet between ken-core4 in sydney and pad-core3 in the order of 200mS additional delay. 21-Sep-2001 19:05 Darren from Telstra just called. Onramp service had problems with Earth. Telstra on site and restored problem. Moruya Batemans Bay service now back on line 21-Sep-2001 16:59 Maro from Telstra called re Moruya link. Telstra found 13308537 errors running on the On Ramp service. Looking into the fault and will advise. At this stage until Telstra rectify the fault, dial up services at Moruya Batemans Bay will NOT be available 21-Sep-2001 16:05 Moruya Link down, Telstra contacted, suspect issues with 13308537 On Ramp service. Waiting for Telstra to call back. 21-Sep-2001 11:30 Well, contrary to telstras earlier notice, there are more problems with international capacity. A submarine cable between Australia and the USA has been badly damaged near China, killing 60% of telstras international capacity. They are tying to rush an extra 622Mbps of capacity through SCC, but are not expecting to have it on line until next week. We're sorry for any inconvenience, but it's beyond our control. 21-Sep-2001 00:00 International traffic situation APPEARS resolved. 20-Sep-2001 19:46 Still can't get any connectivity past 202.84.249.137 into the 13305671 USA. Reported as a fault (not just conjestion). Telstra are still saying they are trying to get alternate capacity to the USA until the singapore-USA cable cut can be fixed. 20-Sep-2001 17:00 Yes, all connectivity to international sites is WAY slow at the moment. Telstra have a problem in Perth, and are re-routing all traffic via Sydney, which has brought the whole network to its knees. They are aware of the problem and are working on it. No restoration time yet available. 17-Sep-2001 08:56 DDoS has returned as soon as the filters were removed. 13291117 Seeing 617 pkts/sec, 440922 bytes/sec of UDP traffic. Tony is creating a UDP filter instead of a total block. ACL169 on lon core2. Applied to all interfaces to albury. Tony is still working on tracking the packets (9:10am). Will call me back shortly with progress updates. 17-Sep-2001 08:52 After several more calls, Tony Van Reit is working on and says he has fixed the filter on lon-core2, which should fix the problems with connectivity on giroc. He also said that the "ACLs" on albury have nearly brought that router to its knees with additional CPU. 17-Sep-2001 07:45 Seems last nights "block" by telstra was over-zealous. They have totally blocked 203.15.244.13/32 to ALL packets at FastEthernet1-1-0.lon-core2.Melbourne.telstra.net (203.50.80.4) Asked for urgent recall to correct the filter. 17-Sep-2001 01:41 Telstra finally called back to advise they have completed the filters and to check things are all working ok now. 17-Sep-2001 00:27 Called Steve @ telstra yet again, he's calling to find out what progress is. After 8 mins on the phone to Steve he still can't tell me anything more. It appears to be not only a DDoS attack, but something else as well. They have called in everyone they can, but can't give any idea of ETR. 16-Sep-2001 23:58 There's a glimmer of light at the end of the tunnel. No more DDoS observed for a couple of minutes, external connectivity is back as far as melbourne now. 1 albury-core.albury.NET.AU (203.15.244.2) 4.748 ms 2 Serial4.alb1.Albury.telstra.net (139.130.63.29) 47.987 ms 3 FastEthernet0-0.alb3.Albury.telstra.net (139.130.240.36) 260.799 ms 4 Serial5-1-7-1.lon-core2.Melbourne.telstra.net (203.50.76.85) 269.162 ms 5 * 6 * Still no word back from telstra yet. Drat. Spoke too soon. Just as I'm about to post this, looks like routing and connectivity are screwed again! 16-Sep-2001 23:10 Looks like telstra may have restored their router, but we are now under a massive DDoS, in excess of 350Kbytes/Sec via UDP directed at one of our hosts (giroc). Telstra tech should be calling me shortly. 16-Sep-2001 22:48 Called telstra back again (Steve) for update. They are (or are supposed to be) in the process of applying filters. Been on hold 10 mins, no update yet (still)! They've had to call in one of the remote techs. Now estimating a further 15 minutes. (ie, 23:07) 16-Sep-2001 22:36 Reported to another party (AT, Sydney) by John at telstra, that there is a ping-flood on the albury telstra router, and telstra are putting filters on there now. ETR: 10 mins. 16-Sep-2001 22:15 We seem to be disconnected from the internet due to a fault 13291117 at the telstra albury router. traceroute to dn.albury.net.au (209.207.141.237), 30 hops max, 40 byte packets 1 albury-core.albury.NET.AU (203.15.244.2) 2.123 ms 2 Serial4.alb1.Albury.telstra.net (139.130.63.29) 6.452 ms 3 * * * 4 * * * 5 * * * 12-Sep-2001 14:54 Modem Card on Corryong site overheating causing issues with dial up access. Admin restored dial up services promptly. 09-Sep-2001 20:36 203.58.128.0/24 De-routed to reduce SOME of the codered crud. 04-Sep-2001 06:20 Annex stopped responding. Admin recalled to site to fix. 26-Aug-2001 16:00 Excessive delays to USA via Sydney (700mS). 13234482 Called in to Dale at telstra - problem appears to have gone away now (evident from 15:30 to 16:00 approx) 24-Aug-2001 16:44 Merimbula Link reset and services restored to Normal 24-Aug-2001 15:00 We are seeing rather large ping times to across to Merimbula 13232958 from Albury causing some delay across the link. Report made to Mat at Telstra to investigate. The issue appeared at the same time Telstra worked on our On-ramps at the Albury Exchange Thursday morning 23.8.2001. Also experiencing some packet loss. 24-Aug-2001 08:36 Albury Annex required a reset. Some services affected from 08:00 to 08:35 this morning. 23-Aug-2001 11:15 Corryong chassis replaced, all services back on-line, checked and all services nominal. 23-Aug-2001 10:30 We will have a brief service outage at our Corryong POP for approx 15 minutes on Thursday morning to facilitate the exchange of a faulty server chassis. 19-Aug-2001 10:08 Found nearly 100 sendmail processes running on secondary mail server - mostly for junkmail from "howareyoutoday.org" to bunge.com (whos mail server was off-line). Deleted all the junk, killed all the sendmails, everything cool. Also added howareyoutoday.org to the access file to prevent a repeat. 14-Aug-2001 09:22 Connectivity problems from albury to melbourne, started 8am. 13201652 12-Aug-2001 17:21 Corryong tigris modem card has developed a fault. Replaced 17:20 12-Aug-2001 06:17 Albury Annex stopped responding. Sysadmin paged, attended and reset it, everything operational again. 08-Aug-2001 09:11 (Originally fault number 13165198) - Spoke to Trevor and they 13182625 changed ports on the albury AXE at 07-Aug-2001 19:22, we have had NO FAULTS on the line since then. Appears that the real cause has been the faulty port, but that there were also bad or marginal (a) radio link (changed to fibre), (b) NT1 (replaced) (c) optomux port (swapped), (d) Krone block (replaced). 06-Aug-2001 13:30 News is restored. 06-Aug-2001 12:09 Telstra have advised that their "upgrade" of the victorian news server have not gone to plan, and news is currently down. They say only: "Services connected to this server will be interrupted for approximately 3 hours while this is being investigated." 05-Aug-2001 20:20 Re fault 13168359. Reported to Chris @ telstra faults, who is going to talk to William who will in turn contact Wayne to get the low-down on the fault and its history etc. (Sheesh, no wonder nothing ever gets done there!) (Between wel-core3 and wel-gw1). Called back by William 21:10 - fault "disappeared" again, without resolution still. (He accidently took down the interface and when he brought it back up, the problem had gone....) 03-Aug-2001 14:14 Telstra techs have replaced the NT1 card at the corryong site as there were still hits on the line since changing over the bearer from radio to fibre. 03-Aug-2001 13:16 Telstra NDC techs have changed the bearer from microwave link from corryong exchange back to albury, onto a fibre link, in a bid to fix the clock-slips that we've been experiencing. 02-Aug-2001 19:30 Further reference to fault 13168359 - so much for "making it easy" or "world class". What a joke! It's taken an HOUR to get someone to answer the after hours fault number, and then nobody is interested in DOING anything about the fault. Bad latency to perth AGAIN (up from 70 to 280+ milliseconds). I'm PROMISED a call back from a recall person shortly. 02-Aug-2001 17:50 Word back from telstra is that they believe they have found 13165198 a faulty krone block at the corryong exchange, which is being changed out, along with moving our 2 megabit bearer from the radio link to fibre (as it should have been from day 1!) We expect a 5-minute interruption tomorrow while changes are made. 02-Aug-2001 13:20 Have spoken to Andrew Cotterill, general manager, albury, 13165198 countrywide, who has undertaken to find out what the hell is going on, and when it will be fixed and what the delays have been (with reference to the Corryong problems) 01-Aug-2001 10:48 Called telstra yet again (about the 12th time since this issue 13165198 started) spoke to Anthea who PROMISED a return call within an hour. Still no reply by midday, called Matt at midday, he is reading the original notes from Mick. He's calling the wideband group again now to find whats going on. 01-Aug-2001 20:38 Last nights fault was reported to Wayne who asked for me to 13171950 let him know if it happened again. Called in tonights fault but they don't want to send it to Wayne and are investigating the fault themselves. Initially, only USA via Syd affected, but by 20:55, perth domestic and international showing the same effect as before. 31-Jul-2001 11:10 Excessive delays international via perth, reported to telstra 13168359 who are investigating it now. 31-Jul-2001 11:10 Office phones, all lines, again dead. Finally got some back, 13166135 called telstra corporate faults - but they're overloaded and have stuck us on hold! 31-Jul-2001 08:39 Ongoing errors at the new Corryong OptoMux 13165198 Started 3:01am. 30-Jul-2001 14:10 Users of our Corryong facilities should change phone number FROM the old number: 763000 to 763300 IMMEDIATELY. 29-Jul-2001 11:40 DoS traffic appears to have been directed at the albury annex, taking it off-line. Admin recalled to site to correct, all back online and operational 10:58. 20-Jul-2001 17:33 Well, telstra still have not confirmed they have (or have NOT) 13139044 fixed the problem, but all lines seem to have been operational from about 30 minutes after the problem was reported. No INDIAL lines were affected, although various customers may have had THEIR phone lines down too. 20-Jul-2001 09:55 Talking with telstra, seems half our office phone lines are 13139044 down. They have confirmed a major exchange problem and are working on it. (Two fault nos, 13139014 data, 13139044 voice) 06-Jul-2001 14:01 Out-of-town modem pool (analogue only) is not holding calls. It's only one pool, seems to be the CentelPlus groups, called telstra who are testing. (But then, while testing it, it has come good! Damn!) Problem definately appears to be in the telstra network as other calls on the same NAS were fine. 03-Jul-2001 16:15 News services restored by telstra. 03-Jul-2001 14:15 Telstra NSWPULL news server is refusing connections. 13093986 Reported to Clive after it's been down an hour, for attention. 23-Jun-2001 13:10 Telstra have confirmed they've identified and fixed the fault. 13068156 Seems one of the Packet-over-SONET links from the USA to AUS that was ACTIVE, was put into loopback test in error. Strange they didn't notice or discover it until I reported the fault. *sigh*. World class carriers indeed. 23-Jun-2001 11:45 Reported to Bruce Mccoll @ telstra. Admits there have been 13068156 problems with sydney routers all this week. Is checking now. 6 GigabitEthernet4-0.win-core1.Melbourne.telstra.net (203.50.77.18) 25.888 ms 7 Pos2-0.ken-core4.Sydney.telstra.net (203.50.6.165) 40.231 ms 8 GigabitEthernet3-1.pad-core3.Sydney.telstra.net (203.50.6.190) 38.760 ms 9 Pos2-3.pad-core3.Sydney.telstra.net (203.50.126.45) 309.558 ms 10 Pos2-3.pad-core3.Sydney.telstra.net (203.50.126.45) 803.851 ms 11 Pos2-3.pad-core3.Sydney.telstra.net (203.50.126.45) 1331.665 ms 12 Pos2-3.pad-core3.Sydney.telstra.net (203.50.126.45) 1879.476 ms 13 Pos2-3.pad-core3.Sydney.telstra.net (203.50.126.45) 2398.007 ms 14 Pos2-3.pad-core3.Sydney.telstra.net (203.50.126.45) 2935.739 ms 15-Jun-2001 09:14 Advised Telstra Outage - Telstra Albury Router The router ALB3 has a faulty card being replaced at 19:45 hrs AEST on 15/06/2001. Services connected to this router will be interrupted for approximately 5 minutes during this activity. 15-Jun-2001 08:00 We plan to shutdown the Albury Tigris for approx 5 minutes on 15-Jun-2001 at approx 8am in order to install an extra 60 port digital modem card. Should cause minimal disruption 12-Jun-2001 15:55 Albury Annex Dropped DTR. On Site - immediate reset Reset and back to normal 16:03 06-Jun-2001 18:03 Slow response to the USA - Telstra are working on it (since 13025368 17:40) - sudden increase in latency to the USA about 1700mS 04-Jun-2001 20:30 Albury Annex has dropped DTR on all modems again. Arghh! Staff recalled to site, server operational again 20:50. 04-Jun-2001 17:41 Albury Annex has dropped DTR on all modems. Staff were still on site, reset immediately, operational again 17:46. 04-Jun-2001 10:34 Maro from ISP Faults centre has done a complete system reset 13014543 of the NT1 and interface to force a reset of the OR30 system. Now running no errors, which is a big improvement. 04-Jun-2001 10:00 Still running slip errors. (Badly....) 13014543 Spoke to Hans, then on to Maro who has done some testing, confirmed it's out of Albury Exchange (ALBG88). Layer 1 up. Layer 2. Telstra will send a tech. 03-Jun-2001 11:30 Chris called back, seems that fault cause was "Unknown" and 13014543 no further action will be taken. I've STRONGLY requested they investigate further tomorrow to actually identify a cause BEFORE they close the fault. 03-Jun-2001 10:36 Cate Valon, Corp faults, Sydney totally useless GIMP, had 13014543 absolutely no idea about anything, finally transferred me to the 1800-624-512 number we're NOT allowed to call as that is for bigpong customers, WE have to call the ISP fault centre (which is what I did, and got this ignorant cow in the 1st place!) Chris at big pond says he can see from the logs their end SES errors (section 1, ie: within telstra network) Dip fault (power feed) Confirmed failure 10:13am Is investigating and will call me back. Fault is that ALL lines to alb-tig1 disconnected ALL users. 06-May-2001 17:10 Telstras news server is refusing connections again (and has been since 16:50). Of course, it's a "non critical" system and won't be attended to until next week. I've reported it so at least they are aware of it. Who knows, perhaps they will fix it soon anyway? 06-May-2001 02:50 Telstra advises: 20010506-0250-01 There is a problem with the trunk network circuit from Sydney to Tokyo, Sydney to Taipei, Sydney to China, Perth to Palo Alto (USA). This may be causing degraded performance to Internet traffic between these regions. Telstra have not given any restoration time, it is possible that the slow international response still being experienced (from 9am onwards) is a direct result. 04-May-2001 16:10 Finally, our phones seem to be fixed. Seems some GIMP at telstra had pulled some wrong jumpers in the exchange! 04-May-2001 11:10 Reported to Maro - our primary office numer (60402692) is 12933544 dead again - side tone but no dial tone. Customers reporting phone ringing out. 03-May-2001 14:15 We're down from 5 phone lines into the office to just ONE. 12931262 Have called fault in to telstra who seem to think it's a line fault and they will have to get a liney out. Could be a couple of days before someone can get here. I'm escalating it as far as I can to expedite it. 01-May-2001 10:00 Bandwidth to our Wangaratta POP was doubled at 10am today. 15-May-2001 01:00 Telstra have advised there will be 5-minute interruption to P36204V service for all our MtBeauty and Wangaratta facilities on 15-May somewhere between 00:01-03:30. 20-Apr-2001 13:00 The time has come - our old Corrying server has been turned OFF now and has been de-commissioned. 17-Apr-2001 11:50 News server is back up. 16-Apr-2001 08:55 Tony from Telstra reports that yes, the Sydney news server is indeed down. It's a non-critical service so no one will look at the problem until Tuesday. 15-Apr-2001 18:55 Telstra's Sydney NNTP (Usenet News) server is down. 12884843 Reported to Frank. 07-Apr-2001 19:55 Alb-anx1 stopped responding. Tech recalled to site 20:00, system restarted 20:03, back on-line 20:08 and taking calls. 23-Mar-2001 17:00 Our new PoP in Corryong is ready for users to start testing. 56K class connections have finally made it to Corryong. For full information, users affected by this change, have been sent details by e-mail. 21-Mar-2001 21:40 Repeat of last nights abysmal performance (1400mS) to first 12818319 hop to the USA via Sydney. (Fault from last night still open) 20-Mar-2001 21:50 Meiling returned my call. All telstras capacity through sydney is at or near 100%. No capacity via perth due to the (still) cut china cables. She will call David Woodgate now to see if they can bring on additional capacity, or loadbalance better. Otherwise, there's nothing they can do. Expect more of the same for the next 3 weeks or so until china cable is fixed. 20-Mar-2001 21:08 Excessive ping times to the USA (>1500mS). Reported to John 12818319 telstra faults. Been like this for the last 35 minutes. 17-Mar-2001 14:20 Power finally restored, although major works are required next week to replace the main switchboard and various of the control, generator-changeover and phase-fail sensors. At least we're back off generator after 13 hours. 17-Mar-2001 09:30 Secondary site still running on generator. Widespread damage from this mornings storm. Power board is blown off the side of the building, electricians are in attendance, but power is not expect back for many hours. Quite a few bits of our equipment have sustained damage, it appears that it was either a direct lightning strike, or the 22KV power has been brought down into contact with the LV supply. 17-Mar-2001 01:25 Secondary site power fail. Whole area is without power, we are running on generator. 15-Mar-2001 01:20 This is unbelievable. Both Mt Beauty and Wangaratta gone for a 3rd time. Heads will surely have to roll.... 15-Mar-2001 01:06 Not before damn time, looks like tel$ux have finally got the lines back up after re-resetting the exchange or lines. Still no idea what the hell caused the problems and basically denying any liability. 15-Mar-2001 00:04 Both Mt Beauty and Wangaratta gone AGAIN! RE-reported. 12802124 14-Mar-2001 23:36 Both Wang and MtBeauty servers back, although no advice from telstra. Nicole (tester, melbourne) advised 10 mins ago that it was a major fault at the wang exchange, ETR 30-60 mins. Compensation paperwork to be sent from Customer fault management tomorrow re all the lost user records. 14-Mar-2001 22:50 Wangaratta completely disappeared. 12802112 Reported to corporate faults. 14-Mar-2001 22:50 Mt Beauty completely disappeared. 12802111 Reported to corporate faults. 14-Mar-2001 13:30 Replacement equipment has been sourced, both ends of our microwave link have been replaced with borrowed equipment. Chartered aircraft will have replacements here by 4pm. 14-Mar-2001 11:30 One Microwave unit has failed. Attempting to affect repair or replacement ASAP. Secondary DNS, camera and fax gateway are unreachable. 10-Mar-2001 19:13 John at telstra advised massive delays at Sydney AGAIN for 12789016 all international traffic. Traceroute to microsoft, or any USA site I've been able to check is the same: 8 GigabitEthernet3-1.pad-core3.Sydney.telstra.net (203.50.6.202) 48.102 ms 38.217 ms 44.713 ms 9 203.50.126.62 (203.50.126.62) 1722.288 ms 1680.338 ms 1718.552 ms 10 f0.pao1.verio.net (198.32.176.14) 1665.194 ms 1713.789 ms 1701.922 ms 07-Mar-2001 10:55 5 minute interruption to service to re-install new UPS for our Wangaratta facilities. All customers connecting to our Wangaratta POP will have experienced a 5-minute outage. 24-Feb-2001 09:00 Telstra news is unreachable since 5:25am, dies at 12748122 FastEthernet1-0.pad-service1.Sydney.telstra.net 22-Feb-2001 20:15 Chris at bigpond has officially recognized (eventually) that 12744629 there is indeed a major routing problem australia wide, has gone to report and get action. 22-Feb-2001 19:56 Damn telstra to the 7th level of hell! They have a broken router in albury has died, yet none of the faults numbers are answering. 22-Feb-2001 13:23 Traffic DEAD yet again at 12742933 GigabitEthernet3-1.pad-core3.Sydney.telstra.net Reported to telstra, seems we're the first to report it. 22-Feb-2001 12:45 News server up again. 12:20 Telstra NNTP (Usenet) server down. Reported to Darren. 12742623 21-Feb-2001 19:45 The Annex terminal server has stopped responding. Technician recalled to site, operational again 19:57 19-Feb-2001 13:02 Most national and all international connectivity failed at 12731333 sydney. Reported to Larry @ telstra. Confirmed fault, they are working on it now. 18-Feb-2001 15:59 Wangaratta site sync lost on OR30 yet again. Reported to 12684785 telstra. Terry. Seems the 4th-Feb-2001 fault HAS been left open on QMR (Quality Monitoring), recalling someone now. Link restored 40 seconds later. 16-Feb-2001 17:20 Power restored at Corryong, site back up and running. 16-Feb-2001 16:05 UPS and remote site shutdown to preserve batteries. 16-Feb-2001 15:35 Power to Corryong has been lost. TXO advises of a widespread failure affecting the whole area, restoration not expected until 18:00 tonight. 16-Feb-2001 15:35 Power to Corryong has been lost. TXO advises of a widespread failure affecting the whole area, restoration not expected until 18:00 tonight. 09-Feb-2001 15:43 General national routing instability, reported to Peter. 12705602 "Complete network avalanche failure" - the core links having failed seems to have just screwed the entire national network and they currently have no idea on where or why, or when it will all be fixed. 09-Feb-2001 14:20 A sand dredging ship has cut a submarine cable off Singapore. No ETA on fix yet. Performance to sites outside Australia may be poor. 10:30 Very high latency Perth -> Palo Alto. In excess of 1500ms. 12703455 International network performance sub optimal. Reported to Darren. 06-Feb-2001 16:30 High latency PER -> SFO. 12693844 Telstra advised egress links at 96% capacity, new capacity coming online "soon". 04-Feb-2001 13:14 Wangaratta site has returned. No fault at our equipment, 12684785 logged yet another fault with telstra. Sooner or later I hope somebody will give up and actually FIX it! 04-Feb-2001 13:10 Wangaratta site has disappeared. Suspecting storm/power problems affecting telstra, calling them now. 01-Feb-2001 15:35 A small technical "glitch" during software testing resulted in interrupted core connectivity for a few minutes. No data loss or loss of customer connectivity resulted. 27-Jan-2001 17:10 Telstra have been unable to find any faults with the OR30 in Wang and closed the fault. If it does happen again, they'll put an analyser on the line and monitor it more closely. 27-Jan-2001 09:38 Wangaratta OR30 failed yet again. Fault reported to telstra. 12658655 We'll see if they actually FIX it this time. 26-Jan-2001 18:40 Telstra technician reports ALL onramp services in wangaratta have been flapping up and down from the same time period, they have no evidence of what the exact problem is/was, but have blamed it on the number and severity of local storms. 26-Jan-2001 17:43 Wangaratta OnRamp service seems to have become faulty. 12658333 Reported to telstra, they are recalling a tech to test/fix it, but no ETA at this stage. The Onramp seems to be bringing the D channel up and dropping it every 30-90 seconds :( 17-Jan-2001 21:45 Some residual impact from the server failures earlier today, some people with only one DNS configured were unable to resolve some names. Tracked to being a perculiar issue with named not binding to an aliased port, full operation should now be restored. 17-Jan-2001 18:00 We've sucessfully managed to return the main UPS to service with zero downtime due to some "creative" cabling solutions. 17-Jan-2001 16:01 Our APC UPS last reserve for our main servers went BANG without any notice, took all our servers out and spiked our primary UPS which went into a latched bypass mode. We have returned all servers to service after approx 7 minutes, however a later shutdown MAY be required to return our main UPS to normal operation. We are still investigating zero-downtime alternatives to bringing back normal service. 16-Jan-2001 Our games server has a developing hardware failure. While the server is currently operational, we are planning a maintenance shutdown for tomorrow to replace it, and will take the oportunity to also perform some other upgrades to the system. 31-Jan-2001 00:30 Telstra advise an exchange software update will be done on our exchange somewhere between midnight and 1am today, which should create approximately 5-minutes of downtime. This will affect all Corryong, Wangaratta and MtBeauty users, and those Albury users calling our digital numbers. 11-Jan-2001 15:44 Appears there are problems with Telstra services in Corryong. 12615310 Many customers reporting inability to get connected, along with "Harry" general faults involving PSTN services (NOT only Internet facilities). A fault has been lodged with Telstra. 11-Jan-2001 02:30 Telstra advise "Urgent Transmission Work" is to take place today sometime between midnight and 5am. An outage of approximately 2 minutes is expected. Customers calling our Mt Beauty, Corryong and Albury digital facilities may be affected. 16-Jan-2001 15:00 A brief outage at our Wangaratta POP (approx 10mins) will occur to upgrade power supply systems. 09-Jan-2001 05:15 Wangaratta power restored. 09-Jan-2001 04:10 Power outage in Wangaratta. Seems fairly widespread. Customers in Wangaratta, Killawarra, Milawa etc affected. 02-Jan-2001 16:20 An unscheduled reboot of our Annex terminal server was required to correct an operational issue (routes not being added). Approximately 5 minutes downtime was experienced by users from Corowa/Rutherglen, Culcairn, Holbrook and surrounding areas.
