Historical status reports for 2004
17/Dec/2004
00:15 Problem seems "fixed" although still no word from
114053568 telstra. Slight increase in latency to everywhere,
suspect they've changed things significantly.
16/Dec/2004
15:30 Packet loss between Albury and Melbourne on the
114053568 telstra network, causing major dramas for incomming
mail and DNS.
Tried to report for hours but telstra are not even
answering their helpdesk.
Response by 19:25 tonight.
Numerous calls to chase up whats happening, they
are claiming it is a ribbon cable that has failed
that connects the ATM interface.
19:00 had to kill sendmail as the packet loss was
crippling the box (load avgs >40 due to 600+ active
mail processes). Tried restarting it several times
but problem instantly re-appeared. Once packet loss
was down to about 4% it was acceptable and mail
was working fine from about 23:30.
05/Dec/2004
09:00 TXU advise interruption to power in Corryong from
09:00 to 15:00 (removal of overhead power lines
in main street). We're trying to arrange genset
for this period.
14/Nov/2004
13:00 After many hours frantic work, Mark and Ross have
rebuilt an access point, re-located the whole
site and got it back on-line and running.
Action to be escalated to the AFP for action.
13/Nov/2004
15:13 Complete loss of our microwave network.
Criminal damage (police are persuing) to our main
access point, approx $10K of wanton destruction.
All microwave access to clients lost. Skycam gone.
09/Nov/2004
16:30 Brief interruption in service to Mount Beauty POP
while a UPS was exchanged.
22-Sep-2004
18:00 Power supply fan in secondary DNS has failed,
resulting in a few unexpected reboots due to
overtemperature. Replacement scheduled ASAP.
07/Sep/2004
11:20 Loss of ISDN to Corryong. Widespread outage, has
disrupted telephone, ISDN and mobile services to
the area. Seems to be telstra only. Also affected
Lavington mobile out, Jindera out. No response or
information from telstra (surprise!).
Restored (without notice) 12:30
30/Aug/2004
21:10 Loss of all Albury microwave sites.
Tech recalled. Main storage cell flat, replaced.
(Due to lack of solar radiation, steps being taken
to provide early-warning in future)
29/Jul/2004
04:15 Loss of all Albury microwave sites.
Tech recalled, problem found and fixed - loss of power
at CPE2 access point. PSU replaced, services restored.
18/Jul/2004
17:24 Loss of all ihug satellite services, affecting Falls,
112311197 Hotham and Beauty. Traceroute dies at ken-core4 in
in Sydney. Reported to telstra. (Seems ok from our
servers in the USA)
16/Jul/2004
12:33 Loss of DSL connectivity and some strange DSL type issues.
Eventually found the problem - some idiot has re-assigned
OUR fixed IP address to another user in Sydney, thus causing
all manner of wierd routing issues.
30/Jun/2004
10:30 Loss of wireless communications from CPE1.
Communications restored after a few minutes, investigations
into the cause continuing.
10/Jun/2004
08:30 A new, dual-redundant power supply/battery/power shelf
has arrived and is scheduled to be installed. Probable
interruption for up to 10 minutes while re-wiring of
the communications rack is undertaken. Will affect all
users in Corryong, Wangaratta, those calling 604946xx
and diversion facilities in Corowa, Culcairn and Holbrook.
09/Jun/2004
06:23 Lost power to Microwave Access Point. Tech called to identify
the cause, and take whatever actions are required.
07/Jun/2004
17:42 Jury-rigged supply is running, all services back on line.
Interruption affected Corryong, Wangaratta, some diversion
sites and people calling the Albury Tigris. MtBeauty,
Albury Max and some diversion sites unaffected.
Replacement redundant supply due tomorrow.
07/Jun/2004
17:01 Bang! from the back of the office area.
A quick sniff in the computer room confirms "something" is wrong.
Tigris Power supply has let out the magic smoke.
05-Jun-2004
14:52 90 minutes later, still no response from telstra (how typical!).
Our tech attended the site, reset the telstra equipment, unplugged
and re-plugged the cables, and error lights have gone off, site now
back on-line. Waiting for explanation and investigation by helstra.
05-Jun-2004
13:22 Mount Beauty ISDN services all seem to be down.
111855889 Taking forever to get someone at telstra to accept
and action the fault. No ETR yet.
Almost impossible to understand "Roshni" in faults
centre who seems to have no idea of what "onramp 30"
services are.
Response *REQUIRED* within an hour.
30/May/2004
Interruption to Construction-Cam expected while
it is relocated. Expected duration 30-60 minutes.
29/May/2004
Brief interruption (approx 2 minutes) expected for
our Albury Microwave Network while new power circuits
are brought on-line. This work should increase the
long-term reliability of the site.
27/May/2004
19:41 Mains restored in Corryong, everything back up.
27/May/2004
18:12 Mains failure in Corryong, UPS batteries have given up,
corryong site is currently off-line.
18/May/2004
08:40 Changeover completed, less than 2 seconds downtime.
Don't you love it when a "best case" scenario works out?!
18/May/2004
08:30 Telstra are changing routers at the Albury exchange. Our
primary link will be off-line for an estimated 2 minutes
during this work.
13/May/2004
17:50 We have been advised of a failure of the upstream News
server which feeds us. No ETR given at this stage.
15/Mar/2004
20:30 Major latency between sydney and perth to all international
15739226 destinations. Reported to Paul at telstra wholesale who
was going to take it up with reach. No ETR. Snip of trace:
8 Pos2-0.ken-core4.Sydney.telstra.net (203.50.6.165) 43.915 ms
9 10GigabitEthernet3-0.pad-core4.Sydney.telstra.net (203.50.6.86) 44.134 ms
10 GigabitEthernet0-0.syd-core01.Sydney.net.reach.com (203.50.13.242) 41.247 ms
11 i-13-1.sjc-core01.net.reach.com (202.84.143.21) 1723.123 ms
12 qwest.sjc-core01.net.reach.com (134.159.63.30) 1750.298 ms
16/Mar/2004
04:30 Telstra advise an interruption to "ALL-Albury" routers
for scheduled maintenance.
06/Mar/2004
18:45 Loss of signal to MtBeauty radio AP.
Tech called to site, found flat battery. It had
not been replaced when it was supposed to have been
and was exhausted. Replaced, service interval fixed.
Service resumed 19:20.
17/Feb/2004
10:50 Unusually high latency on peering link to Sydney.
Appears to be a problem within the telstra adsl
network introducing 5000mS latency. Reported to telstra
and they're working on it. Will not be affecting any
of our local users, but may cause problems for roaming
customers because of slow radius authenication.
08-Feb-2004
15:30 Power failure in Wangaratta CBD. Site running on UPS.
UPS gave out about 16:30, power restored 17:55.
05-Feb-2004
09:04 Barry Keenan from Telstra is on-site to do BERT tests on the
service, following more checks since 30/Jan/2004 show there
are more error-seconds running on this service. All call to
60494600 were interrupted briefly while that line was taken
out of service for testing, but calls were being accepted
on the same number but alternate service channels.
30-Jan-2004
15:05 Several calls connected to Albury pool 60494600 dropped out
simultaneously. Our equipment recorded a loss of sync on one
Onramp30 service. Fault reported to telstra, awaiting action.
