Broadband Junction Status

We currently have a great shortage of BT sessions on our ADSL platform, LLU services are not affected. We believe this is related to a BT MSO as other service providers are having the same issue however BT have not yet posted anything. This outage is not related to last nights engineering works.

09:12 – At 08:06:58 all 20CN and 21CN lines routed via Manchester dropped and at the same time we stopped seeing any platform RADIUS requests from BT. This suggests a major routing issue within BT. It means that nobody can re-connect any broadband line.

11:51 – Sessions started to return at 11:00 and all seems to be working fine now.

11:58 – Following an update from BT, they confirm that the issue is resolved after replacing failed hardware.

We have just been informed by our fiber supplier that they will be performing a required firmware update to the multiplexers spanning our main fiber between London and Chelt.

The maintenance window is scheduled for this weekend at the following time:

Start Date 12th May 2012 @ 22:00
End Date 13th May 2012 @ 06:00

During the above window, there will be three brief outages while the multiplexers are updated.

Once again, we apologize for the short notice which unfortunately was out of our hands. This is a semi-urgent update to correct a number of software related bugs within the DWDM equipment to ensure smooth operation.

Tonight from 8PM we will be moving core equipment and upgrading our current Firewalls in preparation for rolling out a second redundant network, the work will involve moving some equipment, some services will experience some short amounts of downtime as the equipment is powered down, moved and powered up again. We aim to keep the as much of the network up as possible during the works.

We have received the following notification from our Transit provider. Any website peering with BT will be experiencing issues; examples are Lloyds TSB & Natwest. A Trace Route will show if BT are involved in the link between you and the website you are visiting.

BT peering and high latency — Today, 11:21 AM
British Telecom are experiencing issues at the London Internet Exchange

11:36 – We are currently experiencing a transit outage, The issue has been reported to our provider and we are awaiting an update.

11:46 – Services resumed, no word on the cause yet.

12:27 – Update from our provider…

Safehosts suffered an un-planned network disruption at 11:24, 11/04/2012, we have yet to find the cause but believe it to be un-related works at telehouse London.
Whilst re-routing the backup feed, our connection was restored.

We appologise for the outage, our backup feed will be added as a continued redundant network to avoid outages like today.

An announcement will follow, with progress.. as the need for this now has clearly seen today.

Regards

Safehosts.

We are currently experiencing a total loss of services due to loss of connectivity over our Diverse Fibre connection into London. Our provider had provided us with an “At Risk” notice earlier today due to essential emergency works. The notice states that they are not working on our equipment but nearby, however the lights are off at present! We have notified their Network Operations Team but at this stage do not have an ETA on a fix. The period for the “At Risk” notice is 00:00 – 05:00 today however we expect our services to be resumed much sooner than 5am!

01:07 – Update from our supplier “We’re seeing total packet loss out of THN at the moment and have been advised by supplier of these circuits, that they are dealing with a break in a main fibre trunk into THN. We have been told a fusion splice of our pairs is necessary and is being effected immediately.”

03:10 – Lights back on, just checking everything is OK.

03:30 – Looks like everything is OK with BT DSL sessions and all VoIP & Co-Lo Services, BE LLU DSL Services are returning slower than expected So I have put a call into their NOC to get things checked out.

03:37 – Just waiting for BE Wholesale to reset our tunnels.

06:00 – Just a quick note to confirm we can see a healthy number of sessions. If you are still having issues please reboot your router.

02:17 – There is currently an outage of DSL services, we are looking in to the issue and will post here shortly.

02:33 – We are told by datacentre engineers that the fault relates to an external fibre link, we are now awaiting updates from them.

15:15 – We have noticed slow throughput on our transit link, we have reported the issue and will update here shortly. You should notice only slow loading webpages and slower downloads due to this issue. Some voip quaility may be lost on higher bandwidth codecs.

15:48 – An engineer is working on the issue, however during this work we have lost our backhaul also so all services are now offline. We will keep you posted with updates here.

15:52 – Passing traffic again…..

16:00 – Problem appears to have been with our Data Centres external connectivity, we are now waiting for info from them.

16:58 – Problem relates to a memory fault on a Cisco Switch. We are currently running across a different link for connectivity, this will change later tonight once the hardware has been replace by Data Centre staff. There will be some breif down tine when the switch happens between 7pm and 2am tonight.

20:15 – The change of hardware has taken place, we will monitor things over the next few hours.

We will be replacing our edge firewalls between 11pm tonight and 1am tomorrow. There maybe some breif drops of internet connectivity during this time.

10:15 – We have had a number of calls regarding BE LLU lines having no PPP session. BE are not reporting any service outages and our network is running correctly (I am writing this with a PC connected via BE!). We have raised individual faults so far and have escalated within BE as a wider issue.

10:40 – All sessions are returning