💤 Unscheduled downtime

:zzz: Unscheduled downtime
0

#101

OK, that was a biggy!

Our hosting provider (Hostinguk.net - feel free to troll them on Twitter) had a massive outage from about 9 pm last night.

Apparently, and I shit you not, a fibre cable that had been run under a farmer’s field was cut. I

mean, farmers digging up a field, who’d have predicted that?

Anyway, we are up again BUT WE MAY STILL EXPERIENCE MORE DOWNTIME as this is a temporary overground fix.

What is really strange about this is that hosting companies build redundancy into each area of their operations so that if one part fails another one will kick in until full service can be restored. It SEEMS as though Hosting UK did have multiple fibre routes to their data centre, but they all went under farmer Giles’ field.

If you’re a geek, or are particularly warped, here’s the whole brutal saga. See here for continued updates.


#102
  • Update (31/05/2018 20:47): Our NOC has identified a network incident affecting several data centres on our UK network.
  • Update (31/05/2018 21:19): It is suspected that the convergence of two diverse fibre paths, from an underlying provider Zayo, may have been severed resulting in network isolation for DC 8 (St. Asaph). Engineers are working on three resolution paths (transit re-route, fibre repair and new fibre provisioning) with an ETA of 2-12 hours for full service restoration.
  • Update (31/05/2018 21:30): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 21:45): Zayo are investigating the fibre failures in North West England
  • Update (31/05/2018 22:00): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 22:15): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 22:30): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 22:45): We are awaiting confirmation, but current investigation would indicate the break of two, diverse fibre paths.
  • Update (31/05/2018 23:00): There appears to be a fibre event 27km from Manchester. Once we have an ETA from our fibre repair group it will be provided. Further updates to follow as more information is available.
  • Update (31/05/2018 23:15): It is confirmed that there was no planned works occurring at the time of the break. Updates to follow.
  • Update (31/05/2018 23:30): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 23:45): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 00:00): A technician is en route to the break location with an ETA of 20 minutes. We will begin testing as soon as the technician arrives.
  • Update (01/06/2018 00:35): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 00:45): The on-site tech has determined this is due to a large cable cut and narrowed the potential damage location. Our underlying fibre provider, Zayo, have OSP construction and splice crews en-route to a common Splice Enclosure just North of Stretford. Further updates to follow as more information is available.
  • Update (01/06/2018 01:00): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 01:15): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 01:45): Our fibre provider, Zayo, are still attempting to locate the exact location of the fibre break. It has however been narrowed down significantly.
  • Update (01/06/2018 02:00: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 02:15: The location of the fibre break has been narrowed down further. We are awaiting a further update from Zayo.
  • Update (01/06/2018 02:30: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 02:45: The fibre cut has been located and the damage is being assessed now. Zayo note that from the looks of things, there are two 96 count under ground cables that appear to be damaged.
  • Update (01/06/2018 03:00: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 03:15: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 03:30: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 03:35: Zayo have updated: Engineers are still on the ground trying to locate the exact damage location. The Cable crew is en route with an ETA of 4:00 AM BST. The construction crew is en route with an ETA of 5:00 AM BST. Further updates to follow as more information is available.
  • Update (01/06/2018 03:56: No new information available. Updates to follow within 34 minutes.
  • Update (01/06/2018 04:16: After walking the span where the damage should be the OSP crew were unable to locate any sign of construction or damage that would cause the outage. The OSP are now going to open the SEP’s on each side of the suspected damage and trace the exact distance in order to give them the exact location of the physical damage. Further updates to follow as more information is available.
  • Update (01/06/2018 04:30: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 05:03: OSP have identified fibers between the Computer Centre and one of their Splice Enclosures. The location matches the 1.2 km from another one of their SEPs. OSP is currently looking for signs any visible damage. As it is now starting to get light outside, it should assist in finding the damage. Updates to follow within 30 minutes.
  • Update (01/06/2018 05:30: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 05:54: The Zayo OSP has located the damage in a field where it appears a trencher has hit the cable and duct severing the full cable. The crew are currently working to devise a plan to get into the joints on each side of the damage as well as what they will do to repair the conduit. Unfortunately, this is in the middle of a field so there does not look to be spare duct run in the area. Once a plan has been devised we will share the plan as well as push our OSP for a potential estimated time to repair. Updates to follow within 36 minutes.
  • Update (01/06/2018 06:08: The Zayo OSP crew has identified the location of the cut, there is a 5" (five inch) micro trench that is approximately 2 meters in depth, that has been dug across a field our fiber traverses. This trench has severed multiple conduits and completely severed their cables.f 2m. . Zayo is currently assessing the damage to find the nearest chambers with the cabler team in an effort to identify the quickest possible restoration, whether that be a temporary (section throw/splice) or permanent cable pull/splice. The location is in the middle of a farmers field which has added time to our ability to locate. Updates to follow within 30 minutes.
  • Update (01/06/2018 06:40: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 07:02: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 07:20: Zayo have found the 2 chambers either side of the fibre fault and found some slack cable but the chambers are 900 and 1200m away. The preferred option is to dig and pull the slack to create 1 extra joint, if they are unable to do this they may have to add extra joints. They are currently trying to gain access to the fields due to multiple ownership to access areas to where the fault is located but are in conversation with the landowners to get to the fault. Updates to follow within 30 minutes.
  • Update (01/06/2018 07:44: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 08:02: OSP is currently excavating in the field to try get to the cables, and sub-duct. They are doing this to try, and pull the slack cable through. Updates to follow within 30 minutes.
  • Update (01/06/2018 08:30: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 08:50: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 09:20): The excavation continues. We will update you as soon as we any new information.
  • Update (01/06/2018 09:50): Civil crews are still working to complete excavation at this time. Crews expect to be completed soon, and should be pulling slack on the cable to prep for splicing. More updates to follow.
  • Update (01/06/2018 10:00): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 10:30): Excavation has been completed, but the cable did not have enough slack to pull through. Crews are now prepping a new 50m strap cable between the 2 exposed ends of the cable. More updates to follow.
  • Update (01/06/2018 10:55): Fibre engineers estimate a repair schedule of 1-7 hours, with a possible ETA of 18:00
  • Update (01/06/2018 11:20): Fibre engineers intend to run a temporary fix over ground splicing to commence around 12:00 for services to start coming back up.
  • Update (01/06/2018 11:40): Crews have completed prepping the cable into the joints and are now taking the cable over to the excavation location to be prepped and spliced. We estimate we will begin splicing at 12 GMT.
  • Update (01/06/2018 12:00): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 12:30): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 12:40): Crews are nearing completion of all the cable prep and should begin splicing in 30 minutes. Will provide an additional update when splicing begins.
  • Update (01/06/2018 12:45): We have partially restored DNS services for customers. Please note that DNS responses may be intermittent and slower than usual.
  • Update (01/06/2018 13:00): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 13:20): Crews are now starting to splice the cable. You will begin to see services restore as splicing continues. No ETR at this time. Will provide additional updates as splicing continues.
  • Update (01/06/2018 13:45): Connectivity has been restored to our DC 8 (St. Asaph) facility. A temporary overground fix has been put in place. Please note that service is still at risk and there are likely to be interruptions.
  • Update (01/06/2018 14:15): Repair work continues on a proper fix for the fibre cuts. At the moment, both of our paths are back online albeit with a temporary fix. We will provide further updates within the next 60 minutes.

#103

So what happened?


#104

Fucking he’ll I was on the edge of my seat reading that. Gripping or what?!

At least it wasn’t Mrs C_S blocking the website using the parental controls in an effort to make interact with the family…


#105

And who said IT people are tedious dullards?


#106

I hope the farmer has been severly dealt with.


#107

I bet beltch transcibed all of that as well rather than copying and pasting, just for the thrills.


#108

FFS, just as you think Bletch’s posts can’t get any longer or more boring he posts something like that and surpasses himself.


#109

What sort of farm was it? Dairy?


#110

Don’t knock it.

I moved into IT because I couldn’t handle the cut and thrust of accountancy auditing.


#111

I saw this advertised on tv this morning. I thought “What the fuck is that all about and why is it on my tv”

I now know. It’s a sign that we arrange a visit to farmer Giles for a “chat”.


#112

Thats what the Russians want you to think…


#113

So I wonder what the Welsh lads are doing for their temporary fix?

A couple of Ski yogurt pots with some taut string between them?

Or much like a carrier pigeon, a carrier sheep, diligently delivering packets of our bollocks over T-Sheep-IP?


#114

Tweaked to get you started as it’s Friday and I’m sure you’ll be dropping a few c-bombs along the way.

:lou_wink:


#115

If you’re into conspiracies, and fuck knows some of you are, here’s a couple for you…

As @lifeintheslowlane and I discussed on the testing Discourse server earlier, the rumour is that the farmer - heavily pro EU with all the subsidies they get, didn’t want the Brexit thread to continue.

Sabotage. You heard it here first.

Or, did anyone notice that this all happened in Stretford in MANCHESTER?

What do you think THEY are trying to stop us talking about?

Mourinho in for Carrillo?

And a farm in Stretford?

Really?

Yeah, right.

I’ve said enough already, I’m going to h…


#116

Oh, a friend told me pornhub was knocked out too - so he couldn’t knock one out…

:lou_eyes_to_sky:


#117

I see a Papesque tsw conspiracy emerging.

Coincidence?

Visa card system crashes


#118

Visa card have gone down as well. Thought they would have a more professional system, but appear to be at the mercy of Northern farmers as well.

@saintbletch you keep mentioning that no one should be surprised that a farmer is digging his field, but 5ft deep at 9pm? Sounds more like @goatboy has done a disposal deal.


#119

It’s fuckin’ frackin’ I tell eee.


#120

Looking at Belch’s post it was a Diary farm @goatboy