District line lightning strike on 7/7/24

[starlight73]

Yesterday afternoon, the District line was suspended between South Kensington and Ealing Broadway/Richmond/Wimbledon.

The sign I saw said this was due to a "lightning strike on major signals".

If this is public info, which signals were they? I saw somewhere saying this was at "Wimbledon" but that doesn't seem to match with the suspension.

And how do teams repair this? Do they have to put up a new signal or can they introduce some kind of working past the signal?

Mods - please do move this to the District line board if you think that's better

[Colin]

It was indeed a suspected lighting strike on signalling equipment.

It was actually the digital CBTC system that was affected - it uses a lot of aerials so is susceptible to lightning strikes - some 53 wayside radios in the Fulham Broadway. West Kensington, Earls Court and High Street Kensington areas were affected. This caused all trains in those areas to become non communicating with the signalling system. The size of the area affected essentially meant the train service was suspended west of South Kensington whilst stalled trains between stations were identified and moved into platforms manually.

The repair in this case was to go trackside and manually reset the affected equipment.

If a very small section was affected we could have run a service under failure conditions but in yesterdays instance the affected area was far too big to try and run a service under failure conditions.

[spsmiler]

I'm shocked!

When I saw the thread title I thought of the industrial dispute type of lightening strike - not atmospheric conditions / weather related issues.

[starlight73]

Thanks for that comprehensive answer Colin!

That also explains why some conventionally signalled parts of the line were shut down. (Barons Court - Richmond/Ealing Broadway) - as trains only reverse at West Kensington which is within a CBTC area.

Lots of hard work from everyone involved then!

[t697]

One would hope that the ATC spec required something better than tedious manual resets after a lightning strike. However I don't think this is the first case.
I also noted today that Toronto is about the second worst place in Canada for lightning strikes. So not like the supplier has never heard of lightning.

[xtmw]

For somewhere like the UK it would be convenient for the CBTC equip to be lightning proof, but if it costs more money why bother considering we don't get thunder that often. Someone talked to me about making the Underground snow-proof. I laughed. It snows once a year here, not even that...

Canada is another thing though

[zbang]

Jul 8, 2024 22:29:46 GMT xtmw said:

For somewhere like the UK it would be convenient for the CBTC equip to be lightning proof,

"Lightning proof" doesn't exist*, only "Sort of lightning slight resistant". A direct strike to an antenna is going to fry any electronics on the way to the earth. There are some mitigations, like air terminals above the antenna and surge suppressors, but even then things are likely to be damaged. I'd expect that the CBTC antennas already have some sort of mitigation and protection.

*put the antenna in a Faraday cage and it'll be completely protected, it'll also be completely useless

Into the weeds here- the highest point forms a "cone of protection" around the lower points, so an air terminal at 20m will protect antennas at 18m from a direct hit but they'll still see an electric surge. If you look at the BBC tower, you'll see a point above all the other hardware on the roof, there's probably a 4-5mm copper cable running straight down to the dirt from that. It also common to put up a lot of air terminals to help bleed off any static charges so that lighting doesn't form in the first place.

Lightning is weird .

[gefw]

Jul 9, 2024 16:26:38 GMT zbang said:

Jul 8, 2024 22:29:46 GMT xtmw said:

For somewhere like the UK it would be convenient for the CBTC equip to be lightning proof,

"Lightning proof" doesn't exist*, only "Sort of lightning slight resistant". A direct strike to an antenna is going to fry any electronics on the way to the earth. There are some mitigations, like air terminals above the antenna and surge suppressors, but even then things are likely to be damaged. I'd expect that the CBTC antennas already have some sort of mitigation and protection.

*put the antenna in a Faraday cage and it'll be completely protected, it'll also be completely useless

Into the weeds here- the highest point forms a "cone of protection" around the lower points, so an air terminal at 20m will protect antennas at 18m from a direct hit but they'll still see an electric surge. If you look at the BBC tower, you'll see a point above all the other hardware on the roof, there's probably a 4-5mm copper cable running straight down to the dirt from that. It also common to put up a lot of air terminals to help bleed off any static charges so that lighting doesn't form in the first place.

Lightning is weird .

The good news in the info provided by Colin says nothing "fried" - just needed a reset.
The size of the area affected presumably meant it was some form of Radio/wifi area base station lock up or circuit breaker tripping (so slightly surprised at the reference to trackside visits to do the resets on multiple items)
Suspect the lightning affected a key building or power supply - so thoughts of improving resilience or auto recovery may be directed that way.

Note from what I've seen of the "lamposts" that the Antennas are mounted on, there are no specific Lightning "air terminals" - just presumably the normal metalwork earth bonding. 
Interesting what the apparent "back plate" is around each antenna - I guess this is some form of radio signal deflector/direction focus & is probably the most vulnerable to all sorts of interference/ damage/deterioration.

[d7666]

Jul 10, 2024 7:28:02 GMT gefw said:

]The good news in the info provided by Colin says nothing "fried" - just needed a reset.
The size of the area affected presumably meant it was some form of Radio/wifi area base station lock up or circuit breaker tripping (so slightly surprised at the reference to trackside visits to do the resets on multiple items)

I know not of the details of this issue nor what kit was hit.

But I would think like this about the subsequent intervention needed.

If there was equipment to do a remote re-set, that requires electronic gubbins and electrical and data connections to activate.

If a lightning strike has taken out the actual electronic gubbins and or electric and or data connections of the main system you are interested in, that same strike will almost certainly have taken out your remote access kit.

So you are back to square one : go to site by man in white van or down the track in orange as necessary.

[Colin]

Jul 10, 2024 7:28:02 GMT gefw said:

The good news in the info provided by Colin says nothing "fried" - just needed a reset.

Correct, nothing was fried.

Jul 10, 2024 7:28:02 GMT gefw said:

The size of the area affected presumably meant it was some form of Radio/wifi area base station lock up or circuit breaker tripping (so slightly surprised at the reference to trackside visits to do the resets on multiple items)
Suspect the lightning affected a key building or power supply - so thoughts of improving resilience or auto recovery may be directed that way.

The affected part of the railway was a whole VCC area - in very simple terms an area wholly controlled by one computer.


I don't know what was actually hit; I'm merely repeating what I've read on an internal reporting system. From that internal reporting system I can say that 12 feeder pillars and the signal main were affected which in turn affected the 53 wayside radios. Some of the wayside radios picked up when the 12 feeder pillars were reset so it wasn't a case of all 53 wayside radios needing resetting. I gather the resets were carried in various locations around the affected the area rather than in one place so it seems to me the lightning strike caused a domino effect throughout the whole VCC area.

[gefw]

Thanks Colin - An old timer like me gets very confused by the terminology - Basically Wayside/Trackside encompasses anything outside the Control Centre Suites & not on board a train (but does include Local equipment rooms such as SERs/CERs)
With regards how safety critical computer based systems are typically designed to handle "fault" conditions; When a computer encounters "corrupted data" it can automatically force a soft reset or it can "suspend" critical functions/tasks awaiting operator intervention ( eg command or reset) - but hopefully leave network/system management functions running thus allowing remote diagnostics/investigation including a remote reset command. I believe such comms system management tools are available on the Thales SSL because they are used for the trackside WIFI antenna remote condition monitoring system (developed by The LUL signalling support team)

[d7666]

Jul 11, 2024 8:03:35 GMT gefw said:

Thanks Colin - An old timer like me gets very confused by the terminology - Basically Wayside/Trackside encompasses anything outside the Control Centre Suites & not on board a train (but does include Local equipment rooms such as SERs/CERs)
D7666 with regards how safety critical computer based systems are typically designed to handle "fault" conditions; When a computer encounters "corrupted data" it can automatically force a soft reset or it can "suspend" critical functions/tasks awaiting operator intervention ( eg command or reset) - but leave network/system management functions running thus allowing remote diagnostics/investigation including a remote reset command. I believe such comms system management tools are available on the Thales SSL because they are used for the trackside WIFI antenna remote condition monitoring system (developed by The LUL signalling support team)

Don't you just like it when someone tells you your day job. I'll leave it at that.

[gefw]

Jul 11, 2024 8:18:56 GMT d7666 said:

Jul 11, 2024 8:03:35 GMT gefw said:

Thanks Colin - An old timer like me gets very confused by the terminology - Basically Wayside/Trackside encompasses anything outside the Control Centre Suites & not on board a train (but does include Local equipment rooms such as SERs/CERs)
D7666 with regards how safety critical computer based systems are typically designed to handle "fault" conditions; When a computer encounters "corrupted data" it can automatically force a soft reset or it can "suspend" critical functions/tasks awaiting operator intervention ( eg command or reset) - but leave network/system management functions running thus allowing remote diagnostics/investigation including a remote reset command. I believe such comms system management tools are available on the Thales SSL because they are used for the trackside WIFI antenna remote condition monitoring system (developed by The LUL signalling support team)

Don't you just like it when someone tells you your day job. I'll leave it at that.

Understood/sorry I don't know your background , but these forums are about improving the knowledge of the wider audience.
Colins later post does indeed clarify the structured sequence of events undertaken by the Maintenance staff to resolve the failure ASAP.
Hopefully there will be some feedback following design team investigation as to whether the system should have handled the corruption & "snowball" better

[Tom]

Jul 11, 2024 8:18:56 GMT d7666 said:

Don't you just like it when someone tells you your day job. I'll leave it at that.

Knowing who both of your are (behind the username, so to speak), all I can say is this was a highly amusing read! 😂

[Tom]

Jul 8, 2024 19:45:34 GMT t697 said:

One would hope that the ATC spec required something better than tedious manual resets after a lightning strike. However I don't think this is the first case.
I also noted today that Toronto is about the second worst place in Canada for lightning strikes. So not like the supplier has never heard of lightning.

"Works as per design" was the phrase someone used with me today. Interestingly there doesn't seem to be the same issue with the very similar setup in Singapore.

[zbang]

The problem with "works as per design" is that designers seem to forget about all the edge cases and that things might not be used in the intended fashion or circumstances but still need to be predictable....

[gefw]

A perfectly standard/normal response; puts the onus the other party to prove otherwise (and naturally making them decide if it is worth the effort considering the overall picture)
That is why Engineering/Technical support exists to support the Business Client.

Mod note: empty quote sections removed. This post appears to be a reply to Tom's "Works as per design" comment

[zbang]

I recall having a rather strident discussion with someone who called being able to set a date of Feb 31 as "working per design" (seems that all months had a 31st in that system, at least for a couple of weeks).