Signalling System Failure 16-17/3/24

[chilsie]

I was surprised at the magnitude of what happened, though I kind of had an inkling that something was wrong quite quickly. I was at Fulham Broadway leaving a football game when we were held on the platform for about five minutes with the driver informing us we were “waiting for the signalling system to pick the train up”, which was immediately a sign of something wrong. I then saw three trains unusually close to each other on the opposite side as we headed towards West Brompton, though the biggest whopper came when we arrived at High Street Ken, the train’s terminus, and were informed that the entire Edgware Road-Earl’s Court section had been completely suspended!
Would anyone be able to share further knowledge on what happened to the system which caused such carnage?

[Dstock7080]

The failure first occurred on Saturday 16 March at 0600 when VCC4 Vehicle Control Centre crashed, and trains were unable operate across the boundary between 3 and 4 Monument, 4 and 5 Sloane Square.
All trains stopped communicating with the Control Centre.
Unfortunately a trespasser compounded the attempts to move trains manually (one at a time) away from the failure area to reboot the VCC.
After 0900 the boundary were reset and train were moved away from the boundary conflict points.
The VCC was rebooted at 1035 and movement was regained.
Unfortunately the failure reoccurred at 1745 when an engineers train came across the boundary from VCC3, which was under possession for the weekend.
Services resumed around 1830 with severe delays advertised until close of traffic.

Sunday 17 March; VCC4 failure reoccurred 1420, service suspended West Ham-Ealing/Richmond/Wimbledon and Edgware Road service.
Service resumed between Upminster and Whitechapel 2130 but remained suspended for passengers on rest of Line until close of traffic. Test trains were run empty from across the suspended section during the evening.

It was concluded that engineers trains working within the weekend possession Wembley Park-Aldgate East, which included VCC3 Euston Square-Monument/Stepney Green had sometimes incorrectly used their CBTC equipment for movement, disrupting the interface between the passenger trains along the District part of SMA3.

[brigham]

"It was concluded that engineers trains.../ /... had sometimes incorrectly used their CBTC equipment for movement".

I hope that the cause of the odd behaviour of these trains has been investigated.

[zbang]

I rather expect that some of the software folks are all over this.
Me- "When I do YYY, the system crashes."
Programmer- "Well, don't do that."
Me- "Don't let me."
(actual conversation I've had multiple times with software developers)

I'd also love to hear about the root cause, but that's unlikely.

[gefw]

Bearing in mind that it is a safety critical system. I tend to concur with the failsafe principle being employed when an unacceptable/unexpected condition has occured.
The term "software crash" may not be a suitable description - one would expect the system to go into a "protected state", hopefully giving error messages and requiring some intervention to resume/reset/ignore.

[Tom]

Mar 20, 2024 15:37:15 GMT zbang said:

I rather expect that some of the software folks are all over this

They are. (Obviously can't say much more.)

[class411]

Mar 21, 2024 15:38:53 GMT gefw said:

Bearing in mind that it is a safety critical system. I tend to concur with the failsafe principle being employed when an unacceptable/unexpected condition has occured.
The term "software crash" may not be a suitable description - one would expect the system to go into a "protected state", hopefully giving error messages and requiring some intervention to resume/reset/ignore.

On the railways, "I'm sorry Dave, I can't do that", is acceptable, given an unsupported exception condition.

On an Airbus, less so.