Signalling system fault - Nov 26

[ijmad]

It appears the Elizabeth Line CCOS has been shut for over 4 hours at this point.

I just wondered if there was any information about what has failed, I assume it's a major failure in the ATO system.

[Chris M]

It's a signalling failure, I think between Paddington and Bond Street. One staff member said they heard the phrase "worst signalling failure we've ever seen" from a Siemens engineer.

[spsmiler]

Please explain what CCOS stands for

[tut]

Nov 26, 2024 13:25:28 GMT Chris M said:

It's a signalling failure, I think between Paddington and Bond Street. One staff member said they heard the phrase "worst signalling failure we've ever seen" from a Siemens engineer.

Nov 26, 2024 13:26:50 GMT spsmiler said:

Please explain what CCOS stands for

Crossrail Central Operating Section

[ijmad]

It's still completely suspended. Catastrophic.

[goldenarrow]

A raw deal of course for affected passengers today but undeniably the safest option given the extent of the issues that are being investigated. You really do not want to play Canute when the integrity of systems is being called into question.

[ijmad]

Completely agree!

[ducatisti]

Tells you a lot that the look on the passengers this morning was disbelief rather than anything else. We don't expect it to break...

[ijmad]

CCOS still completely down this morning.

[goldenarrow]

Limited services in the core have begun running as of 10:30.

[pgb]

Nov 26, 2024 13:39:24 GMT tut said:

Crossrail Central Operating Section

For some reason, I thought it was Central Core Operating Section

[class411]

Have they tried turning it off and on again?

[xtmw]

They actually did! However no dice

[spsmiler]

What a mess - has anything like this happened on any other services here in London? (exception, DLR during 1990s upgrading when the system became the Docklands Light Failway because of the problems)

Too much AI, too much use of public internet??

[Tom]

We don't use AI for signalling systems. There is talk about the possibility of AI being used for software and logic validation but it is limited to academic papers at the moment. Likewise, other than possibly one installation in rural Wales, we don't use public networks. This is highly safety-critical stuff we're talking about - maintaining the security of the systems is essential.

[class411]

Nov 27, 2024 22:37:58 GMT Tom said:

We don't use AI for signalling systems. There is talk about the possibility of AI being used for software and logic validation but it is limited to academic papers at the moment. Likewise, other than possibly one installation in rural Wales, we don't use public networks. This is highly safety-critical stuff we're talking about - maintaining the security of the systems is essential.

The day LU uses AI for safety critical decisions is the day I limit myself to walking/buses/taxis.

[gefw]

Nov 27, 2024 22:37:58 GMT Tom said:

We don't use AI for signalling systems. There is talk about the possibility of AI being used for software and logic validation but it is limited to academic papers at the moment. Likewise, other than possibly one installation in rural Wales, we don't use public networks. This is highly safety-critical stuff we're talking about - maintaining the security of the systems is essential.

I wouldn't be that surprised or too concerned if AI/learning algorithms was used in non-safety elements of signalling systems such as Passenger info (eg to predict time to arrival) and also in automatic tweeking of timetables and ATO operation (eg to manage headways).
With regards the issue on Eliz line, The failure cause/mode has not been publicly explained yet, but hopefully whatever happened the system "failed safe" (causing the very unfortunate disruption which had some safety related issues particularly relating to congestion)

[spsmiler]

Tom Thank goodness for that - it may be cheaper (headline £costs) to use public telecoms etc infrastructure I see this as extremely risky for something as critical as train control, etc.

I did read once of a tram system in Italy where it was said that a hacker had somehow managed to infiltrate the system and was able to change points where routes diverge. Possibly more too - its probable that the full details were kept out of the public domain, in case it encouraged others to try to do the same.

--------------------------

I will admit that when a teenage schoolboy (in the 1970s) I wished that I could influence train operations - so that Hammersmith & City trains travelling to Barking would no longer have to wait what seemed like ages whilst two or even three Upminster-bound District line trains which arrived on the scene after it were allowed to enter Aldgate East before the train I was on.

[pgb]

Nov 28, 2024 0:02:14 GMT class411 said:

The day LU uses AI for safety critical decisions is the day I limit myself to walking/buses/taxis.

Doesn't ARS on the national network make AI decisions by looking for nearby trains and will regulate accordingly? That's making a decision by itself - what's the difference between NR and TFL?

[ijmad]

I am not a railway engineer but I am involved in cybersecurity, so what I can say is that best practice involves having a safety critical network fully air-gapped from the Internet and even other networks in the organisation, so no traffic in or out, and then to use hardware 'diodes' to allow small amounts of specific information to make their way across the gap in one direction only, so things like real time passenger information can be provided on the Internet etc., for apps.

The servers operating the ticket machines and gate-lines will be on another separate network, as will the passenger information system. These don't have to be quite as well separated as they aren't considered quite as safety critical as the signalling system but they'll still be segregated private networks.

Obviously there have been instances where certain operators have failed to meet these standards, and so there have been hacking incidents where people were able to affect operational railways, but this is very uncommon. TfL's network infrastructure teams are very good from what I gather so I'm sure they are doing what they should!

[Colin]

Nov 28, 2024 15:26:19 GMT ijmad said:

TfL's network infrastructure teams are very good from what I gather so I'm sure they are doing what they should!

Hmm!!

Their reaction to the recent cyber attack makes me wonder. If they were that good and had things tied down so well, they wouldn't have needed to completely kill all access and make all 20,000 or so employee's reset their passwords.

Some three months on we still don't have full access to the applications we had access to prior to the attack.

[gefw]

Regarding segregation of systems - In these days where remote access for diagnostics and monitoring is generally required it is very tricky  !!
Recent example with the Airtraffic control system (where system support was remote)

[Tom]

Nov 28, 2024 15:20:56 GMT pgb said:

Nov 28, 2024 0:02:14 GMT class411 said:

The day LU uses AI for safety critical decisions is the day I limit myself to walking/buses/taxis.

Doesn't ARS on the national network make AI decisions by looking for nearby trains and will regulate accordingly? That's making a decision by itself - what's the difference between NR and TFL?

That's not artificial intelligence per se - ARS systems usually have a hierarchy of decision making rules which they are given to follow rather than being allowed to make their own decisions based on learned experience like a human operator would.

[jimbo]

Three months after the cyber-attack, some FoI replies are now saying that access to statistics is not yet available, and applicant should ask again in six months' time! Presumably, to be sure of access, and avoid a rush of reapplications in a month or two, but that is allowing nine months for things to get straight again!

[class411]

Nov 28, 2024 15:20:56 GMT pgb said:

Nov 28, 2024 0:02:14 GMT class411 said:

The day LU uses AI for safety critical decisions is the day I limit myself to walking/buses/taxis.

Doesn't ARS on the national network make AI decisions by looking for nearby trains and will regulate accordingly? That's making a decision by itself - what's the difference between NR and TFL?

As Tom said, above, that is not AI per se.

A system that works by following a very large and complicated set of rules may appear so, but no real intelligence us involved.

[themightytrex]

Nov 29, 2024 3:08:52 GMT class411 said:

Nov 28, 2024 15:20:56 GMT pgb said:

Doesn't ARS on the national network make AI decisions by looking for nearby trains and will regulate accordingly? That's making a decision by itself - what's the difference between NR and TFL?

As Tom said, above, that is not AI per se.

A system that works by following a very large and complicated set of rules may appear so, but no real intelligence us involved.

ai has no real intelligence. it's just probability of the output result being most likely based on the prompt.

[pgb]

Nov 28, 2024 20:16:58 GMT Tom said:

Nov 28, 2024 15:20:56 GMT pgb said:

Doesn't ARS on the national network make AI decisions by looking for nearby trains and will regulate accordingly? That's making a decision by itself - what's the difference between NR and TFL?

That's not artificial intelligence per se - ARS systems usually have a hierarchy of decision making rules which they are given to follow rather than being allowed to make their own decisions based on learned experience like a human operator would.

And judging by how often ARS gets it wrong when it comes to regulation - AI is probably the better system