Dieselgate, however for trains – some heavyweight {hardware} hacking – BadCyber
A practice manufactured by a Polish firm instantly broke down throughout upkeep. The consultants have been helpless – the practice was advantageous, it simply wouldn’t run. In a determined final gasp, the Dragon Sector crew was known as in to assist, and its members discovered wonders the practice engineers had by no means dreamed of.
On this story, we’ll take you on an uncommon journey. A journey stuffed with sudden discoveries and occasions, a journey underneath stress of money and time, in addition to uncommon applied sciences. A journey wherein the practice performs crucial position – though sadly it doesn’t transfer, regardless that it ought to. Fasten your seatbelts – or a minimum of sit comfortably, as a result of there are sharp turns forward.
Profitable the tender, shedding the service
The story most likely begins a bit earlier, however we’ll enter the scene in spring 2022, when the upkeep for the primary of 11 Impuls 45WE trains (made by Polish firm Newag) operated by the Decrease Silesian Railways ends. The upkeep is carried out by an unbiased practice upkeep firm known as Serwis Pojazdów Szynowych, hereafter known as SPS, SPS received the tender to hold out the obligatory upkeep of the trains after a distance of 1,000,000 kilometres. The practice producer, Newag, additionally competed within the tender to hold out the upkeep, however the producer’s bid was about 750k USD increased and the tender was ultimately received by SPS, which provided to hold out the upkeep of 11 trains for round 5.5 mln USD.
Upkeep a practice is a sophisticated affair – it must be taken aside, the components despatched to the assorted producers, checked, despatched again, the practice put again collectively once more and examined. The SPS carries out the upkeep procedures based on the related upkeep guide (some 20,000 pages) offered by the producer, however the practice doesn’t begin after being put collectively. The pc says every part is okay, the practice is able to run – but it surely doesn’t run. The inverters aren’t supplying voltage to the motors and nobody has any concept why that is occurring. Upkeep technicians search, verify, confirm, seek the advice of the manuals – they discover no reply.
Mysterious breakdowns
The Decrease Silesian Railway has eleven Impulses and, based on the schedule, one other one is about to be despatched for upkeep, whereas the primary – as a substitute of returning to work – continues to be sitting within the workshop. The second practice is present process an an identical upkeep, with an identical outcomes. Earlier than the upkeep it was working, after the upkeep it now not needs to run. The work on getting the primary practice up and working, just like the practice itself, has not progressed one millimetre, whereas the producer refuses to assist. Two motionless trains are already sitting within the workshop. The third misses its inspection as a consequence of battery failure, so a fourth practice (“from the longer term”) is shipped to the upkeep as a substitute. The upkeep firm needs to benefit from its presence to tow a type of that received’t run. When the fourth (working) practice is related to one of many stationary ones, the working one additionally involves a standstill (the rationale for this has not but been established). As well as, at one other workshop in one other Polish city, Szczecin, one other Impulse breaks down in very related circumstances – it doesn’t begin up after servicing.
Poland’s high hackers
At a sure level, the issue turns into severe sufficient to be seen by the media – the six longest trains of the Decrease Silesian Railway out of service imply that timetables need to be diminished, alternative trains need to be despatched to the tracks, and passengers journey in overcrowded, shorter trainsets. Newag explains that the trains have been blocked by a “security system” – however within the 20,000 pages of directions, it’s in useless to seek out even a point out of it. A day of practice downtime within the workshop prices over 1000 USD in contractual penalties, and there are a number of trains caught, so the strain degree within the SPS is rising. Since neither the mechanics nor the electricians have an answer, somebody varieties “Polish hackers” into Google and comes throughout an article in regards to the Dragon Sector group’s successes within the CTF enviornment on the high of the outcomes record. SPS makes contact with DS, whose representatives at first can’t imagine the proposal they hear. Practice hacking? Properly, why not. The events signal a contract. Dragon Sector members Michał “Redford” Kowalczyk and Sergiusz “q3k” Bazański, recognized for hacking Toshiba laptops, tackle the challenge, and Kuba “PanKleszcz” Stępniewicz, who has expertise in industrial automation, joins in. The crew set off briskly to work, with Kuba taking a visit to the workshop. On website, they get a practice that doesn’t transfer, two spare computer systems and the pc producer’s SDK recordsdata. They begin the work by tapping into the CAN bus, but it surely’s tough to learn the visitors with out documentation of the protocols. They take a very long time making an attempt to dump the embedded software program from the on-board laptop. They haven’t any documentation of the pc and the SDK solely permits importing new software program, with no choice to dump present software program. As they experiment with the older model of the software program they discovered, importing it to the primary spare laptop causes it to cease responding – they’re left with just one working spare laptop. Ultimately, they discover a debugging interface and obtain the system’s reminiscence byte by byte.
The pc is predicated on the TriCore structure, like many related options within the automotive trade. Sadly, there’s a lack of fine disassemblers, so the researchers are bettering Ghidra a bit and may lastly look into the code. Admittedly, strings are lacking, however the work is slowly transferring ahead. A month and a half goes by when the SPS passes on the unhealthy information.
Because the deadline chases and trains break down
The Decrease Silesian Railway, unable to attend for its trains, decides to cooperate with Newag on the restore of damaged trainsets and their upkeep, together with trains which, based on the unique tender, have been but to be despatched for upkeep at SPS. The rupture of the contract with SPS is predicted to happen in per week’s time. As is well-known, nothing impacts the depth of the work as a lot as a really shut deadline by which the outcome must be proven, so the researchers set to work with redoubled vitality. In the midst of their work up to now, they’ve downloaded the reminiscence contents of quite a lot of computer systems, each trains which can be working and those who ought to solely be working. Evaluating these pictures is an ordeal, as virtually each practice has a unique set of features and a unique model of software program, however slowly they’re beginning to get a really feel for one thing. They determine values within the computer systems’ reminiscence which can be set in a single practice and zeroed out in one other. They’ll run exams at their desks – the pc, even when taken out of the practice, lets it run for some time (earlier than realising it’s lacking the remainder of the practice) to point out whether or not it’s able to run the inverters.
There may be lower than a day left till the deadline for completion once they discover the configuration of flags that provides the practice an opportunity to run. Sadly, throughout the experiments, the final working on-board laptop burns out. Sure, it burns – a capacitor burns (a reasonably random prevalence). After one other brainstorm and plenty of makes an attempt to place the 2 broken computer systems into one, they handle to restore the burnt one and at 2am, the evening earlier than doomsday, the researchers configure a pc to begin the practice. One of many researchers boards a practice (of a unique operator) to reach with the presumably working laptop on the workshop simply earlier than the representatives of the Decrease Silesian Railway, who’ve introduced a go to for 9:30 a.m. Sadly, the practice on which the researcher travels to the upkeep firm is late. Ultimately, within the morning, the researcher with the pc arrives on the website, connects the pray-it-works-computer to the broken-down practice, however the practice doesn’t begin. One other brainstorm identifies one flag that was forgotten and at 8:42 the practice manages to begin. The Decrease Silesian Railway delegation, seeing at 9:30 that the trains have an opportunity of getting again to life in any case, doesn’t break the contract with SPS.
Why the practice broke down
Determining the best way to get the practice to run wasn’t even half the battle – you continue to had to determine why it broke down, and that is the place the joys trip begins.
Months of research and reverse engineering uncovered some extraordinarily fascinating situations written into the software program code of varied trains equipped by Newag. After a whole lot of hours spent on code dumped from dozens of trainsets, it was doable to determine some very fascinating mechanisms inflicting sudden practice illness.
The numerical values 53.13845 and 17.99011 discovered within the laptop code appeared acquainted at first look. It quickly grew to become obvious that these have been GPS coordinates pointing to the neighborhood of Bydgoszcz Główny Railway Station, particularly the PESA (one other Polish practice producer and upkeep heart) website situated subsequent to it. Quickly the coordinates of different upkeep facilities that would perform practice repairs and upkeep in Poland have been additionally discovered. Beneath we present the pseudo-code of the algorithm (the names of the variables or features are given by the researchers for readability – we have no idea what the unique names have been):
check1 = 53.13845 < lat && lat < 53.13882 && 17.99011 < lengthy && lengthy < 17.99837;
check2 = 53.14453 < lat && lat < 53.14828 && 18.00428 < lengthy && lengthy < 18.00555;
check3 = 52.17048 < lat && lat < 52.17736 && 21.53480 < lengthy && lengthy < 21.54437;
check4 = 49.60336 < lat && lat < 49.60686 && 20.70073 < lengthy && lengthy < 20.70840
&& (this->lock_function_test & 1);
check5 = 53.10244 < lat && lat < 53.10406 && 18.07817 < lengthy && lengthy < 18.08243;
check6 = 50.12608 < lat && lat < 50.12830 && 19.38411 < lengthy && lengthy < 19.38872;
check7 = 52.77292 < lat && lat < 52.77551 && 18.22117 < lengthy && lengthy < 18.22724;
Coordinate pairs outline the workshop areas. A situation has been written within the laptop code to disable the flexibility to run a practice if it spends a minimum of 10 days in one in all these workshops. One of many workshops belongs to Newag itself – however a unique logical situation was outlined for its coordinates, presumably for testing functions.
Different surprises have been quickly found. Amongst them was the blocking of a practice when one in all its elements is changed (verified by its serial quantity). An choice to undo the lockout was additionally found – this didn’t require setting flags at laptop reminiscence degree, simply the suitable sequence of button clicks within the cab and on the on-board laptop display. When information of the profitable launch of the Impulse hit the media, the trains obtained a software program replace that eliminated this ‘repair’ possibility. On one other practice, a code was discovered instructing it to ‘break down’ after 1,000,000 kilometres.
Checking the date is tough
A reasonably humorous scenario was encountered on one other practice that refused to work on November 21, 2022, regardless that it was not in upkeep on the time. The pc reported that the compressor had failed, though the mechanics stated that every part was advantageous with the compressor. Sadly, the practice nonetheless didn’t increase the pantographs. Evaluation of the pc code detected a situation forcing the failure, which seemed like this:
- if the day is bigger than or equal to 21 and
- if the month is bigger than or equal to 11 and
- if the yr is bigger than or equal to 2021
then report a compressor failure. The scenario was humorous as a result of the practice had been scheduled for servicing in November 2021 (the yr earlier than the breakdown), however the situation didn’t work then by coincidence. The practice went into upkeep some time earlier, and was not put again into service till January 2022 – and the delicate logic situation described above now not met this date. Presumably, it was the software program developer’s lack of talent in setting up IFs that made it essential to attend till November 21, 2022 for the scheduled failure.
{Hardware} shock
Surprises lurked not solely within the software program of the computer systems. In a single practice set, researchers found a tool signed as a “UDP<->CAN converter”, presumably enabling distant communication with the practice. Eradicating it didn’t make something cease working. Evaluation confirmed that the on-board laptop was sending lock standing data to this system, and that the system itself was related to a GSM modem.
Not solely in Wrocław
Info that the SPS upkeep heart succeeded in repairing Newag’s ‘damaged’ trainsets rapidly discovered its technique to different firms as nicely. It turned out to be fairly a standard drawback. In Wrocław they analysed 13 Impulses, however there have been additionally damaged ones working in Kolej Mazowieckie (one), two in Opole, 4 in Krakow, one in Zielona Góra, 4 in Szczecin and one in Warsaw. Fortuitously, every was capable of be repaired utilizing a instrument developed by our researchers that removes software program locks from the on-board laptop. In complete, the researchers analysed the software program of 29 trains and in all however 5 they discovered surprises past the official working directions.
What’s subsequent
We depart it to the readers and prospects of this firm to evaluate the options utilized by the producer. Apparently, though there may be litigation within the case, it’s arduous to seek out an establishment in Poland that has carried out something past kindly expressing curiosity within the matter. We’re not conscious of any motion taken both by the Workplace of Client and Competitors Safety or by the Railway Transport Workplace, which might appear to be competent to remove from the market practices which can be damaging to native authorities organisations which can be incurring appreciable losses and to passengers who’re pressured to journey in crowds or use substitute transport for months. The one establishment that has taken motion that we’re conscious of is CERT Polska, notified of the invention by the researchers. From the remark we obtained, it seems that CERT Polska has notified the “related authorities” and the case is being dealt with by legislation enforcement companies.
We congratulate one of the best Polish hackers on their fascinating discovery {and professional} execution of the task. Keep in mind, nothing motivates you want a deadline tomorrow morning.
The above article is barely a cursory abstract of the presentation given on the Oh My H@ck convention on 5 December 2023 by members of the crew consisting of Jakub Stępniewicz, Sergiusz Bazański and Michał Kowalczyk. The article omits lots of element and an entire sizable technical part of the evaluation – it stays to be hoped that the authors of the examine will be motivated to put in writing it up and publish it.