Diving into Starlink’s Consumer Terminal Firmware

Introduction

Starlink is a satellite-based Web entry service supplied by House X. This service already counts greater than 1.5 million subscribers all world wide, utilizing the exact same infrastructure. Starlink depends on 3 parts:

• A consumer terminal, which communicates with the satellites, and on which many of the present analysis is targeted.
• A satellite tv for pc fleet appearing as a mesh community.
• A gateway that connects the satellites to the web.

Quite a few research [1, 2, 3] have already been carried out on the topic, primarily on the consumer terminal.
Throughout my 6-month internship at Quarkslab as a part of my Grasp’s diploma program on the University of Trento, I carried out the evaluation of Starlink by reverse-engineering its firmware and the varied protocols it makes use of.
On the finish of the internship, I gained a great data of how the system works internally and developed a set of instruments that might assist different researchers engaged on the identical matter.
These instruments will likely be described and printed together with this weblog publish.

To conduct this research, we analyzed two common Consumer Terminals model 2 (the spherical one) and a Consumer Terminal model 3 (the squared one) with root entry (researcher entry) which was supplied by SpaceX’s safety workforce towards the top of my internship.

Firmware overview

Step one was to dump the firmware of the system since it is not publicly accessible, and we did that because of a blog post by the COSIC analysis group at KU Leuven.
As soon as we acquired the firmware, we began inspecting the content material, making an attempt to grasp how the reminiscence was structured.
We additionally began wanting into the U-Boot model that was personalized by SpaceX, and it’s getting used as the ultimate bootloader stage (BL33) for the Consumer Terminal.
The U-Boot license requires any modification of its code to be printed with the identical license, therefore you will discover it on GitHub.

From the file include/configs/spacex_catson_boot.h we are able to see how the reminiscence is partitioned, right here is part of it:

+-----------------+ 0x0000_0000
| bootfip0 (1 MB) |
+-----------------+ 0x0010_0000
[...]
+-----------------+ 0x0060_0000
| fip  a.0 (1 MB) |
+-----------------+ 0x0070_0000
[...]
+-----------------+ 0x0100_0000
| linux a (32 MB) |
+-----------------+ 0x0300_0000
| linux b (32 MB) |
+-----------------+ 0x0500_0000
[...]

This enables us to separate the picture into small partitions and analyze every of them individually.
This script can assist you try this routinely.
From right here, we are able to additionally see that nearly each partition is current a number of occasions (e.g. linux a/b).
That is due to the software program replace process, which is able to overwrite the partition that’s not at present getting used in order that within the case of an error, there’ll nonetheless be the unique partition that’s recognized to be “right”.
An summary of the principle partitions might be seen within the following image.

Partitions Boot FIP and FIP include all of the bootloader phases that compose the safe boot chain, most of them are based mostly on the ARM TF-A project, which doesn’t include a GNU-like license, and the final one (BL33) is the U-Boot bootloader we talked about above.

Having a transparent thought of the boot course of is important to carry out the fault injection attack developed by the COSIC analysis group.
The boot chain follows the basic multi-stage safe boot carried out by ARM TF-A, within the following image you possibly can see an outline of it.

Boot phases come from completely different partitions of the eMMC and the primary, which represents the Root of Belief, comes from an inside ROM of the principle processor, they are often extracted from the partition pictures through the use of fiptool, from ARM TF-A.
On the finish of the boot course of, BL31 will reside in Exception Degree 3 (EL3) and act as a safe monitor, whereas the Linux kernel will run in Exception Degree 1 (EL1), within the regular world operating user-land functions in Exception Degree 0 (EL0).

Then, the linux partition, because the identify suggests, comprises the Linux kernel, its ramdisk picture and a few Flattened Gadget Bushes, one for each {hardware} model of the Consumer Terminal.
This partition might be unpacked through the use of dumpimage from the U-Boot mission, the ramdisk is a cpio picture and FDTs come within the type of Gadget Tree Blobs (DTBs) which might be “decompiled” to Gadget Tree Sources (DTSs) textual content with the Device Tree Compiler.
This partition additionally comes with some Error Correcting Code (ECC) data in it, you will have to take away it earlier than with the ability to unpack it.
The ECC mechanism is custom-made by SpaceX and you’ll perceive the way it works by wanting on the code in U-Boot which handles this verification, the subsequent Section explains the way it works and gives a instrument to do this.

The sx (SX Runtime) partition comprises configuration recordsdata and binaries which can be particular to the Consumer Terminal.
This partition will likely be mounted by the Linux’s init script on /sx/native/runtime and after that binaries on this quantity will likely be began.
On this case, integrity verification is finished with sxverity which is one more {custom} instrument by SpaceX.
The subsequent part will clarify how this works.

Different partitions embody some encrypted ones, utilizing the Linux Unified Key System (LUKS), that are the one ones with the write permission, and another smaller partitions that aren’t value mentioning.

Information integrity

Here you will discover a easy Python script that strips out ECC data from a file, with out checking the correctness of the information, that we used to have the ability to unpack the FIT picture. Contained in the ramdisk, there’s a binary (unecc) that does the identical factor additionally checking and making an attempt to right doable errors.

The content material of ECC-protected recordsdata is organized into blocks of various varieties, the above determine reveals how every block is structured.
The file begins with a header block (a), which comprises the magic quantity and the model of the protocol, together with some knowledge and the corresponding management codes.
Then, there might be zero or extra knowledge blocks (b) containing simply knowledge and management codes.
A final knowledge block (c), which is acknowledged by its block kind area ($, as a substitute of *), marks the top of the payload, some padding is added right here if wanted.
Lastly, the footer block (d) comprises the scale of the payload (wanted to know the variety of padding bytes), an MD5 checksum of the entire payload and, after all, ECC code phrases for the footer block itself.

sxverity

sxverity is a {custom} wrapper for the device-mapper-verity (dm-verity) kernel function, which gives clear integrity checking of block gadgets.
The supply code of the instrument is just not publicly accessible, thus we needed to reverse the compiled binary to grasp the internals.
This gives each knowledge integrity and authenticity because of a signature examine that verifies the entire content material of the system.
sxverity internally makes use of the dm-verity kernel function, by instantly interacting with it by means of the /dev/mapper/management system.

SpaceX solely tackled the verification of the foundation hash, every part beneath, which is dealt with by the kernel, has not been reimplemented, a pleasant clarification of how this works might be discovered here.

As now we have seen in earlier sections, sxverity is used to confirm a number of the partitions that reside within the persistent reminiscence, that is to stop persistent exploits.
However as we’ll see within the subsequent sections, additionally it is used to confirm software program updates for the dish. Thus, it’s a essential part for the general safety of the system.

Within the image above you possibly can see the construction of a sxverity picture.
It’s composed of a header, that’s repeated 4 occasions, probably signed with completely different public keys, which comprises:

• The magic bytes "sxverity".
• The model and a few flags indicating which algorithms have been used for signing and hashing.
• The foundation hash, which not directly covers the entire payload (by means of the hash tree).
• The general public key that has been used to signal the picture.
• The signature of all of the fields above, utilizing an elliptic curve (ED25519).

The parsing and verification process carried out by this course of will likely be described within the Fuzzing part.

Runtime overview

On this part, we’ll focus on what occurs after the bootloader chain, ranging from the Linux’s init script to the processes that deal with the runtime of the Consumer Terminal.

The init script is the primary course of began by the kernel and normally has a course of identifier (PID) equal to 1.
Its most important activity is to start out all the opposite runtime processes wanted for the system to be helpful, and stays operating till the system is shut down. It is going to be the “oldest” ancestor of every other course of and additionally it is utilized by the consumer to start out, cease and configure daemons as soon as the system is up and operating.
The most typical init script you will discover in an end-user Linux distribution is systemd, which is a group of instruments to handle the entire runtime of the system (e.g. systemctl), amongst which there’s additionally the init script.

SpaceX’s folks prefer to implement their very own software program, so that they carried out their very own init script, which might be discovered within the ramdisk, at /usr/sbin/sxruntime_start. This makes use of a {custom} formatted configuration file that comprises the directions of which processes to start out, easy methods to begin them and during which order.

#######################################
# consumer terminal frontend
#
# Wait till dish config partition is ready up earlier than launching.
#######################################
proc            user_terminal_frontend
apparmor        user_terminal_frontend
start_if        $(cat /proc/device-tree/mannequin) != 'spacex_satellite_starlink_transceiver'
startup_flags   wait_on_barrier
consumer            sx_packet
custom_param    --log_wrapped

The snippet above reveals how a course of known as user_terminal_frontend is began:

• It’s began provided that the situation $(cat /proc/device-tree/mannequin) != 'spacex_satellite_starlink_transceiver' is happy.
• It’s began after the final course of marked with the barrier flag has exited.
• It’s executed because the Linux consumer sx_packet.
• The command line parameter --log_wrapped is handed to it.

The 2 configuration recordsdata which can be parsed by the init script might be discovered at /and many others/runtime_init (ramdisk) and /sx/native/runtime/dat/widespread/runtime (runtime picture).
The primary one handles system-level providers and configurations, similar to mounting partitions (e.g. the runtime), and organising the community, the console and the logger service.
The second, as a substitute, handles extra high-level processes and configurations, similar to beginning all of the processes contained within the runtime picture and initializing encrypted gadgets.

Moreover, the init script additionally assigns priorities and particular CPU cores to these processes, by following some guidelines listed in one other configuration file, that may be discovered at /sx/native/runtime/dat/widespread/runtime_priorities.

File system & mount factors

Initially, the foundation filesystem is copied in RAM by the final bootloader (U-Boot, BL33), and is mounted at /.
Partitions of the eMMC might be accessed by the system by means of the recordsdata /dev/blk/mmcblk0pN the place mmcblk0 is the identify of the eMMC and N is the partition index, ranging from 1.
For comfort, a script will then create some symbolic hyperlinks to those partitions to make use of a extra express identify, as proven under.

# [...]
ln -s /dev/mmcblk0p1 /dev/blk/bootfip0
ln -s /dev/mmcblk0p2 /dev/blk/bootfip1
ln -s /dev/mmcblk0p3 /dev/blk/bootfip2
ln -s /dev/mmcblk0p4 /dev/blk/bootfip3
# [...]

Since nearly each partition is duplicated, one other script will then create further hyperlinks within the folders /dev/blk/present and /dev/blk/different, the primary one containing the partitions which can be at present being utilized by the system and second one containing the opposite ones, that will likely be utilized in case of a software program replace.
The system is aware of which partitions have been used for the present boot by wanting in /proc/device-tree/chosen/linux_boot_slot which is populated by the bootloader.

Then, the runtime partition is unpacked utilizing sxverity and the content material is extracted to /sx/native/runtime.
This partition comprises two folders:

• bin comprises binaries and executable scripts.
• dat comprises loads of further knowledge like AppArmor guidelines, hardware-specific configurations and generic configuration recordsdata, similar to dat/widespread/runtime which is the (second) configuration file utilized by the init script.

After these operations, the foundation filesystem is remounted with the ro (read-only) flag.
Further partitions are then mounted, similar to:

• /dev/blk/present/version_info on /mnt/version_info (by means of sxverity).
• /dev/blk/dish_cfg on /mnt/dish_cfg (by means of LUKS).
• /dev/blk/edr on /mnt/edr (by means of LUKS).

Daemons

After the flowery boot process and system configuration we lastly attain a state during which some processes are operating, every one with a novel activity, to offer the consumer with the service the system is constructed for, i.e. a satellite tv for pc web connection.
As you might have guessed, many issues occur within the background to make sure a steady sufficient web connection similar to sending and receiving site visitors to and from satellites, selecting which satellite tv for pc to connect with, altering satellite tv for pc (with out disrupting the Web connection) when the present one moved too far, dealing with consumer requests coming from the cellular utility, and many others.
Moreover, these processes want to speak repeatedly with one another to synchronize and cooperate, and with Starlink’s cloud providers for backend functionalities.

A lot of the binaries have been carried out in C++ and a few of them are additionally statically linked.
On account of this, it was difficult to reverse-engineer these binaries and as a result of time constraints, we have been unable to completely comprehend all of them.
Some work was achieved to establish statically linked library features, utilizing the binary diffing method.
Additionally it is possible that these applications have been carried out utilizing the State Machine Design Pattern, which makes heavy use of options of Object Oriented Programming similar to a number of inheritance, digital strategies and generic varieties.
This made the reverse-engineering course of much more tough because of the advanced constructions which can be produced by the compiler when utilizing these options.

We tried to check the community stack of the Consumer Terminal with the recognized ISO-OSI stack, and the next could possibly be a correct mapping:

• phyfw (Bodily Firmware, maybe) handles the bodily layer of the satellite tv for pc communication, which incorporates modulation/demodulation of RF indicators.
• rx_lmac and tx_lmac (rx/tx decrease Medium Entry Management, maybe) fall within the knowledge hyperlink layer, and deal with the bodily entry to the medium, individually for receiving and transmitting.
• umac (higher Medium Entry Management, maybe), may characterize the community layer. It handles the entry to the medium, at the next stage, and coordinates between the transmission and reception of frames. It could even be in command of selecting which satellite tv for pc to connect with.
• connection_manager may characterize the transport layer and, if it is the case, it handles stateful connections between the dish and satellites, during which the site visitors will likely be exchanged.
• ut_packet_pipeline might be used to create an encrypted tunnel during which consumer site visitors will likely be exchanged utilizing the safe aspect on the dish for handshakes. This could possibly be related to the recognized protocols similar to TLS, DTLS, IPsec or, once more, a {custom} one.

Aside from these network-related processes there are also some processes that deal with system telemetry, software program updates, system well being standing and outage detection/reporting and eventually a “management” course of that acts as an orchestrator for all the opposite processes.

However, one of many binaries, specifically user_terminal_frontend, is carried out in Go, an open-source (compiled) programming language from Google.
Go binaries are statically linked they usually embody the go runtime, so they’re fairly huge, however fortunately in addition they embody symbols which can be utilized by the runtime for complete runtime error reporting, which incorporates operate names, supply code line numbers and knowledge constructions.
All this valuable data might be recovered utilizing a plugin for Ghidra known as GolangAnalyzer which was fairly efficient.
The extension additionally recovers advanced knowledge varieties and creates the corresponding C-like constructions in Ghidra, which is extraordinarily helpful when working with an OOP language.
Further guide evaluation is required due to the {custom} calling conference utilized by Go, however after this, the ensuing disassembled C code is well readable.
Our major focus was on the runtime’s higher-level parts, which embody this course of.

To summarize this part, within the image above you possibly can see a sketch of the structure of the runtime (not full), in which you’ll see that processes on the backside, “nearer” to the {hardware}, are statically linked, in all probability for efficiency causes, and talk solely with the management course of, whereas the opposite ones additionally talk with Starlink’s cloud providers, by means of gRPC.
In part Communications, we’ll sort out all of the communications proven by this image in additional element.
And at last, there may be the go binary (which is technically statically linked as effectively, however simply due to the language constraint), which communicates with frontend functions utilized by the consumer.

Runtime emulation

As a result of unfavorable outcomes of our Fault injection assault, we did not have entry to a dwell system on which to examine our findings or carry out some dynamic evaluation.
Thus, we tried to arrange an emulated setting, as related as doable to the true system, that may be able to executing runtime binaries.
We’re emulating the whole system (full-system emulation), ranging from the kernel, utilizing QEMU as an emulator engine.
Within the following paragraphs we describe each problem we needed to take care of, together with those we could not clear up, whereas organising the setting and the ultimate outcome.

The primary option to be made is which {hardware} we would like QEMU to emulate.
If you need to emulate an IoT system utilizing QEMU you normally search for the {hardware} implementation of the actual system for QEMU, which is normally accessible for widespread, off-the-shelf gadgets similar to Arduino, Raspberry Pi, and less-known boards as effectively.
The {hardware} implementation of our system was, after all, not accessible, so we used the (aarch64) virt machine, which is essentially the most generic one.
The correct strategy to emulate the entire system would have been to assemble this machine specification for QEMU, in addition to implement emulators for each piece of {hardware} that’s current on the board.
The issue is that many of the peripherals current on the system will not be open {hardware}, and even when they have been, implementing all of them in QEMU would have been loads of work.
As a substitute, utilizing the virt machine and tweaking the Gadget Tree is way simpler, at the price of not having many of the {hardware} peripherals and thus some limitations.

One other downside was the selection of the kernel to run in QEMU.
We tried with the unique one, extracted from the FIT of the firmware, however that did not work within the emulated setting. So we determined to compile one ourselves.
Sadly, the open-source model of Linux printed by SpaceX is 5.10.90, whereas the one discovered on the dish was 5.15.55, so we used the mainstream Linux kernel.
Numerous tweaks within the compile-time configuration needed to be made for it in addition, a few of them required by QEMU and a few of them required by Starlink’s software program.
It’s doable to extract this configuration from a compiled Kernel picture, utilizing the script extract-ikconfig from the Linux kernel repository, which was used to search out variations between the default one and the one configured by SpaceX.

The system tree not solely comprises details about {hardware} peripherals but in addition knowledge that’s utilized by the runtime, similar to public keys utilized by sxverity.
Moreover, the U-Boot bootloader additionally populates the FDT earlier than booting the Linux kernel, by including, for instance, which set of partitions have been used within the present boot, the identify of the principle community interface and extra.
All this data is after all not included within the FDT set by QEMU for the virt machine, thus we extracted this FDT, utilizing the dumpdtb flag, and added the lacking data, as proven under, which may then be recompiled utilizing the Gadget Tree compiler (dtc) and given to QEMU utilizing the -dtb flag.

# ...
mannequin = "spacex_satellite_user_terminal";
appropriate = "st,gllcff";

chosen {
    linux_boot_slot = "0";
    ethprime = "eth_user";
    board_revision = "rev2_proto3";
    bootfip_slot = <0x00>;
    boot_slot = <0x0006>;
    boot_count = <0x0010>;
    # ...
};

safety {
    dm-verity-pubkey = <REDACTED>;
    # ...
};
# ...

As the foundation filesystem, we used the one we extracted from the dish, with some modifications.

• Since we need to have entry to the emulated dish, it should assume it’s a improvement model in order that password entry is enabled, thus we patched the is_production_hardware script. This might have been achieved in a number of methods, similar to instantly modifying the /and many others/shadow file, or including our public key to the SSH’s authorized_users recordsdata, however what we did is simpler as a result of emulating improvement {hardware} may also allow different debugging options.
• We additionally included the extracted runtime the place it will have been mounted and eliminated the integrity verification and mounting step from the /and many others/runtime_init file to have the ability to additionally tamper with the contents of that partition.
• Within the /and many others/runtime_init file we additionally added some {custom} steps, for instance, one which units up our emulated community, and one which mounts read-write partitions as emulated volumes.

Further patches will likely be wanted for different applications to start out within the emulated setting.
We have now additionally included some further software program that will likely be used for testing functions similar to gdbserver. However for these applications to have the ability to run, we both needed to cross-compile them utilizing the identical construct toolchain, or cross-compile them statically.

Regardless that each the foundation filesystem and the runtime are already positioned in reminiscence when the Linux kernel boots, loads of processes instantly entry some partitions of the eMMC.
So we have additionally instructed QEMU to create a uncooked digital block system, containing the unique picture dumped from the bodily board.
Nonetheless, since it isn’t seen by the kernel as an eMMC chip, the assigned identify is completely different from the one assigned by the bodily system.
Due to this, we needed to change each reference to mmcblk0 to vda, which is the identify assigned by the kernel within the emulator.
Thankfully, as we noticed within the earlier part, the system solely makes use of the identify of the system in a script that creates some symbolic hyperlinks to each partition, so we simply needed to patch that script and the command line argument for the kernel.
Partitions which can be mounted with write permission, are as a substitute mapped to a folder on the host in order that it’s doable to examine their content material afterward.

As for the community, it isn’t crucial to copy the precise community configuration of the dish (as it isn’t very clear to us), we simply have to have the appropriate interface names and web entry.
This was achieved through the use of a faucet interface, bridged to the host which acts as a NAT gateway, as proven within the following scripts.

Host:

#!/bin/bash

ifconfig $1 0.0.0.0 promisc up
brctl addbr virbr0
brctl addif virbr0 $1

ip hyperlink set dev virbr0 up
ip hyperlink set dev $1 up

ip addr add 192.168.100.2/24 dev virbr0
echo 1 > /proc/sys/internet/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o wlp0s20f3 -j MASQUERADE

Visitor:

#!/bin/sh

ip hyperlink set dev eth0 identify eth_user
ip hyperlink set dev eth_user up
ip addr add 192.168.100.1/24 dev eth_user

route add -net 0.0.0.0 netmask 0.0.0.0 gw 192.168.100.2 dev eth_user

Emulation Outcomes

As defined in earlier sections, many of the {hardware} is just not current within the emulated setting, so each part that tries to make use of it should fail.
Decrease-level processes, similar to phyfw and [rx/tx]_lmac, whose most important activity is to work together with Starlink’s {hardware} will not work on this setting.
But additionally different binaries require some {hardware} to be current, the commonest one is the safe aspect, which is utilized in many of the cryptographic exchanges.
So for these binaries to work we patched each instruction that may make this system crash, but when the {hardware} is required for substantial elements of the method, this resolution is pointless.
In the long run, we managed to emulate, among the many most important processes mentioned within the Daemons subsection, solely user_terminal_frontend, starlink_software_update, umac and those associated to telemetry, together with smaller processes.
Additional work on this matter may incrementally add help for some peripherals of the board, therefore with the ability to emulate an increasing number of processes.
Right here you possibly can see the console output of the ultimate part of the boot course of within the emulated setting.

# [...]
# kernel messages and (loads of) errors from functions
# making an attempt to entry lacking {hardware} peripherals
# [...]

setup_console: Improvement login enabled: sure

SpaceX Consumer Terminal.
user1 login: root
Password: 

                                                                *
                                                 +               
                                       +    +                    
                                +     +                          
                           +      +                              
+ + + + +              +     +                                   
  +        +       +     +                                       
     +       + +      +                                          
        +   +      +                                             
          +      + +                                             
      +      +        +                                          
   +       +    +        +                                       
 +       +         +        +                                    
+ + + + +             + + + + +                                


The Flight Software program does not log to the console. If you want to view
the output of the binaries, you can use:

tail -f /var/log/messages

Or view the viceroy telemetry stream.

[root@user1 ~]#

All of the scripts and recordsdata offered above can be found here, however we cannot publish the entire UT’s firmware, so you may nonetheless must extract it your self to run the emulator.

Communications

The communication with front-end functions is dealt with by the method user_terminal_frontend, which we have been in a position to each run within the emulated setting and reverse-engineer, because of the language it has been carried out in (Go).
From the front-end functions, the consumer can see some statistics of the system and might change some high-level settings similar to Wi-Fi credentials, reboot or stow the dish, and many others.
These interactions use gRPC (Google’s Distant Process Calls), which in flip makes use of protobuf beneath.
Protobuf definitions might be gathered both by extracting them from the binary (utilizing pbtk) or by asking the reflection server of the method itself (utilizing grpcurl, for instance).
Some tools have been carried out as various front-end functions that use this protocol.
The aforementioned functions, carried out in Python, use the gRPC APIs uncovered by the frontend binary, offering the consumer with an alternate consumer interface to examine the statistics of the dish.
The authors of those functions in all probability gathered the protocol definitions from the cellular utility or through the use of the reflection server.

Within the following code snippet, you possibly can see the (partial) definition of the Request message, which comprises just a few ids and one particular request, among the many ones listed.
Each internal request has its definition, with the parameters the server must course of the request, and a corresponding response that can maintain the outcome.

message Request {
    uint64 id = 1;
    uint64 epoch_id = 14;
    string target_id = 13;

    oneof request {
        GetNextIdRequest get_next_id = 1006;
        AuthenticateRequest authenticate = 1005;
        EnableDebugTelemRequest enable_debug_telem = 1034;
        FactoryResetRequest factory_reset = 1011;
        GetDeviceInfoRequest get_device_info = 1008;
        GetHistoryRequest get_history = 1007;
        GetLogRequest get_log = 1012;
        GetNetworkInterfacesRequest get_network_interfaces = 1015;
        GetPingRequest get_ping = 1009;
        PingHostRequest ping_host = 1016;
        GetStatusRequest get_status = 1004;
        RebootRequest reboot = 1001;
        SetSkuRequest set_sku = 1013;
        SetTrustedKeysRequest set_trusted_keys = 1010;
        SpeedTestRequest speed_test = 1003;
        SoftwareUpdateRequest software_update = 1033;
        DishStowRequest dish_stow = 2002;
        StartDishSelfTestRequest start_dish_self_test = 2012;
        DishGetContextRequest dish_get_context = 2003;
        DishGetObstructionMapRequest dish_get_obstruction_map = 2008;
        DishSetEmcRequest dish_set_emc = 2007;
        DishGetEmcRequest dish_get_emc = 2009;
        DishSetConfigRequest dish_set_config = 2010;
        DishGetConfigRequest dish_get_config = 2011;
        // [...]
    }
}

There are two methods to speak with this gRPC, both through the use of an insecure channel or through the use of a safe channel, which includes TLS and mutual authentication utilizing certificates saved within the safe aspect.
The cellular utility and the online interface each use the insecure channel, so the encrypted one have to be utilized by one thing else.

Among the many requests that may be made to the server, a lot of them are meant for front-end functions, just a few examples are:

• FactoryResetRequest, which requests a manufacturing facility reset of the dish.
• GetDeviceInfoRequest, which returns some details about the system.
• GetStatusRequest, which requests the standing of the dish.
• RebootRequest, which asks the dish to reboot.

However some requests don’t appear to be they’re utilized by these functions, similar to:

• SetTrustedKeysRequest, which supposedly units the supplied public keys for future use by the method or by the SSH agent.

message SetTrustedKeysRequest {
    repeated PublicKey keys = 1;
}

• GetHeapDumpRequest, which supposedly returns a dump of the Heap part of the method.
• SoftwareUpdateRequest, which supposedly initiates a software program replace with the supplied replace bundle.

Sadly, most of those requests will not be carried out within the binary we have been analyzing (e.g. SetTrustedKeysRequest, GetHeapDumpRequest), and a few of them require authentication (e.g. DishGetContextRequest, DishGetEmcRequest) each on the transport layer (safe gRPC channel) and utility layer (through the use of the AuthenticateRequest).
We aren’t fully positive who is meant to make use of these requests and why most of them will not be carried out within the binary, they could possibly be utilized by one other Stalink product such because the Wi-Fi router, or by Starlink help within the case of {a partially} bricked system, for distant help.
Probably the most fascinating request that’s each carried out and doesn’t require authentication is the SoftwareUpdateRequest, which will likely be defined higher in Part Fuzzing.

Additional work on this matter can be to research additional each request handler for bugs or, higher to fuzz them since there exist efficient mutators for the protobuf protocol, similar to libprotobuf-mutator.

Inter-Course of Communication

Each course of operating within the runtime repeatedly shares data with different processes, to collaborate and share statistics.
As you possibly can see from the determine displaying the runtime structure, each course of solely communicates with the Consumer Terminal Management, which acts as an orchestrator for the entire runtime.
The protocol they’re utilizing is designed by SpaceX and it is known as Slate Sharing.

It makes use of UDP for transport, and each course of begins listening on a distinct port, on the loopback interface and can obtain messages from the management course of on that port.
However, the management course of begins listening on a number of ports, one for each course of that should talk with it, in order that the communication is bidirectional and with out conflicts between processes.
Port numbers are configured by means of a configuration file that may be discovered at /sx/native/runtime/widespread/service_directory, within the following snippet, you possibly can see part of it, which lists the port numbers for communications between software program replace and management, and between frontend and management.

################################################################################
# Software program replace
software_update_to_control                localhost   27012       udp_proto
control_to_software_update                localhost   27013       udp_proto

################################################################################
# Frontend
control_to_frontend                       localhost   6500        udp_proto
frontend_to_control                       localhost   6501        udp_proto

Each couple of processes change completely different varieties of knowledge, and messages don’t include any details about the content material nor the construction of the information they transport, not like different protocols similar to JSON or XML.
Thus, to grasp the content material of messages we needed to reverse engineer one of many binaries that makes use of the protocol, and in our case, the only option was as soon as once more the consumer terminal frontend, which is the one carried out in Go.

Information is exchanged in binary type, by sending uncooked packed (no padding) C constructions, in big-endian.
Each message comprises a header, holding some details about the message, and a physique, containing the precise knowledge to be shared.
Within the following snippet, you possibly can see the information construction representing the message header, from the GolangAnalyzer Ghidra plugin.
With the identical method, we additionally extracted the construction of the physique of messages between the frontend course of and management.

**************************************************************
* Identify: sx_slate.SlateHeader                                 *
* Struct:                                                    *
*   +   0x0    0x4 uint BwpType                              *
*   +   0x4    0x4 uint Crc                                  *
*   +   0x8    0x8 longlong Seq                              *
*   +  0x10    0x4 uint Body                                *
**************************************************************    

The header comprises:

• BwpType, which is a hard and fast worth, acts as a “magic quantity” for the protocol (00 00 01 20).
• Crc, whose identify suggests it’s a Cyclic Redundancy Test, so a form of error-detecting code for the message physique, however by reverse engineering and sniffing messages, this area appears to be mounted as effectively, however it’s completely different for each couple of messages.
• Seq, which is a sequence quantity that’s incremented each message, however the protocol doesn’t embody any acknowledgment mechanisms to resend misplaced messages.
• Body, which is utilized in case of fragmentation, i.e. when the message is greater than the MTU (Most Transmission Unit), which is normally set to 1500 bytes. On this case, the physique of the message is cut up into a number of frames, every one having an equivalent header, aside from the Body area, which is incremented every body, ranging from 0.

The physique of the message is encoded in the identical method, for instance, within the following snippet, you possibly can see a part of the construction of messages despatched by the frontend course of to manage.

**************************************************************
* Identify: slate.FrontendToControl                              *
* Struct:                                                    *
*   +   0x0    0x1 bool AppReboot                            *
*   +   0x1    0x1 bool TiltToStowed                         *
*   +   0x4    0x4 uint StartDishCableTestRequests           *
*   +   0x8    0x1 bool IfLoopbackTest                       *
*   +  0x10    0x8 longlong Now                              *
*   +  0x18    0x8 longlong StarlinkWifiLastConnected        *
[...]    

Because of this data we’d already have the ability to implement a message decoder (which we did), however this could solely work for the communication between the frontend course of and management. To decode different communications we would want to manually reverse engineer each different binary to search out out the construction of messages, maybe with out even discovering area names, however as defined in earlier sections, it was laborious to get helpful data from C++ binaries.
Right here you possibly can see how we parse the header of Slate messages in Python, utilizing the ctypes bundle.

class SlateHeader(BigEndianStructure):
    _pack_ = 1
    _fields_ = [
        ('BwpType', c_uint32),
        ('Crc', c_uint32),
        ('Seq', c_longlong),
        ('Frame', c_uint32)
    ]

Then, to grasp how the protocol is dealt with in C++ binaries, we regarded for some fields of the constructions we knew (those of the frontend course of) within the management course of, which ought to have them with a view to decode these incoming messages.
We could not discover them within the binary, however we discovered one thing method higher, within the folder /sx/native/runtime/widespread there’s a set of configuration recordsdata, similar to frontend_to_control, which include the construction of each message exchanged by processes.
Here’s a snippet containing a part of the aforementioned configuration file.

# Slate share message from gRPC frontend course of to most important management course of.
app_reboot                          bool
tilt_to_stowed                      bool
start_dish_cable_test_requests      uint32
if_loopback_test.enabled            bool
now                                 int64
starlink_wifi_last_connected        int64
# [...]

With this, it’s a lot simpler to implement a extra generic decoder that parses these protocol definitions and decodes messages accordingly.
Such a instrument has been developed and will likely be mentioned within the subsequent part.

See Also

Sound Circle

Slate sniffer & injector

Slate messages are despatched very quick, thus it’s laborious to grasp what is going on with no correct visualization instrument.
That’s the reason we have carried out the Slate sniffer, a instrument that sniffs UDP packets on the loopback interface and decodes them on the fly, highlighting variations between consecutive messages.
Within the following determine, you possibly can see the general structure of the brand new instrument, which we’ll describe in additional element.
This instrument was carried out for the emulated setting however we designed it holding in thoughts that it will have to work additionally on the true dish.
For that reason, many of the work is finished within the Sniffer, which isn’t within the system, and all of the communications between the sniffer and the dish occur by means of SSH.

The primary part that’s used when beginning the sniffer, is the protocol definition parser, which is able to parse configuration recordsdata:

• service_directory to know which “providers” (i.e. message definitions) can be found and on which UDP ports they may talk.
• [P1]_to_[P2] for each accessible couple of processes P1 and P2, to know the format of messages that will likely be exchanged.

This part will create SlateMessageParser objects, that can later be used to decode messages.
The decoding is made utilizing the struct python bundle.

After this, TCPdump will likely be launched on the dish, by means of SSH, and can pay attention on the loopback interface, solely capturing UDP packets which have because the vacation spot port one of many ones discovered by the parser.
The output of TCPdump is piped to Scapy which is able to decode the packet, perceive which service it comes from, by studying the vacation spot port, and can then extract the UDP payload and move it to the message decoder.
When a message is parsed appropriately, will probably be saved within the Storage part, which on this case is an easy in-memory database, solely holding latest messages (configurable, based mostly on accessible reminiscence).
On high of all this, there’s a Flask server, exposing some APIs, to know which providers can be found, to know the schema of a message and, after all, to fetch messages.
We additionally carried out a entrance finish, as a easy net interface, which is proven within the following image.
From the present entrance finish, it’s doable to see messages in real-time, recognizing variations because of the highlighted adjustments, choosing which fields to indicate and filtering or ordering them.
The frontend module is well replaceable because of the uncovered APIs, thus extra advanced interfaces might be built-in into the sniffer, to examine the acquired knowledge in additional methods.

After having a strategy to see messages, we thought it will be fascinating to have the ability to inject {custom} messages, by modifying those we’re receiving or by creating new ones from scratch.
For that reason, we carried out the Slate injector, which shares many of the codebase with the sniffer.
The structure of this instrument is proven under.
Messages, to be acquired by processes, want to come back from the loopback interface, thus we can not ship them instantly from the Injector (which is exterior to the system).
For this reason the injector will begin a socat server on the dish, which is able to pay attention for UDP messages on the “exterior” community interface, and can then ahead them to the appropriate UDP port, by altering the supply handle to localhost.
Some API endpoints have been carried out to have the ability to inject messages from the entrance finish and the present interface allows you to edit and ship a message or create a brand new one.

With the ability to examine messages between processes helped us rather a lot in understanding what every course of does with out with the ability to absolutely reverse-engineer them.
Moreover, having the protocol definition and a simple strategy to inject messages, a pure improvement of the mission is to fuzz the protocol, which will likely be tackled in Part fuzzing.
The total code of the Slate sniffer, injector and fuzzer might be discovered here, however once more with out the protocol definitions, which you may must extract from the dish.

Fuzzing

CHUNK_SIZE = 16384

def replace(knowledge):
    channel = grpc.insecure_channel('192.168.100.1:9200')
    stub = device_pb2_grpc.DeviceStub(channel)

    stream_id = int.from_bytes(os.urandom(4), byteorder='little')

    for i in vary(0, len(knowledge), CHUNK_SIZE):
        chunk = knowledge[i:min(len(data), i + CHUNK_SIZE)]
        request = device_pb2.Request(id = 1, epoch_id = 1, target_id = "unknown",
            software_update = common_pb2.SoftwareUpdateRequest(
                stream_id = stream_id,
                knowledge = chunk,
                open = i == 0,
                shut = i + CHUNK_SIZE >= len(knowledge)
            )
        )

After the software program replace course of is notified {that a} software program replace bundle is able to be utilized, the replace process begins.
From this second, there is no such thing as a distinction between this type of software program replace and the usual one, which is downloading the replace bundle from Starlink’s backend.
The replace bundle is a sxverity picture, which will likely be verified by this system with the identical identify and the internal rom1fs filesystem will likely be mounted.
As soon as mounted, the software program replace course of will search for partition pictures within the mount level.
Each partition picture may also have a SHA512 hashsum for added integrity verification.
Lastly, every accessible partition picture will likely be flashed on the corresponding /dev/blk/different/* eMMC logical partition.

The replace bundle is just not accessed instantly by the software program replace course of, so the primary course of really studying the content material of the supplied enter is sxverity. Thus any fuzz check might be carried out instantly on that binary, skipping all earlier steps.
Within the following determine, you possibly can see how the verification course of is carried out by sxverity.
The fuzzable code may be very restricted as a result of the signature verification is made by a library, which is out of scope, and something taking place after a profitable signature verification is to be thought-about unreachable for us as a result of if we are able to attain that state, it means we have been in a position to craft an replace bundle that may be flashed so we need not discover different bugs there.

The one a part of the enter that will likely be parsed by the code underneath check is the header of the picture, thus that would be the solely a part of the enter that the fuzzer will mutate.
Since that a part of this system might be executed utterly in full isolation, we have fuzzed it inside unicorn, a light-weight CPU emulator that may be instructed from Python, through the use of its bindings.
Step one was with the ability to emulate the code we need to check and organising the harness for our fuzzer, which incorporates:

• Loading the binary.
• Figuring out a great start line within the code, during which it’s straightforward to arrange the entire setting, similar to inserting the enter in the appropriate reminiscence location and organising each different reminiscence construction that will likely be utilized by the code underneath testing.
  For example, the next snippet reveals how the enter is positioned in reminiscence and the way registers holding the addresses to the enter areas are set.

def place_input_cb(mu: Uc, content material: bytes, persistent_round, knowledge):
    content_size = len(content material)

    if content_size < INPUT_MIN_LEN:
        return False

    pubkey, header = key_and_header_from_data(content material)

    # write knowledge in reminiscence
    mu.mem_write(PUBKEY_ADDR, pubkey)
    mu.mem_write(HEADER_ADDR, header)

    # put together operate arguments
    mu.reg_write(UC_ARM64_REG_X2, PUBKEY_ADDR) # pubkey handle
    mu.reg_write(UC_ARM64_REG_X3, 0x40) # nblocks
    mu.reg_write(UC_ARM64_REG_X4, HEADER_ADDR) # header buffer handle

    return True

• Figuring out operate calls, hooking them and emulating them in Python, in order that we don’t spend time testing library code, nor do now we have to load them in reminiscence and deal with dynamically loaded libraries.
  For example, right here is how the libc operate memcpy is hooked and emulated.

if handle == MEMCPY_ADDR:
    # learn arguments from registers
    dest = mu.reg_read(UC_ARM64_REG_X0)
    src = mu.reg_read(UC_ARM64_REG_X1)
    n = mu.reg_read(UC_ARM64_REG_X2)

    # learn the information from src
    knowledge = mu.mem_read(src, n)
    # write knowledge in dst
    mu.mem_write(dest, bytes(knowledge))
    # return the handle of dest 
    mu.reg_write(UC_ARM64_REG_X0, dest)
    # leap to the return handle
    lr = mu.reg_read(UC_ARM64_REG_LR)
    mu.reg_write(UC_ARM64_REG_PC, lr)

• Figuring out an ending level, which must be a degree in this system during which we cease the emulation as a result of the run was profitable (no bugs).

The nicest factor about utilizing Unicorn, aside from being fairly straightforward to configure and instruct, is the flawless help with AFL++ (American Fuzzy Lop plus plus) which we used as fuzzer.
AFL++ working with Unicorn can detect crashes and most significantly, collect protection data, in a clear method, in order that it will probably carry out coverage-guided mutations and organising the fuzzer with Unicorn is fairly easy.
The fuzzer additionally wants some preliminary check instances, known as seeds, for that we have used some legitimate headers, taken from sxverity pictures discovered within the dish, and a few randomly generated headers.

The fuzzer ran for round 24 hours, performing multiple million executions, however sadly, no crash was recorded.
This was anticipated for the reason that examined codebase was very restricted and the construction of the enter was quite simple, not having advanced knowledge constructions or variable size fields, the commonest memory-related bugs are averted.

Slate messages

The opposite part we examined with fuzzing was Inter-Course of Communication (IPC) – which was deeply analyzed within the earlier part – since we already had developed a set of instruments to research and tamper with this communication.
On this case, we’re not going to fuzz a single binary, however quite the entire set of processes that type the runtime of the system, since each considered one of them is utilizing Slate messages to speak.
The fuzzing strategy was utterly completely different from the one we used for sxverity, which was gray-box fuzzing, as a result of:

• The codebase we are attempting to check is gigantic.
• We weren’t in a position to precisely establish the code that handles slate messages in each binary and, extra importantly, bugs might be additionally discovered outdoors this code, due to some inconsistencies in this system state attributable to wrongly interpreted inputs.
• Binaries have to run in a dish-like setting as a result of they repeatedly work together with different parts of the system, most of them do not even run in our emulated setting.
• Additionally recording protection would have been difficult, as a result of for that we have to instrument the binaries since we do not have the supply code to recompile them, and the fuzzer would have to be run on the dish.

For the above causes, we used black-box fuzzing, with out coverage-guided mutations, which is normally known as “dumb” fuzzing.
Boofuzz was used as fuzzer, it’s a easy and easy-to-use fuzzer particularly designed for community protocols, which was an ideal match for what we have been on the lookout for.
Boofuzz doesn’t generate enter in a very random method since you are giving it the protocol definition for use within the communication, with the potential of defining sequences of messages utilizing a finite state machine.
In our case, each message was disconnected from the others (aside from the sequence quantity), so defining the format of messages was sufficient.
The fuzzer will then mutate each area of the message, by making an attempt some values that might set off a bug, e.g. for an int32 the fuzzer will strive values similar to {0, 1, -1, INT_MAX, -INT_MAX, ...}.
For example, right here is how some fields of Slate messages are “translated” to Boofuzz protocol definition.

if param.dtype.identify == "BOOL":
    return Easy(
        identify=param.identify,
        default_value=b"x00x00x00x00",
        fuzz_values=[b"x00x00x00x00", b"x00x00x00x01"],
    )
if param.dtype.identify == "INT8" or param.dtype.identify == "UINT8":
    return Byte(identify=param.identify)
if param.dtype.identify == "INT32" or param.dtype.identify == "UINT32" or param.dtype.identify == "FLOAT":
    return DWord(identify=param.identify)

Each knowledge kind used within the Slate message protocol could possibly be encoded utilizing Boofuzz’s normal varieties, aside from the Sequence quantity, which must retailer an inside state to increment itself each iteration, you possibly can see its implementation within the following snippet.
Static fields such because the Bwptype and Crc might be encoded utilizing the Static kind from Boofuzz.

class SequenceNumber(Fuzzable):
    def __init__(self, *args, **kwargs):
        tremendous().__init__(*args, **kwargs, fuzzable=False, default_value=0)
        self._curr = 0

    def encode(self, worth: bytes = None, mutation_context=None) -> bytes:
        curr = self._curr
        self._curr += 1
        return int.to_bytes(curr, size=8, byteorder="huge")

As soon as the message construction has been outlined, the fuzzer can use the code from the Slate injector to ship the messages.
The one part that must be carried out at this level is one thing that may detect if a program has crashed after a message was despatched.
At first, we have been issuing the pgrep command by means of SSH, however this was including an overhead that was slowing the fuzzer.
So we have carried out a easy script that runs on the dish, opening a TCP socket and ready for a connection which is able to then be used to instantly talk with the fuzzer.
The a part of the method monitor that can run on the consumer (fuzzer machine) might be built-in into the fuzzer, by inheriting Boofuzz’s BaseMonitor and implementing its strategies, similar to alive (examine if the goal course of continues to be alive) and restart_target (restart the goal course of).
The ensuing structure is proven within the following determine.

Some crashes have been discovered by fuzzing the control_to_frontend protocol however none of them seemed to be exploitable in methods aside from merely crashing this system, inflicting a Denial of Service for the frontend functions.
It’s because the frontend course of is a Go binary and the Go runtime makes the method crash (by means of the panic operate) as a result of it detects that one thing fishy is happening.

For example, the next are the small print of considered one of these crashes.
Within the following snippet, you possibly can see a part of the stack hint produced by the Go runtime upon the crash, from which you’ll perceive that the crash is attributable to the operate UpdateObstructionMap, which tries to allocate an excessive amount of reminiscence.

deadly error: runtime: out of reminiscence

goroutine 5 [running]:
[...]
runtime.(*mheap).alloc(0x5046800?, 0x28234?, 0xd0?)
[...]
most important.(*DishControl).UpdateObstructionMap(0x40003be000, {0x7a90f8?, 0x4000580380?})
[...]

By inspecting additional this operate, we understood how the obstruction map is transferred to the frontend course of.
Initially, the obstruction map is a 3D map of the sky above the antenna, indicating whether or not the antenna has a transparent view of the sky or is obstructed by an impediment similar to a tree or different buildings, that the consumer can see from the frontend functions.
This map is just not produced by the frontend course of, thus it must be despatched to it by means of Slate messages.

obstruction_map.config.num_rows                                                 uint32
obstruction_map.config.num_cols                                                 uint32
obstruction_map.present.obstructed                                              bool
obstruction_map.present.index                                                   uint32

Within the snippet above you possibly can see a part of the message construction definition that carries details about the obstruction map.
The obstruction map is represented in reminiscence as a matrix, during which every level might be obstructed or not.
The management course of sends this data by sending one Slate message for every level within the matrix, by setting the appropriate index and setting obstructed to true or false.
The scale of the matrix is just not mounted, and its dimensions might be set by the management course of through the use of the num_rows and num_cols fields within the message.
That is the place the bug resides, in actual fact when sending huge values in these two fields, this system tries to allocate sufficient reminiscence for the matrix and panics because of this.

len = ObstructionMapConfigNumCols * ObstructionMapConfigNumRows;
if (len == (this->obstructionMap).snr.__count)
    goto LAB_0050b7f4;
(this->obstructionMap).numRows = ObstructionMapConfigNumRows;
(this->obstructionMap).numCols = ObstructionMapConfigNumCols;
puVar5 = runtime.makeslice(&datatype.Float32.float32,len,len);

The snippet above reveals the decompiled and annotated code of the frontend binary which handles the scale of the obstruction upon the reception of a slate message.
Line 1 computes the scale of the matrix and line 2 compares it with the present dimension of the matrix this system has in reminiscence, if the 2 differ then the size are up to date within the inside reminiscence construction on Traces 4 and 5, after which the brand new matrix is allotted utilizing the makeslice methodology of the Go runtime on Line 6.
As you possibly can see, no checks are carried out on the scale to be allotted, nor on the results of the multiplication between the 2 given dimensions.
This is able to be very harmful in C, however the Go runtime handles all of the nook instances routinely, by checking that the scale of the requested reminiscence is optimistic and never too huge.
The Go runtime additionally checks each array entry, in any other case, an arbitrary write would in all probability be doable by enjoying with the index and the scale of the matrix.

Notice that this bug can solely be triggered by sending crafted UDP packets to a service certain to localhost solely.
Due to this fact it isn’t doable to set off it from an exterior community.
Moreover, the iptables configuration of the UT filters out incoming UDP packets, so spoofing packets with a localhost supply IP wouldn’t work both.
Due to this fact we didn’t think about this a vulnerability however merely a bug.

After we carried out the fuzzer and used it within the emulator, we have been supplied a rooted UT by Starlink, then we confirmed the presence of the aforementioned bug on an actual system and fuzzed some extra processes that weren’t working within the emulator.

Conclusion

Yow will discover extra particulars in my grasp’s thesis which will likely be printed on the finish of this yr, keep tuned!
The offered instruments and script might be present in this repo.

This work and the instruments now we have printed are supposed to be reused for additional analysis on Starlink’s Consumer Terminal.
Sadly, as a result of some technical points and time constraints, we didn’t handle to completely examine the community stack and protocols used within the satellite tv for pc communications, however hopefully, this data base of the higher-level administration operate of the runtime can be utilized sooner or later to help in that effort.

I encourage analysis on this matter additionally as a result of SpaceX’s safety workforce is there that can assist you they usually provide some juicy bounties.

Many because of:

• Maxime Rossi Bellom, my internship tutor, for guiding me on this analysis.
• Lennert Wouters, who was the writer of the weblog publish relating to the dumping of the firmware and the fault injection assault of Starlink’s Consumer Terminal, for serving to us within the early phases of this analysis.
• Tim Ferrell, from SpaceX’s safety workforce, for sending us a testing dish with root entry.
• Ivan Arce, Salwa Souaf and Guillaume Valadon for reviewing my weblog publish.
• Many different wonderful colleagues for serving to me on matters of their fields of experience.

If you want to be taught extra about our safety audits and discover how we can assist you, get in touch with us!