Enjoyable with macOS’s SIP MetalBear ????
Whereas creating mirrord, which closely depends on injecting itself into different folks’s binaries, we bumped into some challenges posed by macOS’s SIP (System Integrity Safety). This put up particulars how we finally overcame these challenges, and we hope it may be of assist to different folks hoping to find out about SIP, as we’ve discovered the laborious method that there’s little or no written about this topic on the web.
What’s mirrord?
mirrord allows you to run an area course of within the context of a cloud service, which suggests we are able to take a look at our code on our staging cluster with out really deploying it there. This results in shorter suggestions loops (you don’t have to attend on lengthy CI processes to check your code in staging situations) and a extra secure staging setting (since untested companies aren’t being deployed there). There’s a detailed overview of mirrord in this weblog put up.
What’s SIP and why does mirrord care about it?
Apple launched SIP in 2015 to stop tampering with system binaries as a result of these often have excessive permissions and entitlements.
mirrord works by injecting its library (.dylib or .so) into the native course of, the one you wish to “mirror” into the cloud. In an effort to inject itself into binaries, it makes use of LD_PRELOAD on Linux and DYLD_INSERT_LIBRARIES on macOS.
One of many options of SIP is that it disallows the usage of DYLD_INSERT_LIBRARIES with protected binaries. Bummer – we are able to’t use mirrord to domestically run e.g. bash or ls in a cloud context. we are able to’t run mirrord exec bash or mirrord exec ls. We’ll need to discover a technique to get round SIP!
Detecting if a binary is SIP-protected
In an effort to begin bypassing SIP, we wanted to discover a technique to test if a binary is even SIP protected to start with. We initially used fairly coarse heuristic, assuming {that a} binary is SIP-protected if it’s in considered one of these areas:
Nonetheless, we discovered that the “stat” operate can return a flag known as RESTRICTED which we learn might be associated, and determined to make use of that as a substitute. The code is kind of easy:
let metadata = std::fs::metadata(&complete_path)?;
if (metadata.st_flags() & SF_RESTRICTED) > 0 {
return Okay(SipStatus::SomeSIP(complete_path, None));
}
Shebang!
Detection was easy sufficient when mirrord was run straight on a SIP-protected binary. Nonetheless, we quickly ran right into a much less trivial (however frequent) state of affairs – an interpreter script that begins with a shebang pointing to a SIP-protected binary, for instance yarn/npm/pyenv. If we take a look at npm as an illustration, it factors to a file which begins with the next code:
#!/usr/bin/env node
require('../lib/cli.js')(course of)
On this instance it’ll execute env, which can execute node with the following line.
The issue? /usr/bin/env is SIP protected, that means it’ll strip our DYLD_INSERT_LIBRARIES then run node with out mirrord. Thanks for nothing SIP!
So we additionally wanted to test whether or not the file is a “shebang” file (i.e begins with #!), what file the shebang factors to, and whether or not that file is a SIP-protected binary.
Bypassing SIP
Now that we discovered a technique to detect whether or not we’re being run on a SIP-protected binary, we have to determine how one can bypass SIP and let mirrord load into the binary with DYLD_INSERT_LIBRARIES. When googling round, we discovered folks saying you possibly can bypass SIP by copying the binary to a different listing and re-signing it. We discovered that to be partially true.
Why partially? As a result of in the event you tried to do it on Apple Silicon (arm), it wouldn’t work. It is because starting with M1, macOS ships with arm64e binaries. The e signifies an arm64 extension that provides pointer authentication. It’s one other safety measurement added by Apple (kudos to Apple for having nice safety, too dangerous it impacts us).
We gained’t go into particulars about what pointer authentication does, however you possibly can learn extra about it here.
So why was this an issue? First, mirrord is written in Rust, which doesn’t assist compiling arm64e binaries. The opposite downside is that solely Apple-signed binaries can run with arm64e structure.
That is what occurs if we attempt the “outdated” trick:
➜ /tmp cp /usr/bin/env /tmp/env
➜ /tmp codesign -f -s – /tmp/env
/tmp/env: changing current signature
➜ /tmp /tmp/env
[1] 20114 killed /tmp/env
Killed! And it was so younger. 🙁
Recording utilizing Console (macOS’s in-built log viewer) whereas operating the binary reveals the rationale:
From Apple’s perspective, arm64e is preview solely, i.e the ABI can change they usually don’t desire a third social gathering constructing on prime of it, because it’s not assured to work. You’ll be able to allow operating third social gathering executables with arm64e ABI provided that you boot into restoration mode and alter the settings, which isn’t one thing we wish to ask our customers to do.
Dealing with arm64e
Initially we tried to transform the arm64e ABI into arm64 on the fly. Sure, folks aware of how this ABI works most likely assume we’re insane, however we have been optimistic.. and it really labored! for instance, in the event you take /usr/bin/env and simply change the file headers to say it’s arm64, you’d have the ability to re-sign it and run it usually! We really do it for our binaries to have the ability to load to arm64e binaries:
# from our launch.yaml https://github.com/metalbear-co/mirrord/blob/primary/.github/workflows/launch.yaml
- title: construct mirrord-layer macOS arm/arm64e
# Modifying the arm64 binary, since arm64e might be loaded into each arm64 & arm64e
# >> goal/debug/libmirrord_layer.dylib: Mach-O 64-bit dynamically linked shared library arm64
# >> magic bits: 0000000 facf feed 000c 0100 0000 0000 0006 0000
# >> After modifying utilizing dd -
# >> magic bits: 0000000 facf feed 000c 0100 0002 0000 0006 0000
# >> goal/debug/libmirrord_layer.dylib: Mach-O 64-bit dynamically linked shared library arm64e
run: |
cargo +nightly construct --release -p mirrord-layer --target=aarch64-apple-darwin
cp goal/aarch64-apple-darwin/launch/libmirrord_layer.dylib goal/aarch64-apple-darwin/launch/libmirrord_layer_arm64e.dylib
printf 'x02' | dd of=goal/aarch64-apple-darwin/launch/libmirrord_layer_arm64e.dylib bs=1 search=8 depend=1 conv=notrunc
It didn’t work for all binaries although (ls for instance) and once we began digging we came upon that there are numerous new options being utilized in arm64e, for instance particular relocations that comprise pointer authentication stuff. We determined to surrender on ABI conversion in the intervening time.
Fortunately, Apple ships fats binaries on each structure machines. Fats binaries are Apple’s title for Mach-O information containing two completely different binaries, every constructed for a distinct structure, so the runtime can resolve which one it’ll use. By default, it’ll select arm64e, however we are able to do one thing good with the x64 binary.
/bin/ls: Mach-O common binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/bin/ls (for structure x86_64): Mach-O 64-bit executable x86_64
/bin/ls (for structure arm64e): Mach-O 64-bit executable arm64e
The thought is that we take the binary we wish to load ourself into, extract solely the x64 binary (on arm), re-sign it, and run it. The one draw back right here is that we require Rosetta to be put in on the system and there’s a efficiency affect – however often system binaries are used for easy operations like env or bash (see the shebang case).
Placing all of it collectively
-
Detect SIP (Shebang/Restricted)
-
Patch
a. Extract x64 binary into a brand new file
b. chmod +x it
c. Signal it
-
Execute
/// Learn the contents (or simply the x86_64 part in case of a fats file) from the SIP binary at
/// `path`, write it into `output`, give it the identical permissions, and signal the brand new binary.
fn patch_binary<P: AsRef<Path>, Okay: AsRef<Path>>(path: P, output: Okay) -> End result<()> {
let knowledge = std::fs::learn(path.as_ref())?;
let binary_info = BinaryInfo::from_object_bytes(&knowledge)?;
let x64_binary = &knowledge[binary_info.offset..binary_info.offset + binary_info.size];
std::fs::write(output.as_ref(), x64_binary)?;
// Give the brand new file the identical permissions because the outdated file.
std::fs::set_permissions(
output.as_ref(),
std::fs::metadata(path.as_ref())?.permissions(),
)?;
codesign::signal(output)
}
Integration into mirrord took two steps:
-
When utilizing mirrord straight on a SIP-protected binary, do the patch
-
When utilizing mirrord on a course of, and that course of executes a SIP-protected binary, substitute it on the fly. This was achieved by having mirrord hook
execvewithin the course of
execve hook:
/// Hook for `libc::execve`.
///
/// Patch file whether it is SIPed, use new path if patched.
/// If any args in argv are paths to mirrord's temp listing, strip the temp dir half.
/// So if argv[1] is "/var/folders/1337/mirrord-bin/choose/homebrew/bin/npx"
/// Change it to "/choose/homebrew/bin/npx"
/// then name regular execve with the probably up to date path and argv and the unique envp.
#[hook_guard_fn]
pub(crate) unsafe extern "C" fn execve_detour(
path: *const c_char,
argv: *const *const c_char,
envp: *const *const c_char,
) -> c_int {
// Do unsafe a part of path conversion right here.
let rawish_path = (!path.is_null()).then(|| CStr::from_ptr(path));
let mut patched_path = CString::default();
let final_path = patch_if_sip(rawish_path)
.and_then(|s| match CString::new(s) {
Okay(c_string) => {
patched_path = c_string;
Success(patched_path.as_ptr())
}
Err(err) => Error(Null(err)),
})
.unwrap_or(path); // Proceed even when there have been errors - simply run with out patching.
let argv_arr = Nul::new_unchecked(argv);
// If we intercept args, we create a brand new array.
// execve takes a null terminated array of char pointers.
// ptr_vec will personal the vector that might be handed to execve as an array.
let mut ptr_vec: Vec<*const c_char> = Vec::new();
let final_argv = intercept_tmp_dir(argv_arr)
.map(|new_vec| {
ptr_vec = new_vec; // Be sure vector nonetheless lives once we go the pointer to execve.
ptr_vec.as_ptr()
})
.unwrap_or(argv);
// Name execve's default implementation
FN_EXECVE(final_path, final_argv, envp)
}
Bonus content material: Why is that this potential?
You could be asking your self, “If this can be a safety characteristic by Apple, why is it potential to only bypass it that method?” The reply is that we don’t bypass the safety characteristic, simply the issue it created for us. Apple working techniques have the idea of “entitlements”, that are definitions of which particular operations an executable is allowed to carry out, and which particular sources it ought to have entry to. Earlier than launched purposes can have entitlements, they should undergo some approval course of with Apple. Shared libraries get the entitlements of the host executable, so if we might load any library to any course of, a non-Apple-approved library might get pleasure from entitlements it shouldn’t have by loading into an entitled course of. That might be a reasonably straight ahead privilege escalation of that library’s code. SIP and the hardened runtime stop that from taking place.
After we, in our bypassing mechanism, copy an executable and resign it, it loses its entitlements. So it’s nonetheless assured that our dynamic library couldn’t run with any ungranted entitlements. The integrity of granted entitlements is preserved.
The lack of entitlements isn’t an issue for mirrord, as a result of we don’t anticipate to ever execute with mirrord any software that requires Apple entitlements. So we hand over the mirrored software’s entitlements (which don’t exist or should not wanted), so as to have the ability to load our library into that software.
Afterword
You’re welcome to take a look at the complete implementation in our GitHub repository, Be a part of our Backend Engineers Discord neighborhood, subscribe to our e-newsletter and naturally use mirrord!