Swift was at all times going to be a part of the OS // -dealloc

Just lately on the Swift Boards, somebody complained that placing Swift within the OS has solely made issues worse for builders. My fast response is a snarky “welcome to the world of libraries shipped with the OS”, however that’s not useful and in addition doesn’t refute their level. So right here’s a weblog publish that talks about how we obtained the place we did, protecting time after I labored on Swift at Apple. However I’m going to have to start out rather a lot earlier to clarify the issue…

OS Dependencies

In my earlier publish “Dynamic Linking Is Bad For Apps And Static Linking Is Also Bad For Apps”, I talked about how there’s a trade-off between static and dynamic linking for libraries you ship together with your app. However what about libraries which might be shipped with the OS? At that time, you might be invoking the first energy and curse of dynamic linking: the library you construct and take a look at towards might not be the library you run towards, at which level it’s important to be way more cautious about what you depend on.

At this level we will think about a spectrum, the place at one finish you utilize nothing from the host OS, and on the different you might be fully depending on it:

• The place to begin doesn’t actually exist anymore until you are the working system, however as soon as upon a time private computer systems labored by handing over management of your entire laptop to the working course of, which was allowed to do no matter it favored with the entire machine, reminiscence, no matter. Even for these applications the OS would possibly present “library capabilities” that both obtained loaded with this system or have been left in reminiscence at identified places when this system was loaded, however you may have a program that eschewed all of that and was principally simply an OS of its personal, with no dynamic dependencies in any respect.

• Today, even a program that’s “statically linked” nonetheless isn’t “in cost” of your entire machine. The working system kernel is at all times working as nicely, offering fundamental OS companies and exposing some kind of “system name” interface by which applications can request sure privileged operations (similar to studying a file, or allocating extra reminiscence pages). Most fashionable OSs let you produce applications that work this fashion, together with Home windows and Linux. Linux is the best-known fashionable OS that enables applications that work this fashion, the place the one factor you rely upon is the set of system calls out there to you.

• The commonest technique to program lately is to rely upon a minimum of some libraries shipped with the OS, even when it’s simply the C commonplace library and its minimal runtime. Right now’s Apple OSs have this because the “minimal system API boundary”: the system name interface is not steady throughout OS variations and so you might be “required” to make use of the C/POSIX commonplace library and its extensions to do even fundamental primitive operations. ^{EDIT: Home windows is comparable, although its “steady” interface is the library beneath the C commonplace library (thanks Slava and Kyle).}

• I don’t suppose you may actually declare that any program is on the different excessive, with all its behaviors coming from dependencies, besides in sure trivial examples. However from a sure viewpoint, each interpreted program works like this. A shell script doesn’t immediately execute any machine directions; it’s the shell that does that primarily based on what the script says. (Bytecode runtimes and JIT compilers blur the road right here too.)

So given this, we will see that applications can rely upon their host OSs to various levels, and that Apple specifically goes all-in on the “libraries” mannequin. Let’s take a better take a look at that:

Apple’s Mannequin (pre-Swift)

Earlier than Swift, almost all of Apple’s public APIs have been written in both C or Goal-C and supplied in compiled type as native code, reasonably than some form of bytecode (like JVM or CIL). New OS variations would come with new binary-compatible variations of current libraries, presenting a superset (in concept) of the earlier model’s APIs. Thus outdated apps would proceed to run, however new apps may reap the benefits of new options. “Weak linking” and the inherent name-based dispatch in Goal-C even allowed new apps to conditionally reap the benefits of new APIs however nonetheless work on older OSs. This ultimately grew to become formalized because the availability model, however that’s leaping forward.

The principle draw back of this mannequin is that new options and new APIs are tied to the brand new OS model. When you’ve got an app, and also you wish to use new API introduced alongside OS v9 even when working on OS v7, nicely, you may’t. Apple after all needs everyone to maneuver to the brand new OS promptly, however not everyone does; perhaps the brand new OS doesn’t assist their laptop, or perhaps it’s not suitable with an app they use, and so forth. So app builders are caught ready for sufficient folks to get on OS v9 that they will drop assist for OS v7 and v8 with out dropping income, person goodwill, no matter. And on the opposite aspect, Apple can’t change the habits of an current library with out doubtlessly breaking current apps, even when the present habits was buggy.

We’ll speak extra about alternate options later, however for now this units the stage: Swift was designed to not require adjustments to this mannequin. Libraries would nonetheless be compiled native code; new releases would nonetheless be binary-compatible with outdated ones. We accepted these as design constraints, in addition to interoperating tightly with Goal-C and never counting on a JIT.

Swift “betas” 1..<5

Between Swift 1 and Swift 5 there was a huge quantity of change, from shifting API design to altering ARC operate conventions to including lacking optimizations to studying what was going to be sensible and idiomatic on this new language. Looming over us the entire time was “ABI stability”, the purpose at which code utilizing two totally different variations of Swift may interoperate. Why was this necessary, when so many different languages didn’t appear to hassle? As a result of this was the very premise of Apple’s OS-based library distribution mannequin: apps compiled for Swift 5 would work with an OS constructed on Swift 6; apps compiled with Swift 6 would nonetheless be capable to “backwards-deploy” to an OS constructed on Swift 5. With out this, Apple couldn’t use Swift in its personal public APIs.

So this was at all times a purpose for Swift. And builders complained too: everybody who used Swift needed to bundle all its libraries in with their app, growing the app measurement by fairly a bit. As well as, these builders who made their very own closed-source libraries needed to launch a separate model for each new model of Swift, as a result of they weren’t essentially binary-compatible. However on the similar time, ABI stability means you may’t change issues anymore! (There are inefficiencies in Swift’s enum structure to at the present time that may’t be improved as a result of that will change the ABI.)

With Swift 5, we lastly reached some extent the place most issues have been fairly good, the place we had labored out find out how to deal with outdated apps working earlier variations of Swift, and the place we have been already maxing out on in poor health will from source-breaking adjustments. To not point out strain from inside Apple: this challenge had been occurring for years and but there have been nonetheless all types of restrictions.

The Transition

When it got here time to place Swift into the OSs, we had to determine find out how to do it with out breaking

• current constructed apps, which embedded their very own earlier model of Swift
• current initiatives, which wished to proceed to assist earlier variations of iOS

The primary drawback was solved by intentionally altering Swift’s ABI, in order that Swift 5 wouldn’t collide with something from Swift 1-4. Within the few locations the place outdated and new Swift wanted to work together—Goal-C—metadata for Swift 5 sorts was marked otherwise from Swift 1-4, in order that the older Swift would see newer courses as bizarre Goal-C courses. This was rather a lot to work out however in the end fairly easy.

The second drawback couldn’t be solved the identical method; we wished newer apps to work together with the system Swift APIs. So one way or the other we needed to proceed permitting delivery the Swift libraries with apps whereas additionally avoiding having two copies of Swift on new sufficient OSs. Definitely we wouldn’t need an app’s embedded Swift 5 library to supersede the next yr OS’s Swift 5.1! That might break elements of the next-year OS that trusted Swift 5.1.

We ended up (ab)utilizing a function known as “rpath”, or “runtime search path”, which allowed an executable to seek out its dynamic libraries not by hardcoded path however by looking out a collection of directories. By making the search order begin with /usr/lib/swift/ and following that with the app bundle, we may assure that apps would use the OS model of Swift if current and fall again to their embedded model in any other case.

I wish to be very clear right here: it is a approach for backwards-deploying the first model of an OS library. You may’t play the identical rpath trick with the second model, as a result of the primary model will already be current on disk. This similar approach was used simply final yr for Swift Concurrency, after which a critical bug was found in that first launch…and it may possibly’t be “fastened” within the backwards-deployment library as a result of that wouldn’t assistance on the OSs that also comprise libswiftConcurrency 1.0. Which in flip would create a discontinuity in assist:

…okay, bugs are launched after which fastened in OS updates on a regular basis, so perhaps there can be occasions when that is price it, however the principle level is it’s not a 100% answer, and for new APIs it breaks the supply mannequin.

What we misplaced

So, Swift is now within the OS! Apps not have to embed Swift! A bunch of efficiency enhancements might be made now that the Swift and Goal-C runtimes ship collectively! And at last, lastly, Apple can ship Swift APIs as a part of the OS.

And fairly shortly, exterior Swift contributors came upon what Apple’s framework engineers cope with yearly:

• To a primary approximation, you may by no means take away something, solely deprecate it, as a result of in any other case current apps would possibly fail to launch or lose user data.

• The discharge cycle is tied to the OS launch cycle. (Swift’s launch cycle had at all times been tied to Xcode’s, which was dangerous sufficient, however OSs are even greater “automobiles” for change and so it’s tougher to vary their course even in the event you actually suppose it’s essential.)

• And the principle immediate for this publish, new APIs are solely current within the new OS. (And so as to add insult to damage, they have to subsequently all be annotated with availability.)

This final is an issue that Go doesn’t have, that Rust doesn’t have, and that Swift 1-4 didn’t have, and it’s all for a similar cause: these languages ship(ped) their commonplace libraries with the app that makes use of them. They’re not a part of the OS, and consequently there’s method much less motivation to have a steady ABI and assist library evolution in any respect (they usually don’t).

We (Apple) tried to manage all these expectations on the time of the changeover. However it actually is a trade-off, and so within the years following Swift-in-the-OS Apple and the Swift workforce have provide you with a couple of strategies that assist some frequent backwards-deployment eventualities:

• Particular capabilities might be marked with a not-yet-supported attribute to have them copied into purchasers as a substitute of referenced by image title, very similar to a C static operate outlined in a header file. This solely works for issues like capabilities, strategies, and so on that don’t want id, although; if the identical implementation have been used with a worldwide saved variable, you’d find yourself with a number of copies of the worldwide. And it additionally solely works when the implementation is suitable with each previous and future variations of the APIs it makes use of in flip. (There have been precise bugs round this.)

• Some Apple libraries are intentionally not shipped with the OS; you may select to embed them in your app like third-party dependencies. ResearchKit was one of many earliest of those, made even earlier than Swift was a part of the OSs.

• In sure circumstances, the Swift workforce features a “compatibility” library that’s linked in to the principal app solely (in order that there’s precisely one) to carry among the older OS’s assist updated. These generally use “customization factors” within the runtime the place we knew issues would possibly change, and generally use terrible hacks counting on implementation particulars of the older runtime or OS (a observe that’s barely higher for Apple to do to itself reasonably than for exterior builders to do, as a result of Apple can promise itself it gained’t change an older OS model any additional).

None of those are good, nonetheless. There’s no commonplace mechanism to “backwards-deploy-if-needed” additions to the usual library and different Swift libraries within the OS (one thing that may very well be difficult if a number of targets in your app tried to be liable for it), and backporting modifications could also be even tougher. What in case your new function is a generic operate that is dependent upon a protocol that didn’t exist final yr? What if the protocol did exist however among the sorts you wish to use with the brand new function didn’t undertake it till this yr?

Swift was required to be a language that you would be able to outline OS libraries in, and that robotically makes a bunch of issues tougher. That is the place that knee-jerk response comes from: being an OS library developer is tougher than being a third-party library developer (or an OS-internal-only library developer!). However that’s one thing that needs to be understood as a trade-off, and it’s legitimate to weight the perimeters of the trade-off otherwise, particularly when among the negatives show up nearly every year given how Apple does issues.

Alternate options

There are some affordable issues Apple may have completed, or nonetheless may do, to enhance the state of affairs for builders. Will Apple do them? Perhaps, perhaps not.

Library Deduplication

When fifteen totally different apps in your telephone all use the identical open-source library, they’re all going to make use of it a bit otherwise. Perhaps they’ve totally different construct settings, perhaps they made native modifications. But when all of them use the identical closed-source library, and it’s a dynamic library, Apple may in concept deduplicate these libraries on disk, saving the top person some house. Code signing makes this trickier, however I don’t suppose it’s inconceivable, since (if I recall accurately) the App Retailer already re-signs all of the libraries in your app bundle. This undoubtedly applies to the Swift compatibility libraries, and I even keep in mind obscure discussions of implementing this, so it would even have occurred. However it wouldn’t change the listed obtain measurement for any app within the retailer (since you wouldn’t need that to vary primarily based on what else you could have put in, until maybe it’s from the identical developer), and for first-party libraries it solely ever applies when backwards-deploying, which robotically makes it much less fascinating to Apple.

This additionally doesn’t actually work on macOS, the place apps aren’t 100% managed by the OS, however macOS can be the Apple platform the place you’re least more likely to care about code measurement issues on a per-app foundation.

Express “polyfill” assist

“Polyfill” is a time period that arose within the land of internet growth to backport options to earlier browsers in addition to paper over variations between browsers. In JavaScript that is simpler as a result of you may examine whether or not a function is current earlier than putting in your personal implementation, one thing that you are able to do in Goal-C a bit clunkily, and that you would be able to’t do in Swift in any respect. (The distinction between “compile time” and “run time” makes this all extra difficult for ObjC and particularly for Swift.)

Extra frequent in ObjC and Swift is including a way that appears like this:

See Also

Peeter Joot’s Weblog » Geometric Algebra for Electrical Engineers.

func backportFoo() -> String {
  if #out there(iOS 16, *) {
    return foo()
  }
  // Fallback implementation
  return "foo: (self)"
}

This will get trickier if you’re wrapping a kind reasonably than an operation, however it’s normally nonetheless attainable with a protocol, or generally a base class the place one implementation defers to the OS and the opposite is for backporting. It might be good™ if Swift explicitly supported this, however it’s an extraordinarily difficult function to design, very a lot topic to the “what if two people did this” drawback. When folks do it advert hoc like this, it’s a minimum of clear what the habits is for customers of this workaround, as a result of the workaround doesn’t have something to do with the non-backport API so far as the language is worried.

Express backwards-deployment assist

Moderately than a third-party “polyfill” strategy, or an specific wrapper operate, wrapper kind, or abstracting protocol, Swift may simply make it simpler for the usual library (and different Swift libraries which might be a part of the OS) to assist backwards-deploying new options by bundling them with the app “as wanted”, and equally loading them “as wanted”. That is difficult as a result of it means any consumer code has to now work two alternative ways (once more, we will’t play the rpath trick for brand new options in an current library). I don’t wish to make this sound easy or simple to implement as a result of it wouldn’t be, however I consider it’s attainable. (For a style of what this is able to take, take a look at an older publish of mine, “Backwards-deployable Conformances”.)

Android really does do that pretty typically, a minimum of with its Java APIs. It’s a bit simpler to arrange as a result of its apps and libraries use an intermediate format reasonably than native code, and in addition as a result of Java doesn’t have extensions and subsequently there are fewer methods to switch current sorts. They name this “desugaring”.

ABI Stability with out Swift-in-the-OS

A last risk is that Apple may have dedicated to Swift’s ABI stability, however not really put Swift within the OS. Any closed-source Swift libraries Apple wished to ship can be made out there for embedding in every app that wished to make use of them, probably augmented with the “deduplication” technique talked about above. However Apple needs to have the ability to replace their libraries as a part of OS releases, in addition to safety updates. It’s this functionality that enables them to do system-wide UI changes and redesigns with out forcing everybody to publish new variations of their app forward of time and with comparatively minimal conditionalizing even after the very fact. You may argue whether or not or not you suppose that’s a great factor, however it’s one thing Apple gained’t ever hand over.

Conclusions

Was Swift-in-the-OS a foul trade-off for app builders? Perhaps, in the event you’re not pressed for code measurement. ^{However then Apple wouldn’t have been in a position to write system libraries in Swift, and that was by no means an choice.}

P.S. Swift on Home windows continues to be a younger challenge, and it doesn’t have the identical OS library considerations that Apple platforms do…however Home windows can be dwelling to many extra closed-source third-party libraries than Linux is. So there’s a risk we’ll have a platform with ABI stability but without Swift-in-the-OS within the not-too-distant future.

EDIT: Little of that is new to Swift

Numerous what I’ve talked about is a consequence of libraries being shipped with the OS. Notably, Goal-C used “ARCLite”, a compatibility static library “linked into the principle app solely” as described above, to rigorously assist backwards-deployment of a number of new language options, however some have been nonetheless restricted to the newest OS variations (thanks, Nick). Extra typically, Apple has by no means supported backwards-deployment of latest options added to Basis, successfully “the stdlib of Goal-C” on Apple platforms. Folks have by no means been pleased about that both, however there was additionally no concerted push to “repair” it.

What modified with Swift? There’s a technical change, the place the steadiness between “compile time” and “run time” is extra on the “compile time” aspect in comparison with Goal-C. However I actually suppose the most important adjustments are social, not technical:

1. The Swift stdlib is a part of an open supply challenge, whereas Basis isn’t.

2. There have been a number of years the place Swift behaved like a third-party dependency reasonably than an OS library, and thus had fewer constraints. (That is what most of this publish focuses on, making an attempt to persuade you that there’s not a lot level in being nostalgic for that.) Swift on non-Apple platforms nonetheless works like this, so the sentiment by no means fairly goes away.

3. As a result of Apple’s OS launch schedules, for the final a number of years there’s been an Xcode that incorporates a New Swift Compiler with an Outdated Mac SDK, and thus an Outdated Swift Stdlib. That’s a compatibility hazard for the Apple Swift team, as a result of their compiler has to assist each SDKs for a bit (perhaps the outdated stdlib had bugs!), however it’s additionally irritating for builders who now have to differentiate between Swift language adjustments and Swift stdlib adjustments. This might have occurred with Goal-C as nicely, however Clang and Basis don’t share model numbers, so it’s much less in-your-face about it.

Is that convincing? That’s as much as you. However supporting updateable OS libraries and backwards-compatibility is Tough, and it ought to come as no shock that forward-looking, “everyone replace to the newest OS” Apple prioritized the previous over the latter.

This entry was posted on
October
09,

2022
and is filed beneath
Technical.

Tags:

Swift,

Apple