Cosmopolitan Third Version

Oct 31st, 2023 @ justine’s web page
![[cosmpolitan honeybadger logo]](https://blinkingrobots.com/wp-content/uploads/2023/11/Cosmopolitan-Third-Edition.png)
Cosmopolitan Honeybadger
After almost one 12 months of growth, I am happy to announce our
version
3.0.1 launch of the Cosmopolitan library. The venture is an
solely new animal. For starters, Mozilla sponsored our work as a part of
their MIECO program.
Google additionally
awarded
me an open source peer bonus for my work on Cosmopolitan, which is a
uncommon honor, and it is good to see our venture listed up there among the many
greats, e.g. curl, linux, and many others. By way of this launch, we’re dwelling up
to the good expectations you have all held for this venture in a quantity
of how. The primary is we invented a brand new linker that permits you to construct fats
binaries which may run on these platforms:
- AMD64
- Linux
- MacOS
- Home windows
- FreeBSD
- OpenBSD
- NetBSD
- ARM64
- Linux
- MacOS
- Home windows (non-native)
It is known as
apelink.c
and it is a tremendous piece of poetry that weaves collectively the Transportable
Executable, ELF, Mach-O, and PKZIP file codecs into shell scripts that
run on most PCs and servers with no need to be put in. That is an
thought whose time has come; POSIX even modified their guidelines about binary in
shell scripts particularly to allow us to do it. So we have been utilizing it to
create a “Fats Linux Distro” which I’ve named the “Cosmos”. Within the
Cosmos, each program is statically linked and incorporates a PKZIP central
listing the place its /usr/share dependencies are embedded. You possibly can assume
of it as a coalition of individualistic executables, the place every program
could be separated from the entire and run on different OSes. To this point it
contains applications like Emacs, Vim, CoreUtils, Curl, Git, and many others.
cosmos-3.0.1.zip
213mb – PE+ELF+MachO+ZIP+SH executables
For AMD64+ARM64 on Linux+Mac+Home windows+FreeBSD+NetBSD+OpenBSDce256ededf106748a09f13bf47ace9ef0e6f115d963353d3d63c21302c5f28f4
Extra particularly, the above zip file incorporates fats binaries for ape
, python
,
lua
, qjs
, vim
, emacs
, emacsclient
, nano
, llama
, bash
, sprint
, much less
, git
, grep
,
curl
, wget
, tidy
, zip
, unzip
, zstd
, bzip2
, sqlite3
, life
, nesemu1
, make
(GNU + SECCOMP + Landlock), gmake
(GNU), redbean
, greenbean
, datasette
, assimilate
, rusage
, ctags
, wall
, pledge
, verynice
,
discover
, tree
, basename
, chgrp
, cp
, date
, du
, expr
, teams
, ls
, mknod
, nl
,
pathchk
, pwd
, rm
, seq
, shred
, stat
, tee
, uname
, customers
, who
, basenc
,
chmod
, csplit
, dd
, echo
, issue
, head
, mktemp
, nohup
, pinky
,
rmdir
, shuf
, stty
, take a look at
, true
, unexpand
, vdir
, whoami
, awk
, chown
, df
,
false
, id
, hyperlink
, mv
, nproc
, pr
, rsync
, sleep
, sum
, truncate
, uniq
, sure
,
b2sum
, chroot
, dir
, fmt
, set up
, md5sum
, numfmt
, printenv
, readlink
,
runcon
, kind
, sync
, timeout
, tsort
, unlink
, base32
, cat
, cksum
, minimize
,
dircolors
, env
, fold
, be a part of
, ln
, mkdir
, od
, printf
, realpath
, script
,
break up
, tac
, contact
, tty
, wc
, base64
, chcon
, comm
, dirname
, increase
, kill
,
logname
, mkfifo
, good
, paste
, ptx
, sed
, sha1sum
, sha224sum
, sha256sum
,
sha384sum
, sha512sum
, tail
, tr
, ttyinfo
, and uptime
.
This solely turned doable in the previous few months, partially due to
Gautham Venkatasubramanian, who spent just a few weekends of his PhD research
modifying
the C language so it is doable to construct standard software program with
Cosmopolitan. Since then it has been out of the frying pan and into the
hearth, testing our library to see if it could actually help a number of the most
complicated and mature tasks within the open supply group. Working the
./configure
scripts and make examine
guidelines of
tasks (e.g. GMP) has carried out a lot to assist us repair bugs and battle take a look at
new fetaures.
One of many issues we’re most proud of, is that Cosmo’s cross platform
help is now adequate to help Cosmo growth. We have
historically solely compiled code on x86 Linux. Devs utilizing Cosmo would
construct their applications on Linux, after which copy the binaries to different OSes.
Specializing in Linux-only helped us achieve appreciable velocity initially
of the venture; the Cosmopolitan monorepo has two million strains of code.
In the present day was the primary day the entire thing compiled on Apple Silicon and
Microsoft Home windows programs, and utilizing Cosmo-built instruments.
Home windows Enhancements
With the intention to get applications like GNU Make and Emacs to work on Home windows, we
applied new
libraries for POSIX signals emulation. Cosmopolitan is now capable of
preempt i/o and ship asynchronous alerts on Home windows, utilizing
a SetThreadContext()
trick I discovered from the Go builders. Cosmo does a significantly
higher
job spawning
processes now too. For instance, we wrote
a brand
new posix_spawn()
function that
goes 10x faster than
the posix_spawn()
included with Cygwin.
Cosmo’s execve()
can now reparent
subprocesses, inherit
non-stdio file descriptor, and
our read()
function now contains a termios driver which, for the primary time,
lets us ballot()
customary enter on consoles. Cosmo binaries
cleanly combine with WIN32, relying just about solely on KERNEL32,
and your fats binaries will not must stay on a separate partition like
WSL.
MacOS Enhancements
Whereas MacOS might not be the prodigal baby of our help vector, this
launch brings enhancements to MacOS customers which might be equally essential.
For starters, we now have first-class native ARM64
help. APE
Loader additionally now dynamically hyperlinks the formally blessed Apple
libraries (e.g. libSystem.dylib) on ARM64, so there’s much less likelihood that
Apple will break your binaries. We have additionally
made semaphores
and futexes
a lot better on XNU, due to Grand Central Dispatch, and ulock on
AMD64.
Portability and Efficiency (Choose Two)
The tip result’s that in the event you swap your Linux construct course of to
use cosmocc
as an alternative of cc
then the applications
you construct, e.g. Bash and Emacs, will simply work on the command prompts of
completely completely different platforms like Home windows and MacOS, and whenever you run
your applications there, it will really feel such as you’re on Linux. Nevertheless
portability is not the one promoting level. Cosmo Libc will make your
software program quicker and use much less reminiscence too. For instance, after I construct Emacs
utilizing the cosmocc toolchain, Emacs thinks it is constructing for Linux. Then,
after I run it on Home windows:
It really goes 2x quicker than the native WIN32 port that the Emacs
authors wrote on their very own. Cosmo Emacs masses my dotfiles in 1.2 seconds
whereas GNU Emacs on Home windows masses them in 2.3 seconds. A few years in the past
after I began this venture, I had this unproven perception that portability
toil may very well be abstracted by having a greater C library. Now I feel this
is all of the proof we want that it isn’t solely doable to make software program
immediately transportable, however really higher too. For instance, one
of the issues you might be questioning is, “these fats binary recordsdata are enormous,
would not that waste reminiscence?” The reply is not any, as a result of Cosmo solely pages
into reminiscence the components of the executable you want. Take for instance one
of Linux’s best hits: the Debian Almquist shell.
$ ls -hal /usr/bin/sprint -rwxr-xr-x 1 root root 107K Nov 21 2022 /usr/bin/sprint $ ls -hal /choose/cosmos/bin/sprint -rwxr-xr-x 1 jart jart 983K Oct 15 19:14 /choose/cosmos/bin/sprint
Right here we see Cosmo’s six OS + two structure fats binary sprint is 30%
greater than the one which comes with Alpine Linux (which solely helps
x86-Linux and dynamically hyperlinks a separate 600kb Musl library). But when I
run them:
$ rusage /usr/bin/sprint -c true took 231µs wall time ballooned to 688kb in dimension wanted 183us cpu (0% kernel) triggered 34 web page faults (100% memcpy) $ rusage /choose/cosmos/bin/sprint -c true took 217µs wall time ballooned to 544kb in dimension wanted 172us cpu (0% kernel) triggered 36 web page faults (100% memcpy)
Right here we see Cosmo’s fats binary model of sprint went quicker and used a
much less reminiscence than an x86-Linux-only binary constructed for Musl Libc. That is
because of (1) the magic of contemporary reminiscence administration, the place CPU MMUs lazily
load 4096 byte blocks at a time; and (2) how
fastidiously apelink
plans your executable structure. For
instance, all that code which is required to help Home windows (it takes a
lot of code to help Home windows) will get linked into its personal particular part
of the binary, distant from what the MMU will wish to web page on UNIX
programs. The identical goes for the embedded ARM64 construct. Since I am working
on AMD64 right here, the ARM64 code isn’t loaded off disk.
You now not want to decide on between the amalgamation launch or the
cosmo monorepo. We now have a 3rd most well-liked possibility, which is our
new cosmocc
command. It really works just about the identical as
the cc
command you already know. The Cosmopolitan README
file
has getting
started directions, which mainly boil right down to this:
sudo mkdir -p /choose sudo chmod 1777 /choose git clone https://github.com/jart/cosmopolitan /choose/cosmo export PATH="/choose/cosmo/bin:/choose/cosmos/bin:$PATH" echo 'PATH="/choose/cosmo/bin:/choose/cosmos/bin:$PATH"' >>~/.profile ape-install # optionally set up a quicker systemwide ape loader cosmocc --update # pull cosmo and rebuild toolchain
In case your growth setting is not x86_64 Linux, then you definately’ll have to
obtain the most recent toolchain, reasonably than utilizing the one which’s vendored
contained in the cosmo repository.
cd /choose/cosmo rm -rf o/third_party/gcc wget https://github.com/jart/cosmopolitan/releases/obtain/3.0/cosmocc-0.0.14.zip unzip cosmocc-0.0.14.zip
Should you’re on Home windows, then you definately want a shell earlier than you are able to do any of the
above. Should you obtain the cosmos.zip file on the prime of the web page, it will
have a bin/ and libexec/ folder. Simply extract these to
your C:
drive. Then
set up Terminal
Preview from the Microsoft Retailer, and configure it in order that it
launches C:binbash -l
as your shell. You will naturally
want a great unzip program so as to do that: one which helps
symbolic hyperlinks (a.ok.a. reparse factors). The cosmos zip contains InfoZIP,
however you’ll be able to obtain it immediately
right here: unzip.exe.
Now that your cosmo compiler is put in and added to
your $PATH
, you’ll be able to construct customary C and C++ applications as
follows:
cosmocc -o foo foo.c cosmoc++ -o bar bar.cc
If you wish to construct applications as fats binaries, there is a particular
compiler now which may make the method simpler than utilizing the
x86_64-unknown-cosmo-cc
, aarch64-unknown-cosmo-cc
, fixupobj
,
and apelink
instructions immediately. This compiler works by
manipulating your compiler flags so {that a}
concomitant .aarch64/foo.o
will get created for
each foo.o
you compile. It runs the x86_64 and aarch64
compilers in parallel too, so it would not go a lot slower
than cosmocc
on the finish of the day.
fatcosmocc -o foo foo.c fatcosmoc++ -o bar bar.cc
Integrating with GNU Autotools tasks is straightforward as nicely.
export CC=cosmocc export CXX=cosmoc++ ./configure --prefix=/choose/cosmos make -j make set up
Confer with
the README
file for extra detailed directions. Additionally remember to try
the ahgamut/superconfigure
repository for the most important supply of high-quality examples on easy methods to
construct open supply software program utilizing Cosmo.
This cosmos launch contains the most recent model of
the redbean web server. This can be a fats
single-file forking Lua+SQLite+MbedTLS stack written by Cosmopolitan
authors to showcase the capabilities of the library for greenfield
growth. Cosmopolitan is sweet for extra than simply constructing previous GNU
code and redbean proves that. It is
the third most upvoted passion
venture in Hacker Information historical past. A couple of weeks in the past Berwyn
Hoyt independently
determined it to be the quickest Lua internet server too!
One of many the reason why redbean is a forking internet server is as a result of we
did not develop our personal
POSIX
Threads implementation till final 12 months. So we put a number of thought
into writing an instance of how one can construct a naked minimal threaded internet
server that is even quicker than redbean, and it is known as
greenbean. There is a prebuilt fats binary for it within the cosmos.zip
distribution above. Greenbean is
400
lines of liberally commented perfection. I nonetheless cannot imagine how
good we bought its reminiscence utilization, due to tips
like MAP_GROWSDOWN
. On Linux, if I ask greenbean to spawn
over 9,000 persistent employee threads:
$ sudo prlimit --pid=$$ --nofile=18000 $ sudo prlimit --pid=$$ --nproc=18000 $ rusage greenbean 9001 listening on http://127.0.0.1:8080 listening on http://10.10.10.237:8080 greenbean employees=0 connections=0 messages=0 ^C shutting down... took 7,959,324µs wall time ballooned to 40,352kb in dimension wanted 929,917us cpu (92% kernel) triggered 10,039 web page faults (100% memcpy) 54,689 context switches (99% consensual)
Then it one way or the other solely makes use of 40mb of peak resident reminiscence, and in keeping with
htop, greenbean’s digital reminiscence utilization is 76,652kb. That is for 9,001
threads. Like redbean, greenbean is ready to deal with a whole lot of hundreds
of requests per second on my Intel Core i9-9900, besides (1) greenbean
has higher shared reminiscence help, (2) it units up and tears down
connections quicker, and (3) it allows you to expertise the enjoyment of utilizing Mike
Burrows’ *NSYNC library,
which is the premise of Cosmopolitan’s POSIX synchronization primitives.
Should you’re not accustomed to the person, he is the man who coded Chubby and
Altavista, a world search engine which was so environment friendly it solely wanted
to function on a single server. However you would not assume *NSYNC is as
prolific as it’s in the event you’re solely going off star depend.
My favourite factor about greenbean is how elegantly it reacts to CTRL-C.
Whenever you interrupt greenbean, it will use the POSIX pthread_cancel() API
to instantly terminate all of the employee threads, in order that shutdown
occurs at once. Cancelation is among the tricker ideas for a
C library to get proper. It is up there with threads, alerts, and fork in
phrases of how its ramifications pervade every thing. So much like Musl
Libc, we have put a number of thought into making certain that Cosmo does it
appropriately. Cosmo avoids cancelation race circumstances the identical method as Musl
and Cosmo additionally implements Musl’s PTHREAD_CANCEL_MASKED
extension to the POSIX customary, which has significantly bought to be one in every of
Wealthy Felker’s most good concepts. Learn the commentary in greenbean.c
if you wish to study extra.
A few of you would possibly view greenbean as only a toy instance. In that case, if
you wish to see one thing based mostly on greenbean that’s really working in
manufacturing, then pay a go to
to https://ipv4.games/ whose supply
code is
in net/turfwar/turfwar.c
and net/turfwar/.init.lua.
This can be a hybrid redbean + greenbean service, the place redbean does the
HTTPS frontend, and greenbean does the HTTP backend. Hackers like to
unleash their botnets on the IPv4 Video games, and it truthfully is not that onerous
to resist a DDOS with 49,131,669 IPs when your internet server can do a
million requests per second. That is for a service the place 99% of the
requests are write requests. If you wish to have enjoyable with this
server, then you definately’re additionally welcome to take a look at
our monitoring metrics whereas
you do it.
This cosmos.zip launch contains a number of video games you’ll be able to play which have
been vetted on all our help platforms. For instance you’ll be able to really
play Nintendo within the terminal and it will even work contained in the Home windows 10
command immediate. Getting ballot(stdin) to work in Home windows is a messier
downside than even naming or cache invalidation, so it is tremendous candy that
Cosmopolitan Third Version is now capable of summary that complexity.
Though within the case of our port of Bisqwit’s fabulous NESEMU1 program,
it turned out we did not want ballot() in any respect! Simply calling learn() and
letting it’s EINTR
‘d by setitimer()
sixty
instances a second to pipe audio did the trick. So enjoying these video games
helped battle take a look at our new alerts implementation too.
We could not have GUI help but, however you should use your mouse within the
terminal. On Home windows, your mouse cursor will generate the identical ANSI
XTERM fashion management codes because it does on Linux, MacOS, BSD, and many others. Strive
working the life
program that is included in Cosmos. Left
click on attracts cells. Area runs an iteration of the life recreation. Proper click on
can drag the show. It’s also possible to ctrl+wheel to zoom the show in and
out. To learn the supply code to this straightforward program, examine
out tool/viz/life.c.
Boulder startup dylibso simply introduced just a few weeks in the past that they’ve
adopted Cosmopolitan for his or her new
product Hermit:
Actually Portable Wasm. Hermit allows you to create safe cross-platform
executables for WebAssembly modules. It is price a attempt! Cosmopolitan has
a protracted historical past of serving the wants of the Wasm group. Our first
main adopter again in early 2021 was really
the wasm3 project which
offers a equally nice resolution to working Wasm exterior the browser.
I am completely happy to see these tasks benefiting from the benefits
Cosmopolitan has to supply. Will you be the subsequent adopter? In that case, really feel
free to achieve out to me personally and I am going to see what I can do to assist
you be
profitable: jtunney@gmail.com.
You are additionally invited to join our
Discord group.
Funding for Cosmopolitan Third Version was crowdsourced from Justine
Tunney’s GitHub sponsors
and Patreon subscribers, the
backing of
Mozilla’s MIECO program,
and the beneficiant contributions of our
developer community on
Discord. Your help is what makes tasks like Cosmopolitan doable.
Thanks!