Now Reading
Go, Containers, and the Linux Scheduler

Go, Containers, and the Linux Scheduler

2023-11-07 13:10:04

Final up to date on

Like many Go builders my purposes are normally deployed in containers.
When working in container orchestrators it’s essential to set CPU limits to make sure that the container doesn’t eat all of the CPU on the host.
Nevertheless, the Go runtime just isn’t conscious of the CPU limits set on the container and can fortunately use all of the CPU accessible.
This has bitten me prior to now, resulting in excessive latency, on this weblog I’ll clarify what’s going on and repair it.

How the Go Rubbish Collector works

That is going to be a fairly excessive degree overview of the Go Rubbish Collector (GC).
For a extra in depth overview I like to recommend studying the go docs
and this excellent series of blogs
by Will Kennedy.

The overwhelming majority of the time the Go runtime performs rubbish assortment concurrently with the execution of your program.
Which means the GC is working similtaneously your program. Nevertheless, there are two factors within the GC course of the place the Go runtime must cease each Goroutine.
That is required to make sure information integrity. Earlier than the Mark Part of the GC the runtime stops each Goroutine to use the write barrier, this ensures no objects created after this level are rubbish collected. This section is called Sweep Termination.
After the mark section has completed there may be one other cease the world section, this is called Mark Termination and the identical course of occurs to take away the write barrier. These normally takes within the order of tens of microseconds.

I created a easy internet utility that allocates a number of reminiscence and ran it in a container with a restrict of 4 CPU cores with the next command.The Supply code for that is accessible here.

docker run --cpus=4 -p 8080:8080 $(ko construct -L principal.go)

It’s price noting the docker CPU restrict is a gentle restrict, that means it’s solely enforced when the host is CPU constrained. Which means the container can use greater than 4 CPU cores if the host has spare capability.

You’ll be able to gather a hint utilizing the runtime/trace bundle then analyze it with go software hint. The next hint reveals a GC cycle captured on my machine. You’ll be able to see the Sweep Termination and the Mark Termination cease the world section on Proc 5 (They’re labelled STW for cease the world).

GC Trace

This GC cycle took slightly below 2.5ms, however we spent virtually 10% of that in a cease the world section. This can be a fairly important period of time, particularly in case you are working a latency delicate utility.

The Linux Scheduler

The Completely Fair Scheduler (CFS) was launched in Linux 2.6.23 and was the default Scheduler till Linux 6.6 which was launched final week. It’s doubtless you’re utilizing the CFS.

The CFS is a proportional share scheduler, because of this the burden of a course of is proportional to the variety of CPU cores it’s allowed to make use of. For instance, if a course of is allowed to make use of 4 CPU cores it can have a weight of 4. If a course of is allowed to make use of 2 CPU cores it can have a weight of two.

The CFS does this by allocating a fraction of CPU time. A 4 core system has 4 seconds of CPU time to allocate each second. If you allocate a container numerous CPU cores you’re basically asking the Linux Scheduler to offer it n CPUs price of time.

Within the above docker run command I’m asking for 4 CPUs price of time. Which means the container will get 4 seconds of CPU time each second.

The Downside

When the Go runtime begins it creates an OS thread for every CPU core. This implies you probably have a 16 core machine the Go runtime will create 16 OS threads – no matter any CGroup CPU Limits. The Go runtime then makes use of these OS threads to schedule goroutines.

The issue is that the Go runtime just isn’t conscious of the CGroup CPU limits and can fortunately schedule goroutines on all 16 OS threads. Which means the Go runtime will count on to have the ability to use 16 seconds of CPU time each second.

Lengthy cease the world durations come up from the Go runtime needing to cease Goroutine on threads that it’s ready for the Linux Scheduler to schedule. These threads is not going to be scheduled as soon as the container has used it’s CPU quota.

See Also

The Answer

Go lets you restrict the variety of CPU threads that the runtime will create utilizing the GOMAXPROCS atmosphere variable.
This time I used the next command to start out the container

docker run --cpus=4 -e GOMAXPROCS=4 -p 8080:8080 $(ko construct -L principal.go)

Under is a hint captured from the identical utility as above, now with the GOMAXPROCS atmosphere variable matching the CPU quota.

GC Trace

On this hint, the rubbish assortment is far shorter, regardless of having the very same load. The GC Cycle took underneath 1ms and the cease the world section was 26μs, roughly 1/10 of the time when there was no restrict.

GOMAXPROCS ought to be set to the variety of CPU cores that the container is allowed to make use of, if you happen to’re allocating fractional CPU spherical down, until you’re allocating lower than 1 CPU core wherein case spherical up. GOMAXPROCS=max(1, ground(CPUs)) can be utilized to calculate the worth.
When you discover it simpler Uber has open sourced a library automaxprocs to calculate this worth for you out of your container’s cgroups robotically.

There’s an impressive Github Issue with the Go runtime to help this out the field so hopefully it is going to be added ultimately!


When working Go in a containerised utility it’s essential to set CPU limits. It’s additionally essential to make sure that the Go runtime is conscious of those limits by setting a wise GOMAXPROCS worth or utilizing a library like automaxprocs.

Source Link

What's Your Reaction?
In Love
Not Sure
View Comments (0)

Leave a Reply

Your email address will not be published.

2022 Blinking Robots.
WordPress by Doejo

Scroll To Top