Now Reading
When allocators are hoarding your treasured reminiscence

When allocators are hoarding your treasured reminiscence

2023-10-06 10:15:30

Whereas switching to the newest fashionable framework or language has turn out to be considerably of a cliché within the engineering world, there are occasions when upgrading is warranted and obligatory. For our search engineering crew, devoted to sustaining the core engine options, this contains upgrading the working system model to get the newest kernel or library options with the intention to shortly to accommodate our newest code releases. We’re all the time eagerly ready for an improve of the OS, and have been pushing to do it for fairly a while.

Upgrading an working system in manufacturing generally is a huge deal while you’re dealing with a whole lot of servers throughout 70 information facilities world wide. For our manufacturing servers, we’re at the moment utilizing Ubuntu 16.04 (which was launched greater than 4 years in the past), and our basis crew (managing all manufacturing servers, and guaranteeing we all the time have a operating service exceeding our SLA) has been working prior to now few months on getting ready the improve to a newer 20.04 long-term model.

However upgrading is hard

It’s good to be ready for disagreeable surprises as you’re altering the model of a number of elements: kernel and its related drivers, working system’s C library, numerous libraries, but in addition probably a number of methods, insurance policies, default configuration settings that include the working system.

One disagreeable shock we had whereas upgrading was the general reminiscence consumption. A graph (courtesy of Wavefront metrics) being higher than anything, I’ll allow you to guess when the “inexperienced” server was upgraded to a better working system model beneath:

If you happen to managed to identify the slight improve of reminiscence consumption, sure, we moved from a consumption ranging between 10 and 30 GB of reminiscence (with a median of 12) to a consumption between 30 and 120 GB of reminiscence (with a median above 100). To be truthful, consumption could be even greater, however we solely have 128GB on this particular machine (which is greater than sufficient, and is usually used as an enormous web page cache), and the drops in reminiscence consumption are the occasions when the method consuming our treasured reminiscence is both forcibly being reloaded by us, or is forcibly being slaughtered by the indignant kernel (which is type of a hassle, because it triggers a number of alarms right here and there).

All in all, now we have an issue. As a startup firm, we’re anticipated to satisfy the stereotypes of the 10x engineer, however that is most likely not the 10x we’re speaking about.

Investigating extreme reminiscence hoarding

The irony was not misplaced on us: we pushed for upgrading the system, and now we’re liable for the server taking over a lot reminiscence. Effectively, there was just one factor to do:  we wanted to grasp why the improve had such a dramatic affect.

The leak

The primary concept that got here to thoughts was to suspect some type of a reminiscence leak. However there’s a catch: solely current Linux variations have it. Can or not it’s a reminiscence leak triggered by a situation that we didn’t meet on the earlier system?

To validate this speculation, we normally leverage the extremely highly effective Linux perf profiler from Brendan Gregg. Each developer serious about efficiency ought to know this instrument, and we extremely suggest watching some of the presentations made by the grasp himself.

A typical means to do that is to connect the operating daemon, and search for unreleased reminiscence (after 10 minutes) every so often:

sudo memleak-bpfcc -a --older 600000 --top 10 -p 2095371 120

Sadly, we didn’t see any leaks — even when within the meantime the method ate 100 gigabyte of our treasured RAM.

So this wasn’t a leak, however we had been shedding reminiscence. The following logical suspect was the underlying reminiscence allocator.

The grasping allocator

Okay, you is perhaps confused, as now we have a number of allocators right here. As a developer, you most likely heard about malloc, sometimes positioned within the C library (the glibc). Our course of is utilizing the default glibc allocator, which may be seen as some type of a retailer for any dimension of reminiscence allocations. However the glibc itself can’t allocate reminiscence, solely the kernel can. The kernel is the wholesaler, and it solely sells giant portions. So the allocator will sometimes get giant chunks of reminiscence from the kernel, and cut up them on demand. When releasing reminiscence, it’s going to consolidate free areas, and can sometimes launch giant chunks by calling the kernel.

However allocators can change their technique. You’ll have a number of retailers, to accommodate a number of threads operating inside a course of. And every retailer can resolve to retain a number of the giant reminiscence chunks launched for later reuse. Retailers can turn out to be grasping, and will refuse to launch their inventory.

To validate this new speculation, we determined to play straight with the glibc allocator, by calling its very particular “rubbish collector”:

MALLOC_TRIM(3)             Linux Programmer's Guide            MALLOC_TRIM(3)

       malloc_trim - launch free reminiscence from the highest of the heap

       #embody <malloc.h>
       int malloc_trim(size_t pad);

       The  malloc_trim()  perform makes an attempt to launch free reminiscence on the prime
       of the heap (by calling sbrk(2) with an acceptable argument).
       The pad argument specifies the quantity of free house to go away  untrimmed
       on the prime of the heap.  If this argument is 0, solely the minimal quantity
       of reminiscence is maintained on the prime of  the  heap  (i.e.,  one  web page  or
       much less).   A nonzero argument can be utilized to take care of some trailing house
       on the prime of the heap with the intention to permit future allocations to be  made
       with out having to increase the heap with sbrk(2).

       The  malloc_trim()  perform  returns 1 if reminiscence was truly launched

A easy but hacky answer is to connect to the method with a debugger (gdb -p pid), and manually name malloc_trim(0). The end result speaks for itself:

The  orange is the upgraded server reminiscence consumption, the 2 different curves are the earlier working system’s variations. The sudden drop at round 09:06 is the decision to the malloc_trim perform.

To corner-case the issue, we additionally used one other slightly helpful glibc-specific perform, dumping some states of the allocator:

MALLOC_INFO(3)             Linux Programmer's Guide            MALLOC_INFO(3)

       malloc_info - export malloc state to a stream


       int malloc_info(int choices, FILE *stream);

       The  malloc_info()  perform  exports  an XML string that describes the
       present state of the memory-allocation implementation  in  the  caller.
       The  string  is printed on the file stream stream.  The exported string
       contains details about all arenas (see malloc(3)).
       As at the moment applied, choices should be zero.

       On success, malloc_info() returns 0; on  error,  it  returns  -1,  with
       errno set to point the trigger.

Sure, right here once more, connect and straight use gdb:

(gdb) p fopen("/tmp/debug.xml", "wb")
$1 = (_IO_FILE *) 0x55ad8b5544c0
(gdb) p malloc_info(0, $1)
$2 = 0
(gdb) p fclose($1)
$3 = 0

The XML dump did present attention-grabbing data. There are virtually 100 heaps (the “resellers”), and a few of them present bothersome statistics:

<heap nr="87">
    ... ( skipped not so attention-grabbing half )
    <dimension from="542081" to="67108801" complete="15462549676" rely="444"/>
    <unsorted from="113" to="113" complete="113" rely="1"/>
  <complete kind="quick" rely="0" dimension="0"/>
  <complete kind="relaxation" rely="901" dimension="15518065028"/>
  <system kind="present" dimension="15828295680"/>
  <system kind="max" dimension="16474275840"/>
  <aspace kind="complete" dimension="15828295680"/>
  <aspace kind="mprotect" dimension="15828295680"/>
  <aspace kind="subheaps" dimension="241"/>

The “relaxation” part seems to be free blocks, after a look at glibc’s sources:

fprintf (fp,
"<complete kind="quick" rely="%zu" dimension="%zu"/>n"
"<complete kind="relaxation" rely="%zu" dimension="%zu"/>n"
"<complete kind="mmap" rely="%d" dimension="%zu"/>n"
"<system kind="present" dimension="%zu"/>n"
"<system kind="max" dimension="%zu"/>n"
"<aspace kind="complete" dimension="%zu"/>n"
"<aspace kind="mprotect" dimension="%zu"/>n"
total_nfastblocks, total_fastavail, total_nblocks, total_avail,
mp_.n_mmaps, mp_.mmapped_mem,
total_system, total_max_system,
total_aspace, total_aspace_mprotect);

and is accounting 901 blocks, for greater than 15GB of reminiscence.  Total statistics are in line with what we noticed:

<complete kind="quick" rely="551" dimension="35024"/>
<complete kind="relaxation" rely="511290" dimension="137157559274"/>
<complete kind="mmap" rely="12" dimension="963153920"/>
<system kind="present" dimension="139098812416"/>
<system kind="max" dimension="197709660160"/>
<aspace kind="complete" dimension="139098812416"/>
<aspace kind="mprotect" dimension="140098441216"/>

Sure, that’s 137GB of free reminiscence not reclaimed by the system. Speak about being grasping!

Tuning the internals with glibc

At this stage, we reached out to the glibc mailing-list to lift the difficulty, and we’ll fortunately present any data if that is confirmed to be a difficulty with the glibc allocator (on the time of scripting this publish, now we have not discovered something new).

Within the meantime, we tried tuning the internals utilizing the GLIBC_TUNABLES options (glibc.malloc.trim_threshold and glibc.malloc.mmap_threshold sometimes), unsuccessfully. We additionally tried to disable the latest options, such because the thread cache (glibc.malloc.tcache_count=0), however clearly the per-thread allocation cache was for small (a whole lot of bytes at most) blocks.

From right here, we envisioned a number of choices to maneuver ahead.

The (non permanent) repair – The rubbish collector

Calling malloc_trim commonly is a slightly soiled non permanent hack, however seems to be fairly working, with runs generally taking a number of seconds to five minutes every:

  "periodMs": 300000,
  "elapsedMs": 1973,
  "message": "Purged reminiscence efficiently",
  "rss.earlier than": 57545375744,
  "rss.after": 20190265344,
  "rss.diff": -37355110400

Notably, GC time appears to be linear with collected free house. Dividing the interval by 10 additionally divides each reclaimed reminiscence and frolicked in GC by the identical issue:

  "periodMs": 30000,
  "elapsedMs": 193,
  "message": "Purged reminiscence efficiently",
  "rss.earlier than": 19379798016,
  "rss.after": 15618609152,
  "rss.diff": -3761188864,

The affect of calling the GC is slightly apparent on this graph:

In inexperienced, the brand new glibc, in blue, the previous model. The orange curve is the forcer glibc with an everyday GC.

The server embedding the brand new glibc slowly drifts, taking an growing quantity of house (virtually 60GB right here).

See Also

At 4:00pm, the GC is began within the new (inexperienced) glibc code, and also you see the reminiscence consumption stats low.

And the inexperienced curve beneath demonstrates the affect on altering the GC interval from 5 minutes to 30 seconds:

Lastly, we additionally efficiently examined an override of free that triggers trim operations when liberating a certain quantity of reminiscence:


#if (!__has_feature(address_sanitizer))

static std::atomic freeSize = 0;
static std::size_t freeSizeThreshold = 1_Gi;

extern "C"
    // Glibc "free" perform
    extern void __libc_free(void* ptr);

    void free(void* ptr)
        // If function is enabled
        if (freeSizeThreshold != 0) {
            // Free block dimension
            const size_t dimension = malloc_usable_size(ptr);

            // Increment freeSize and get the end result
            const size_t totalSize = freeSize += dimension;

            // Set off compact
            if (totalSize >= freeSizeThreshold) {
                // Reset now earlier than trim.
                freeSize = 0;
                // Trim

        // Free pointer

All these exams had been executed on a number of clusters to substantiate totally different workloads.

What’s the price of the GC ? We didn’t see any detrimental affect by way of CPU utilization (and notably system CPU utilization), and, malloc_trim implementation, it seems every area is locked individually and cleaned one after the other, slightly than having a “huge lock” mannequin:

__malloc_trim (size_t s)
  int end result = 0;

  if (__malloc_initialized < 0) ptmalloc_init (); mstate ar_ptr = &main_arena; do = mtrim (ar_ptr, s);
      __libc_lock_unlock (ar_ptr->mutex);

      ar_ptr = ar_ptr->subsequent;
  whereas (ar_ptr != &main_arena);

  return end result;

Devoted allocator

Utilizing a distinct allocator (which may embody jemalloc or tcmalloc) is one other attention-grabbing risk. However transferring to a very totally different allocator code has some drawbacks. First, it requires a protracted validation, in semi-production after which manufacturing. Variations may be associated with very particular allocation patterns we might have sometimes (and we actually have unusual ones generally, I can assure that). And since we’re utilizing a much less frequent C++ library (libc++ from llvm), mixing each this much less frequent case with an ever much less frequent allocator may result in completely new patterns in manufacturing. And by new, it means attainable bugs no person else noticed earlier than.

Investigating the bug

The underlying bug is something however apparent.

Enviornment leak

A bug crammed in 2013, malloc/free can’t give the memory back to kernel when main_arena is discontinuous, appears a bit like the difficulty we’re experiencing, however that is nothing new in our methods, and regardless of having malloc_trim reclaiming reminiscence in glibc 2.23, the order of magnitude of our present drawback is completely unprecedented. This bug is nonetheless nonetheless pending, probably solely impacting corner-cases.

Variety of arenas

Improve of arenas is perhaps one other risk. We grew from 57 arenas in 2.23 to 96 arenas in 2.31, on the identical {hardware} and surroundings. Whereas it is a notable improve, right here once more, the order of magnitude is simply too huge to be the set off. Alex Reece recommended on his weblog publish about Arena “leak” in glibc to scale back the variety of arenas all the way down to the variety of cores by way of the glibc.malloc.arena_max tunable. This completely is sensible when your course of doesn’t have extra threads than cores (which is true in our case), and will in idea mitigate the wasted reminiscence difficulty. In follow, sadly, it didn’t: decreasing the variety of arenas from 96 all the way down to 12 nonetheless leaves  the identical points:

Even decreasing to 1 (that’s, the primary sbrk() area), truly:

Enviornment threshold

Curiously, every rising area stops rising after some time, capping as much as the utmost allotted reminiscence within the course of historical past. The free-but-not-released blocks may be big (as much as a gigabyte), which is slightly shocking as these blocks are purported to be served with mmap.

To be continued

We’ll proceed taking part in with totally different situations, and ideally have a trivially reproducible case at some point quickly that might assist fixing the foundation trigger. Within the meantime, now we have a workaround (a type of a GC thread) which is way from being excellent, however will permit us to maneuver ahead.

The takeaway

Upgrading working methods and libraries has potential affect which are typically neglected. Limiting dangers can embody upgrading totally different elements (akin to upgrading the kernel, or linked libraries to the latest ones), one after the other over a time period, every for lengthy sufficient to detect regressions (reminiscence or CPU utilization, modifications of habits, instability…). Upgrading extra commonly can also be one other level that might have helped right here (leaping two system releases without delay might be not the most secure alternative). Lastly, executing rolling upgrades, and collected units of metrics very intently must be part of wholesome manufacturing deployment procedures. We now have actually discovered and can be implementing these classes in our processes going ahead.

Source Link

What's Your Reaction?
In Love
Not Sure
View Comments (0)

Leave a Reply

Your email address will not be published.

2022 Blinking Robots.
WordPress by Doejo

Scroll To Top