A Journey to 1 TiB/s

I am unable to consider they figured it out first. That was the thought going via my head again in mid-December after a number of weeks of 12-hour days debugging why this cluster was sluggish. This was most likely probably the most intense efficiency evaluation I might accomplished since Inktank. Half-forgotten superstitions from the 90s about appeasing SCSI gods flitted via my consciousness. The 90s? Man, I am getting outdated. We have been about two-thirds of the way in which via the work that might allow us to begin over initially. Talking of which, I am going to begin over initially.

Again in 2023 (I nearly mentioned earlier this yr till I remembered we’re in 2024), Clyso was approached by a reasonably hip and leading edge firm that wished to transition their HDD backed Ceph cluster to a ten petabyte NVMe deployment. They have been instantly attention-grabbing. They’d no particular want for RBD, RGW, or CephFS. They’d put collectively their very own {hardware} design, however to my delight approached us for suggestions earlier than truly buying something. They’d barely uncommon necessities. The cluster needed to be unfold throughout 17 racks with 4U of house obtainable in every. Energy, cooling, density, and vendor desire have been all elements. The brand new nodes wanted to be migrated into the present cluster with no service interruption. The community nevertheless was already constructed, and it is a beast. It is one of many quickest Ethernet setups I’ve ever seen. I knew from the start that I wished to assist them construct this cluster. I additionally knew we would must do a pre-production burn-in and that it could be the proper alternative to showcase what Ceph can do on a system like this. What follows is the story of how we constructed and examined that cluster and the way far we have been in a position to push it.

I might first prefer to thank our superb buyer who made all of this doable. You have been a pleasure to work with! Thanks as properly for permitting us right here at Clyso to share this expertise with the Ceph group. It’s via this sharing of data that we make the world a greater place. Thanks to IBM/Red Hat and Samsung for offering the Ceph group with the {hardware} used for comparability testing. It was invaluable to have the ability to consider the numbers we have been getting towards earlier checks from the lab. Thanks to the entire Ceph contributors who’ve labored tirelessly to make Ceph nice! Lastly, thanks particularly to Anthony D’Atri and Lee-Ann Pullar for his or her superb copyediting abilities!

When the shopper first approached Clyso, they proposed a configuration using 34 dual-socket 2U nodes unfold throughout 17 racks. We supplied a few various configurations from a number of distributors with a give attention to smaller nodes. Finally they determined to go together with a Dell structure we designed, which quoted at roughly 13% cheaper than the unique configuration regardless of having a number of key benefits. The brand new configuration has much less reminiscence per OSD (nonetheless comfortably 12GiB every), however quicker reminiscence throughput. It additionally offers extra combination CPU assets, considerably extra combination community throughput, a less complicated single-socket configuration, and makes use of the latest era of AMD processors and DDR5 RAM. By using smaller nodes, we halved the impression of a node failure on cluster restoration.

The client indicated they want to restrict the added per-rack energy consumption to round 1000-1500 watts. With 4 of those nodes per rack, the combination TDP is estimated to be at the least 1120 Watts plus base energy utilization, CPU overage peaks, and energy provide inefficiency. IE it is probably we’re pushing it a bit below load, however we do not anticipate important deviation past the appropriate vary. If worse got here to worst, we estimated we might shave off roughly 100 watts per rack by reducing the processor cTDP.

Specs for the system are proven under:

An extra advantage of using 1U Dell servers is that they’re primarily a more moderen refresh of the programs David Galloway and I designed for the upstream Ceph efficiency lab. These programs have been examined in a variety of articles over the previous couple of years. It seems that there was a serious performance-impacting situation that got here out throughout testing that didn’t have an effect on the earlier era of {hardware} within the upstream lab however did have an effect on this new {hardware}. We’ll speak about that extra later.

With out moving into too many particulars, I’ll reiterate that the shopper’s community configuration may be very well-designed and fairly quick. It simply has sufficient combination throughput throughout all 17 racks to let a cluster of this scale actually stretch its legs.

To do the burn-in testing, ephemeral Ceph clusters have been deployed and FIO checks have been launched utilizing CBT. CBT was configured to deploy Ceph with a number of modified settings. OSDs have been assigned an 8GB osd_memory_target. In manufacturing, a better osd_memory_target needs to be acceptable. The client had no want to check block or S3 workloads, so one would possibly assume that RADOS bench could be the pure benchmark alternative. In my expertise, testing at a big scale with RADOS bench is difficult. It is powerful to find out what number of situations are wanted to saturate the cluster at given thread counts. I’ve run into points prior to now the place a number of concurrent swimming pools have been wanted to scale efficiency. I additionally did not have any preexisting RADOS bench checks useful to check towards. As an alternative, we opted to do burn-in testing utilizing the identical librbd backed FIO testing we have used within the upstream lab. This allowed us to partition the cluster into smaller chunks and examine outcomes with beforehand revealed outcomes. FIO can be very well-known and well-trusted.

A significant advantage of the librbd engine in FIO (versus using FIO with kernel RBD) is that there aren’t any points with stale mount factors probably requiring system reboots. We didn’t have IPMI entry to this cluster and we have been below a decent deadline to finish checks. For that purpose, we in the end skipped kernel RBD checks. Primarily based on earlier testing, nevertheless, we anticipate the combination efficiency to be roughly related given ample purchasers. We have been ready, nevertheless, to check each 3X replication and 6+2 erasure coding. We additionally examined msgr V2 in each unencrypted and safe mode utilizing the next Ceph choices:

ms_client_mode = safe
ms_cluster_mode = safe
ms_service_mode = safe
ms_mon_client_mode = safe
ms_mon_cluster_mode = safe
ms_mon_service_mode = safe

OSDs have been allowed to make use of all cores on the nodes. FIO was configured to first pre-fill RBD quantity(s) with giant writes, adopted by 4MB and 4KB IO checks for 300 seconds every (60 seconds throughout debugging runs). Sure background processes, similar to scrub, deep scrub, PG autoscaling, and PG balancing have been disabled.

A Observe PG counts ¶

Later on this article, you may see some eye-popping PG counts being examined. That is intentional. We all know from earlier upstream lab testing that the PG depend can have a dramatic impact on efficiency. A few of this is because of clumpiness in random distributions at low pattern (PG) counts. This probably will be mitigated partially via further balancing. Much less generally mentioned is PG lock rivalry contained in the OSD. We have noticed that on very quick clusters, PG lock rivalry can play a major function in total efficiency. This sadly is much less simply mitigated with out rising PG counts. How a lot does PG depend truly matter?

With simply 60 OSDs, Random learn efficiency scales all the way in which as much as 16384 PGs on an RBD pool utilizing 3X replication. Writes high out a lot earlier, however nonetheless advantages from as much as 2048 PGs.

Let me be clear: You should not exit and blindly configure a manufacturing Ceph cluster to make use of PG counts as excessive as we’re testing right here. That is very true given a few of the different defaults in Ceph for issues like PG log lengths and PG stat updates. I do, nevertheless, wish to encourage the group to start out fascinated with whether or not the standard knowledge of 100 PGs per OSD continues to make sense. I would love us to rethink what we have to do to attain larger PG counts per OSD whereas preserving overhead and reminiscence utilization in examine. I dream a couple of future the place 1000 PGs per OSD is not out of the atypical, PG logs are auto-scaled on a per-pool foundation, and PG autoscaling is a much more seldom-used operation.

We have been first in a position to log into the brand new {hardware} the week after Thanksgiving within the US. The plan was to spend per week or two doing burn-in validation checks after which combine the brand new {hardware} into the present cluster. We hoped to complete the migration in time for the brand new yr if all the things went to plan. Sadly, we bumped into hassle proper initially. The preliminary low-level efficiency checks regarded good. Iperf community testing confirmed us hitting just below 200Gb/s per node. Random sampling of a few the nodes confirmed cheap baseline efficiency from the NVMe drives. One situation we instantly noticed was that the working system on all 68 nodes was unintentionally deployed on 2 of the OSD drives as a substitute of the inner Dell BOSS m.2 boot drives. We had deliberate to check outcomes for a 30 OSD configuration (3 nodes, 10 OSDs per node) towards the outcomes from the upstream lab (5 nodes, 6 OSDs per node). As an alternative, we ended up testing 8 NVMe drives per node. The primary Ceph outcomes have been far decrease than what we had hoped to see, even given the diminished OSD depend.

The one consequence that was even near being tolerable was for random reads, and that also wasn’t nice. Clearly, one thing was occurring. We stopped working 3-node checks and began taking a look at single-node, and even single OSD configurations.

That is when issues began to get bizarre.

As we ran totally different mixtures of 8-OSD and 1-OSD checks on particular person nodes within the cluster, we noticed wildly totally different habits, however it took a number of days of testing to essentially perceive the sample of what we have been seeing. Methods that originally carried out properly in single-OSD checks stopped performing properly after multi-OSD checks, solely to start out working properly once more hours later. 8-OSD checks would often present indicators of performing properly, however then carry out terribly for all subsequent checks till the system was rebooted. We have been ultimately in a position to discern a sample on contemporary boot that we might roughly repeat throughout totally different nodes within the cluster:

The preliminary single-OSD check regarded improbable for giant reads and writes and confirmed almost the identical throughput we noticed when working FIO checks instantly towards the drives. As quickly as we ran the 8-OSD check, nevertheless, we noticed a efficiency drop. Subsequent single-OSD checks continued to carry out poorly till a number of hours later once they recovered. As long as a multi-OSD check was not launched, efficiency remained excessive.

Confusingly, we have been unable to invoke the identical habits when working FIO checks instantly towards the drives. Simply as complicated, we noticed that through the 8 OSD check, a single OSD would use considerably extra CPU than the others:

4MB Random Learn ¶

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND 
511067 root      20   0 9360000   7.2g  33792 S  1180   3.8  15:24.32 ceph-osd                                              
515664 root      20   0 9357488   7.2g  34560 S 523.6   3.8  13:43.86 ceph-osd                                              
513323 root      20   0 9145820   6.4g  34560 S 460.0   3.4  13:01.12 ceph-osd                                              
514147 root      20   0 9026592   6.6g  33792 S 378.7   3.5   9:56.59 ceph-osd                                              
516488 root      20   0 9188244   6.8g  34560 S 378.4   3.6  10:29.23 ceph-osd                                              
518236 root      20   0 9390772   6.9g  33792 S 361.0   3.7   9:45.85 ceph-osd                                              
511779 root      20   0 8329696   6.1g  33024 S 331.1   3.3  10:07.18 ceph-osd                                              
516974 root      20   0 8984584   6.7g  34560 S 301.6   3.6   9:26.60 ceph-osd  

A wallclock profile of the OSD below load confirmed important time spent in io_submit, which is what we usually see when the kernel begins blocking as a result of a drive’s queue turns into full.

Instance tp_osd_tp Thread io_submit Wallclock Profile ¶

+ 31.00% BlueStore::readv(increase::intrusive_ptr<ObjectStore::CollectionImpl>&, g...
 + 31.00% BlueStore::_do_readv(BlueStore::Assortment*, increase::intrusive_ptr<Blu...
  + 24.00% KernelDevice::aio_submit(IOContext*)
  |+ 24.00% aio_queue_t::submit_batch(std::_List_iterator<aio_t>, std::_List_it...
  | + 24.00% io_submit
  |  + 24.00% syscall

Why would working an 8 OSD check trigger the kernel to start out blocking in io_submit throughout future single OSD checks? It did not make very a lot sense. Initially, we suspected throttling. We noticed that with the default cooling profile within the bios, a number of of the core complexes on the CPU have been reaching as much as 96 levels Celsius. We theorized that maybe we have been hitting thermal limits on both the CPU or the NVMe drives through the 8-OSD checks. Maybe that left the system in a degraded state for a time frame earlier than recovering. Sadly, that concept did not pan out. AMD/Dell confirmed that we should not be hitting throttling even at these temperatures, and we have been in a position to disprove the idea by working the programs with the followers working at 100% and a decrease cTDP for the processor. These modifications stored them constantly round 70 levels Celsius below load with out fixing the issue.

For over per week, we checked out all the things from bios settings, NVMe multipath, low-level NVMe debugging, altering kernel/Ubuntu variations, and checking each single kernel, OS, and Ceph setting we might consider. None this stuff totally resolved the difficulty.

We even carried out blktrace and iowatcher evaluation throughout “good” and “dangerous” single OSD checks, and will instantly observe the sluggish IO completion habits:

Blkparse Output – Good vs Unhealthy ¶

At this level, we began getting the {hardware} distributors concerned. Finally it turned out to be pointless. There was one minor, and two main fixes that received issues again on observe.

Repair One ¶

The primary repair was a straightforward one, however solely received us a modest 10-20% efficiency acquire. A few years in the past it was found (Both by Nick Fisk or Stephen Blinick if I recall) that Ceph is extremely delicate to latency launched by CPU c-state transitions. A fast examine of the bios on these nodes confirmed that they weren’t working in maximum performance mode which disables c-states. This was a pleasant win however not sufficient to get the outcomes the place we wished them.

Repair Two ¶

By the point I used to be digging into the blktrace outcomes proven above, I used to be about 95% positive that we have been both seeing a problem with the NVMe drives or one thing associated to the PCIe root advanced since these programs haven’t got PCIe switches in them. I used to be busy digging into technical manuals and looking for methods to debug/profile the {hardware}. A really intelligent engineer working for the shopper provided to assist out. I arrange a check setting for him so he might repeat a few of the similar testing on an alternate set of nodes and he hit a house run.

Whereas I had targeted totally on wallclock profiles and was now digging into attempting to debug the {hardware}, he wished to grasp if there was something attention-grabbing taking place kernel facet (which on reflection was the plain subsequent transfer!). He ran a perf profile throughout a nasty run and made a really astute discovery:

    77.37%  tp_osd_tp        [kernel.kallsyms]             [k] native_queued_spin_lock_slowpath
            |
            ---native_queued_spin_lock_slowpath
               |          
                --77.36%--_raw_spin_lock_irqsave
                          |          
                          |--61.10%--alloc_iova
                          |          alloc_iova_fast
                          |          iommu_dma_alloc_iova.isra.0
                          |          iommu_dma_map_sg
                          |          __dma_map_sg_attrs
                          |          dma_map_sg_attrs
                          |          nvme_map_data
                          |          nvme_queue_rq
                          |          __blk_mq_try_issue_directly
                          |          blk_mq_request_issue_directly
                          |          blk_mq_try_issue_list_directly
                          |          blk_mq_sched_insert_requests
                          |          blk_mq_flush_plug_list
                          |          blk_flush_plug_list
                          |          |          
                          |          |--56.54%--blk_mq_submit_bio

An enormous period of time is spent within the kernel contending on a spin lock whereas updating the IOMMU mappings. He disabled IOMMU within the kernel and instantly noticed an enormous enhance in efficiency through the 8-node checks. We repeated these checks a number of instances and repeatedly noticed significantly better 4MB learn/write efficiency. Rating one for the shopper. There was nevertheless nonetheless a problem with 4KB random writes.

Repair Three ¶

After being overwhelmed to the punch by the shopper on the IOMMU situation, I used to be nearly grateful that we had an extra downside to resolve. 4K random write efficiency had improved with the primary two fixes however was nonetheless considerably worse than the upstream lab (even given the diminished node/drive counts). I additionally seen that compaction was far slower than anticipated in RocksDB. There beforehand have been two important circumstances that offered equally and seemed to be related:

1. Ceph will be very sluggish when not correctly compiled with TCMalloc help.
2. Ceph will be very sluggish when not compiled with the precise cmake flags and compiler optimizations.

Traditionally this buyer used the upstream Ceph Ubuntu packages and we have been nonetheless utilizing them right here (relatively than self-compiling or utilizing cephadm with containers). I verified that TCMalloc was compiled in. That dominated out the primary situation. Subsequent, I dug out the upstream construct logs for the 17.2.7 Ubuntu packages. That is after I seen that we weren’t, actually, constructing RocksDB with the proper compile flags. It is not clear how lengthy that is been occurring, however we have had basic construct efficiency points going again so far as 2018.

It seems that Canonical fixed this for their very own builds as did Gentoo after seeing the be aware I wrote in do_cmake.sh over 6 years in the past. It is fairly unlucky that our upstream Deb builds have suffered with this for so long as they’ve, nevertheless, it at the least does not seem to have an effect on anybody utilizing cephadm on Debian/Ubuntu with the upstream containers. With the difficulty understood, we constructed customized 17.2.7 packages with a repair in place. Compaction time dropped by round 3X and 4K random write efficiency doubled (Although it’s kind of powerful to make out within the graph):

4KB random write efficiency was nonetheless decrease than I wished it to be, however at the least now we have been in roughly the precise ballpark provided that we had fewer OSDs, solely 3/5 the variety of nodes, and fewer (although quicker) cores per OSD. At this level, we have been nearing winter break. The client wished to redeploy the OS to the proper boot drives and replace the deployment with the entire fixes and tunings we had found. The plan was to take the vacation break off after which spend the primary week of the brand new yr ending the burn-in checks. Hopefully, we might begin migrating the cluster the next week.

On the morning of January 2nd, I logged into Slack and was greeted by a scene I am going to describe as reasonably managed chaos. A very totally different cluster we’re concerned in was having a serious outage. With out getting too into the small print, it took 3 days to drag that cluster again from the brink and get it right into a steady and comparatively wholesome state. It wasn’t till Friday that I used to be in a position to get again to efficiency testing. I used to be in a position to safe an additional day for testing on Monday, however this meant I used to be below an enormous time crunch to showcase that the cluster might carry out properly below load earlier than we began the info migration course of.

Destiny Smiles ¶

I labored all day on Friday to re-deploy CBT and recreate the checks we ran beforehand. This time I used to be ready to make use of all 10 of the drives in every node. I additionally bumped up the variety of purchasers to keep up a mean of roughly 1 FIO consumer with an io_depth of 128 per OSD. The primary 3 node check regarded good. With 10 OSDs per node, We have been reaching roughly proportional (IE larger) efficiency relative to the earlier checks. I knew I wasn’t going to have a lot time to do correct scaling checks, so I instantly bumped up from 3 nodes to 10 nodes. I additionally scaled the PG depend on the similar time and used CBT to deploy a brand new cluster. At 3 nodes I noticed 63GiB/s for 4MB random reads. At 10 Nodes, I noticed 213.5GiB/s. That is nearly linear scaling at 98.4%. It was at this level that I knew that issues have been lastly taking a flip for the higher. Of the 68 nodes for this cluster, solely 63 have been up at the moment. The remaining have been down for upkeep to repair numerous points. I cut up the cluster roughly in half, with 32 nodes (320 OSDs) in a single half, and 31 consumer nodes working 10 FIO processes every within the different half. I watched as CBT constructed the cluster over roughly a 7-8 minute interval. The preliminary write prefill regarded actually good. My coronary heart soared. We have been studying knowledge at 635 GiB/s. We broke 15 million 4k random learn IOPS. Whereas this may occasionally not appear spectacular in comparison with the person NVMe drives, these have been the very best numbers I had ever seen for a ~300 OSD Ceph cluster.

I additionally plotted each common and tail latency for the scaling checks. Each regarded constant. This was probably attributable to scaling the PG depend and the FIO consumer depend concurrently OSDs. These checks are very IO-heavy nevertheless. We have now a lot consumer site visitors that we’re probably properly into the inflection level the place efficiency does not enhance whereas latency continues to develop as extra IO is added.

I confirmed these outcomes to my colleague Dan van der Ster who beforehand had constructed the Ceph infrastructure at CERN. He wager me a beer (Higher be an excellent one Dan!) if I might hit 1 TiB/s. I instructed him that had been my plan for the reason that starting.

I had no further consumer nodes to check the cluster with at full capability, so the one actual possibility was to co-locate FIO processes on the identical nodes because the OSDs. On one hand, this offers a really slight community benefit. Shoppers will be capable of talk with native OSDs 1/63rd of the time. Then again, we all know from earlier testing that co-locating FIO purchasers on OSD nodes is not free. There’s typically a efficiency hit, and it wasn’t remotely clear to me how a lot of a success a cluster of this scale would take.

I constructed a brand new CBT configuration concentrating on the 63 nodes I had obtainable. Deploying the cluster with CBT took about quarter-hour to face up all 630 OSDs and construct the pool. I waited with bated breath and watched the outcomes as they got here in.

Round 950GiB/s. So very very shut. It was late on Friday evening at this level, so I wrapped up and turned in for the evening. On Saturday morning I logged in and threw a few tuning choices on the cluster: Decreasing OSD shards and async messenger threads whereas additionally making use of the Reef RocksDB tunings. As you possibly can see, we truly harm learn efficiency a short time serving to write efficiency. The truth is, random write efficiency improved by almost 20%. After additional testing, it regarded just like the reef tunings have been benign although solely serving to just a little bit within the write checks. The larger impact appeared to be coming from shard/thread modifications. At this level, I needed to take a break and wasn’t in a position to get again to engaged on the cluster once more till Sunday evening. I attempted to go to mattress, however I knew that I used to be right down to the final 24 hours earlier than we wanted to wrap this up. At round midnight I gave up on sleep and received again to work.

I discussed earlier that we all know that the PG depend can have an effect on efficiency. I made a decision to maintain the “tuned” configuration from earlier however doubled the variety of PGs. Within the first set of checks, I had dropped the ratio of purchasers to OSDs down provided that we have been co-locating them on the OSD nodes. Now I attempted scaling them up once more. 4MB random learn efficiency improved barely because the variety of purchasers grew, whereas small random learn IOPS degraded. As soon as we hit 8 FIO processes per node (504 complete), sequential write efficiency dropped via the ground.

To grasp what occurred, I reran the write check and watched “ceph -s” output:

  companies:
    mon: 3 daemons, quorum a,b,c (age 42m)
    mgr: a(lively, since 42m)
    osd: 630 osds: 630 up (since 24m), 630 in (since 25m)
         flags noscrub,nodeep-scrub
 
  knowledge:
    swimming pools:   2 swimming pools, 131073 pgs
    objects: 4.13M objects, 16 TiB
    utilization:   48 TiB used, 8.2 PiB / 8.2 PiB avail
    pgs:     129422 lively+clear
             1651   lively+clear+laggy
 
  io:
    consumer:   0 B/s rd, 1.7 GiB/s wr, 1 op/s rd, 446 op/s wr

As quickly as I threw 504 FIO processes doing 4MB writes on the cluster, a few of the PGs began going lively+clear+laggy. Efficiency tanked and the cluster did not get better from that state till the workload was accomplished. What’s worse, extra PGs went laggy over time regardless that the throughput was solely a small fraction of what the cluster was able to. Since then, we have discovered a few studies of laggy PGs on the mailing list together with a few options which may repair them. It is not clear if these concepts would have helped right here. We do know that IO will briefly be paused when PGs go right into a laggy state and that this occurs as a result of a duplicate hasn’t acknowledged new leases from the first in time. After discussing the difficulty with different Ceph builders, we predict this might presumably be a problem with locking within the OSD or having lease messages competing with work in the identical async msgr threads.

Regardless of being distracted by the laggy PG situation, I wished to refocus on hitting 1.0TiB/s. Lack of sleep was lastly catching up with me. Sooner or later I had doubled the PG depend once more to 256K, simply to see if it had any impact in any respect on the laggy PG situation. That put us solidly towards the higher finish of the curve we confirmed earlier, although frankly, I do not suppose it truly mattered a lot. I made a decision to change again to the default OSD shard counts and proceed testing with 504 FIO consumer processes. I did nevertheless scale the variety of async messenger threads. There have been two massive takeaways. The primary is that dropping right down to 1 async messenger allowed us to keep away from PGs going laggy and obtain “OK” write throughput with 504 purchasers. It additionally dramatically harm the efficiency of 4MB reads. The second: Ceph’s defaults have been truly ideally suited for 4MB reads. With 8 shards, 2 threads per shard, and three msgr threads, we lastly broke 1TiB/s. Here is the view I had at round 4 AM Monday morning as the ultimate set of checks for the evening ran:

  companies:
    mon: 3 daemons, quorum a,b,c (age 30m)
    mgr: a(lively, since 30m)
    osd: 630 osds: 630 up (since 12m), 630 in (since 12m)
         flags noscrub,nodeep-scrub
 
  knowledge:
    swimming pools:   2 swimming pools, 262145 pgs
    objects: 4.13M objects, 16 TiB
    utilization:   48 TiB used, 8.2 PiB / 8.2 PiB avail
    pgs:     262145 lively+clear
 
  io:
    consumer:   1.0 TiB/s rd, 6.1 KiB/s wr, 266.15k op/s rd, 6 op/s wr

and the graphs from the FIO outcomes:

After lastly seeing the magical “1.0 TiB/s” display I had been ready weeks to see, I lastly went to sleep. However, I received up a number of hours later. There was nonetheless work to be accomplished. All the testing we had accomplished thus far was with 3X replication, however the buyer could be migrating this {hardware} into an present cluster deployed with 6+2 erasure coding. We would have liked to get some thought of what this cluster was able to within the configuration they’d be utilizing.

I reconfigured the cluster once more and ran via new checks. I picked PG/shard/consumer values from the sooner checks that appeared to work properly. Efficiency was good, however I noticed that the async messenger threads have been working very exhausting. I made a decision to strive rising them past the defaults to see if they could assist given the added community site visitors.

We might obtain properly over 500GiB/s for reads and almost 400GiB/s for writes with 4-5 async msgr threads. However why are the learn outcomes a lot slower with EC than with replication? With replication, the first OSD for a PG solely has to learn native knowledge and ship it to the consumer. The community overhead is actually 1X. With 6+2 erasure coding, the first should learn 5 of the 6 chunks from replicas earlier than it will probably then ship the constructed object to the consumer. The general community overhead for the request is roughly (1 + 5/6)X*. That is why we see barely higher than half the efficiency of 3X replication for reads. We have now the other state of affairs for writes. With 3X replication, the consumer sends the item to the first, which then additional sends copies over the community to 2 secondaries. This ends in an combination community overhead of 3X. Within the EC case, we solely must ship 7/8 chunks to the secondaries (nearly, however not fairly, the identical because the learn case). For giant writes, efficiency is definitely quicker.

* Initially this text acknowledged that 7/8 chunks needed to be fetched for reads. The proper worth is 5/6 chunks, except quick reads are enabled. In that case it could be 7/6 chunks. Because of Joshua Baergen for catching this!

IOPS nevertheless, are one other story. For very small reads and writes, Ceph will contact all taking part OSDs in a PG for that object even when the info they retailer is not related for the operation. As an example, in case you are doing 4K reads and the info you have an interest in is completely saved in a single chunk on one of many OSDs, Ceph will nonetheless fetch knowledge from all OSDs taking part within the stripe. In the summertime of 2023, Clyso resurrected a PR from Xiaofei Cui that implements partial stripe reads for erasure coding to keep away from this further work. The impact is dramatic:

It is not clear but if we can get this merged for Squid, although Radoslaw Zarzynski, core lead for the Ceph venture, has provided to assist attempt to get this over the end line.

Lastly, we wished to offer the shopper with a tough thought of how a lot msgr-level encryption would impression their cluster in the event that they determined to make use of it. The adrenaline of the earlier evening had lengthy light and I used to be lifeless drained at this level. I managed to run via each 3X replication and 6+2 erasure coding checks with msgr v2 encryption enabled and in contrast it towards our earlier check outcomes.

The most important hit is to giant reads. They drop from ~1 TiB/s to round 750 GiB/s. Every part else sees a extra modest, although constant hit. At this level, I needed to cease. I actually wished to do PG scaling checks and even kernel RBD checks. It was time, although, at hand the programs again to the shopper for re-imaging after which to certainly one of my glorious colleagues at Clyso for integration.

So what’s occurred with this cluster for the reason that finish of the testing? All {hardware} was re-imaged and the brand new OSDs have been deployed into the shopper’s present HDD cluster. Dan’s upmap-remapped script is getting used to manage the migration course of and we have migrated round 80% of the present knowledge to the NVMe backed OSDs. By subsequent week, the cluster needs to be totally migrated to the brand new NVMe based mostly nodes. We have opted to not make use of the entire tuning we have accomplished right here, at the least not at first. Initially, we’ll be sure the cluster behaves properly below the present, largely default, configuration. We now have a mountain of information we are able to use to tune the system additional if the shopper hits any efficiency points.

Since there was a ton of information and charts right here, I wish to recap a few of the highlights. Here is a top level view of one of the best numbers we have been in a position to obtain on this cluster:

What’s subsequent? We have to work out easy methods to repair the laggy PG situation throughout writes. We won’t have Ceph falling aside when the write workload scales up. Past that, we discovered via this train that Ceph is completely able to saturating 2x 100GbE NICs. To push the throughput envelope additional we are going to want 200GbE+ when utilizing 10 NVMe drives per node or extra. IOPS is extra nuanced. We all know that PG depend can have a giant impact. We additionally know that the overall OSD threading mannequin is enjoying a giant function. We constantly hit a wall at round 400-600K random learn IOPS per node and we have seen it in a number of deployments. A part of this can be how the async msgr interfaces with the kernel and a part of this can be how OSD threads get up when new work is put into the shard queues. I’ve modified the OSD code prior to now to attain higher outcomes below heavy load, however on the expense of low-load latency. Finally, I think bettering IOPS will take a multi-pronged method and a rewrite of a few of the OSD threading code.

To my data, these are the quickest single-cluster Ceph outcomes ever revealed and the primary time a Ceph cluster has achieved 1 TiB/s. I feel Ceph is able to fairly a bit extra. You probably have a quicker cluster on the market, I encourage you to publish your outcomes! Thanks for studying, and if in case you have any questions or want to discuss extra about Ceph efficiency, please be happy to reach out.