QEMU AioContext elimination and the way it was achieved

This publish is in regards to the AioContext lock elimination in QEMU 9.0 (deliberate for launch in 2024), how we received right here, and what it means for multi-threaded code in QEMU.

Early QEMU as a single-threaded program

Till 2009 QEMU was largely a single-threaded program. This had the profit
that the code did not want to think about thread-safety and was thus less complicated and
much less bug-prone. The principle loop interleaved operating the following piece of visitor code
and dealing with exterior occasions equivalent to timers, disk I/O, and community I/O. This
structure had the draw back that emulating multi-processor visitors was
bottlenecked by the only host CPU on which QEMU ran. There was no parallelism
and this turned problematic as multi-processor visitors turned fashionable.

Multi-threading with vCPU threads and the Massive QEMU Lock

The structure was modified to assist operating devoted vCPU threads for KVM visitors. This made parallelism doable for multi-processor visitors however the function was initially solely accessible for KVM visitors. The Multi-Threaded TCG (MTTCG) function ultimately allowed translated code
to additionally make the most of vCPU threads in 2016.

A simple strategy to creating all present code thread-safe was taken: the Massive QEMU
Lock (BQL) was launched to serialize entry to QEMU’s inside state. The BQL is a single world mutex that’s used to guard the vast majority of QEMU’s inside state. KVM vCPU threads don’t want entry to
QEMU’s inside state whereas executing visitor code, so they do not maintain the BQL more often than not. The principle loop thread drops the BQL whereas blocking in ppoll(2) and this permits vCPU threads to accumulate the lock once they come out of visitor code.

Multi-threading with IOThreads and the AioContext lock

Though the vCPU bottleneck had been solved, gadget emulation nonetheless ran with the BQL held. This meant that solely a single QEMU thread might course of I/O requests at a time. For I/O sure workloads this was a bottleneck and particularly disk I/O efficiency suffered as a result of this limitation. My first try at eradicating the bottleneck in 2012 amounted to writing a brand new “dataplane” code path exterior the BQL, however it lacked the options that customers wanted like disk picture file codecs, I/O throttling, and so forth as a result of it could not use the present code that relied on the BQL. The long run answer can be introducing thread-safety to the present code and that led to the creation of the AioContext lock.

The AioContext lock was like a mini-BQL however for an occasion loop (QEMU calls this an AioContext) as an alternative of your complete program. Initially the occasion loop would purchase the lock whereas operating occasion handlers, thereby guaranteeing mutual exclusion for all handlers related to the occasion loop. One other thread might purchase the lock to cease the occasion loop from operating and safely entry variables. This was a crude strategy although and propagated the BQL mind-set additional. QEMU started to endure from deadlocks and race circumstances now that multi-threading was doable. Though I wrote developer documentation about how the mannequin labored, it turned tough to realize confidence within the security of the code as the entire QEMU block layer wanted to grapple with AioContext locking and did so incompletely and inconsistently.

The upshot of all of this was that disk I/O processing might run in a devoted occasion loop thread (QEMU calls this an IOThread) whereas the QEMU monitor might purchase the AioContext lock for a short second to examine the emulated disk for an “information block” monitor command, for instance. In contrast to the sooner “dataplane” strategy, it was now doable for the QEMU block layer to run exterior the BQL and as an alternative depend on the AioContext lock.

Eradicating the AioContext lock

Paolo Bonzini had the concept to step by step remove the AioContext lock in favor of fine-grained locks as a result of we stored hitting issues with the AioContext lock that I described above. His perception was to alter the mannequin in order that handler capabilities would explicitly take their AioContext’s lock as an alternative buying the lock across the whole occasion loop iteration. The benefit to letting handlers take the lock was that they may additionally substitute it with one other mechanism. Ultimately it could be doable to maneuver away from the AioContext lock.

What got here after was a multi-year journey that I credit score to Paolo’s imaginative and prescient. Emanuele Giuseppe Esposito labored with Paolo on placing fine-grained locking into apply and on sorting via your complete QEMU block layer to find out beneath which threads and locks variables have been accessed. This was an enormous effort and required a variety of persistence. Kevin Wolf discovered how you can use clang’s Thread Security Evaluation (TSA) to verify among the locking guidelines at compile time. Kevin additionally spent a variety of time defending the block driver graph with a reader/author lock in order that in-flight I/O doesn’t crash amidst modifications to the graph. Emanuele and Kevin gave a chat at KVM Discussion board 2023 in regards to the bigger QEMU multi-queue block layer effort and the slides can be found here (PDF).

As soon as every little thing that beforehand relied on the AioContext lock had switched to a different type of thread-safety, it was doable to take away the AioContext lock as nothing used it anymore. The BQL continues to be broadly used and covers world state that’s accessed from few threads. Code that may run in any IOThread now makes use of its personal locks or different mechanisms. The complexity of the codebase continues to be roughly the identical as with the AioContext lock, however now there are fine-grained locks, that are simpler to know and there are fewer undocumented locking assumptions that led to deadlocks and races up to now.

Conclusion

QEMU’s AioContext lock enabled multi-threading however was additionally related to deadlocks and race circumstances as a result of its ambient nature. From QEMU 9.0 onwards, QEMU will swap to fine-grained locks which can be extra localized and make thread-safety extra express. Altering locking in a big program is time-consuming and troublesome. It took a multi-person multi-year effort to finish this work, however it types the idea for additional work together with the QEMU multi-queue block layer effort that push multi-threading additional in QEMU.