When Debug Symbols Get Massive
TL;DR – improve your instruments, together with Visible Studio, windbg, and Home windows Efficiency Toolkit, if you wish to deal with Chromium’s image information.
Particulars:
Loss of life, taxes, and browser engines relentlessly rising – these are the three issues which you can actually make certain of. And so it was in early 2020 after I realized that Chromium’s inexorable progress meant that we had been ultimately going to provide PDB (Home windows debug image) information that exceeded the PDB format’s 4 GiB restrict.
I filed a Visual Studio bug in February 2020 requesting that the restrict be raised, and three years and three days later we flipped the switch in order that Chromium can produce bigger PDBs. At that time the PDB for Chrome was at 95% of 4 GiB, and several other take a look at binaries had already crossed over the brink, so it was simply in time.
My understanding of the PDB format is that it’s web page based mostly. The format permits two to the twentieth (2^20, or 1,048,576) pages, and the default web page dimension is 4 KiB. These two numbers multiplied collectively give the utmost dimension of 4 GiB. The utmost variety of pages can’t be elevated, however the web page dimension may be. In truth, the PDB format all the time acknowledged the necessity for various web page sizes, however primarily no instruments had ever supported a web page dimension apart from 4 KiB.
“All” that needed to be accomplished to help bigger PDBs was to replace just a few instruments to help bigger pages. The instruments that wanted updating included:
Visible Studio debugger- hyperlink.exe – Microsoft’s linker
- lld-link – the linker utilized by Chromium
- windbg and different debuggers (kd.exe, ntsd.exe, and so forth.)
- dbghelp.dll (used for loading symbols)
- pdbstr.exe (used for supply indexing)
- symstore.exe (used for importing to image servers)
- msdia140.dll (COM API for loading symbols)
- Windows Performance Analyzer (WPA, the ETW hint viewer)
- Most likely different instruments
Straightforward!
In line with feedback on the bug an inside repair was launched at Microsoft in August, 2021. By December this repair had shipped in updates to Visible Studio 2019 and within the simply released Visual Studio 2022, and lld-link additionally supported it. As soon as the linkers supported bigger web page sizes we added a use_large_pdbs construct setting which might switch the PDB page size to 8 KiB. Nevertheless this setting initially wanted to be off by default as a result of a scarcity of full instrument help.
Ideally the opposite instrument updates would have shipped within the fall of 2021 however… they didn’t. I reached out to the Windbg and WPA groups and – although I may very well be fallacious – I obtained the impression that they didn’t notice that they wanted to ship updates till I instructed them.
There are numerous launch channels for windbg (and the numerous instruments related to it) and WPA so it’s exhausting to say when fixes to those instruments first shipped. Massive PDB help appeared within the Microsoft Retailer variations someday in 2022, and in nuget, but it surely wasn’t till fall 2022 when the Home windows 11 22H2 SDK shipped that it was sensible to replace all the variations of all of those instruments.
The opposite factor that held us again was Home windows 7. Newer variations of dbghelp.dll had issues working on Home windows 7. So, so long as we had been working checks on Home windows 7 we needed to generate PDBs that labored with the previous variations of dbghelp.dll. Version 109 of Chrome is the last version that supports Windows 7 in order quickly as we had branched for that model I might begin engaged on massive PDB help on trunk.
Whereas engaged on the repair I hit various mysterious failures – you possibly can see this by noticing that my first public try and create a change to switch to large PDBs was on December 28th – greater than a month earlier than the change landed. The primary drawback is that one take a look at was stubbornly failing to load the up to date PDB information. I step by step narrowed that all the way down to the take a look at failing to load the brand new model of dbghelp.dll. I then wasted a whole lot of time determining which DLL import was inflicting the issues earlier than lastly realizing that… the brand new dbghelp.dll was not being deployed to the take a look at machines. It’s tough to load a DLL that isn’t there. Hilariously, we had not been deploying dbghelp.dll to our take a look at machines for years, however the system model was ok so it hadn’t mattered, so this bug was by no means beforehand observed. I deployed some fairly cool “loader snaps” diagnostics (based mostly on this tool) however in the long run it was overkill for “DLL not current” being the rationale.
The following mysterious failure solely occurred on our official builders, which drastically difficult testing. This failure was as a result of the fallacious model of msdia140.dll was being loaded. In an virtually excellent parallel to the primary concern we had by no means correctly been deploying msdia140.dll. I nonetheless don’t know the place it had been loaded from earlier than, however copying the suitable model to our output listing resolved that.
Lastly, 37 days and 38 patch units after importing the primary model of my change I landed it. The brand new PDB dimension restrict is 8 GiB however we must always be capable of improve the web page dimension to double this once more anytime we wish to.
Modifications like this could simply be disruptive – it’s exhausting to anticipate all the hidden dependencies or locations this may fail – so I used to be happy when the change landed and I heard virtually nothing again. A few individuals complained that varied instruments couldn’t load Chrome Canary symbols however in all instances an replace of the mandatory instruments to the most recent Home windows 11 SDK solved the issue. Word that the Home windows SDK doesn’t set up the Debuggers and Home windows Efficiency Toolkit (which accommodates WPA) by default so for those who rely on these instruments it is advisable make sure you choose them when putting in or else you’ll nonetheless have the previous variations.
Most individuals don’t work on Chromium, however for those who run Chrome it’s possible you’ll sooner or later need to have the ability to profile or debug the official builds you might be working. You possibly can configure your tools to point at Chrome’s 
image server after which symbols and supply might be downloaded on demand – however solely if you’re working large-PDB suitable instruments. I do know that some recreation studios are additionally hitting the 4 GiB restrict, so these builders additionally want the most recent instruments.
I’ve up to date UIforETW so that it’ll set up the most recent model of WPA while you run it. Yow will discover the most recent launch (at the moment 1.58) here.
