How we diminished the dimensions of our JavaScript bundles by 33%

When was the final time you have been about to click on a button on a web site, solely to have the web page shift—inflicting you to click on the fallacious button as a substitute? Or the final time you rage-quit a web page that took too lengthy to load?

These issues are solely amplified in purposes as wealthy and interactive as ours. The extra front-end code is written to assist extra advanced options, the extra bytes are despatched to the browser to be parsed and executed, and the more severe efficiency can get.

At Dropbox, we perceive how extremely annoying such experiences could be. Over the previous 12 months, our net efficiency engineering staff narrowed a few of our efficiency issues right down to an oft-overlooked perpetrator: the module bundler.

Miller’s Law states that the human mind can solely maintain a lot data at any given time—which is partially why most fashionable codebases (together with ours) are damaged up into smaller modules. A module bundler takes the assorted elements of an utility—resembling JavaScript and CSS—and amalgamates them into bundles, that are then downloaded by the browser when a web page is loaded. Mostly, this takes the type of a minified JavaScript file that comprises a lot of the logic for an online app.

The primary iteration of our module bundler was conceived means again in 2014—across the time that performance-first approaches to module bundling have been gaining popularity (most notably by Webpack and Rollup in 2012 and 2015, respectively). Because of this, it was fairly barebones relative to extra fashionable choices; our module bundler didn’t incorporate many efficiency optimizations and was onerous to work with, hampering our person expertise and slowing down growth velocity.

Because it turned clear our present bundler was exhibiting its age, we determined one of the simplest ways to optimize efficiency going ahead could be to interchange it. That was additionally the right time to take action since we have been in the midst of migrating our pages to Edison—our new web serving stack—which offered a chance to piggyback on an present migration plan and likewise offered an structure that made it less complicated to combine a contemporary bundler into our static asset pipeline.

Whereas our present bundler was comparatively build-time environment friendly, it resulted in huge bundle sizes and proved to be a burden for engineers to keep up. We relied on engineers to manually outline which scripts to bundle with a package deal, and we merely shipped all packages concerned in rendering a web page with few optimizations. Over time, the issues with this strategy turned clear:

Drawback #1: A number of variations of bundled code
Till lately we used a customized net structure known as Dropbox Web Server (DWS). In brief, every web page consisted of a number of pagelets (i.e. subsections of pages), leading to a number of JS entry factors per web page, with every servlet being served by its personal controller on the backend. Whereas this sped-up deployment in instances the place a web page was being labored on by a number of groups, it generally resulted in pagelets being on totally different backend code variations. This required DWS to assist delivering separate variations of packaged code on the identical web page, which may doubtlessly end in consistency points (e.g. a number of situations of a singleton being loaded on the identical web page). Our migration to Edison would eradicate this pagelet structure, giving us the flexibleness to undertake a extra industry-standard bundling scheme.

Drawback #2: Guide code-splitting
Code splitting is the method of splitting a JavaScript bundle into smaller chunks, in order that the browser solely hundreds the components of the codebase which might be mandatory for the present web page. For instance, assume a person visits dropbox.com/house, then dropbox.com/recents. With out code-splitting, your entire bundle.js is downloaded, which may considerably decelerate the preliminary navigation to a web page.

After code-splitting, nonetheless, solely the chunks wanted by the web page are downloaded. This hurries up the preliminary navigation to dropbox.com/house, since much less code is downloaded by the browser—and has a number of further advantages too. Vital scripts are loaded first, after which non-critical scripts are loaded, parsed, and executed asynchronously. Shared items of code are additionally cached by the browser, additional lowering the quantity of JavaScript downloaded when shifting between pages. The entire above can tremendously cut back the load time of net apps. 

Since our present bundler didn’t have any built-in code-splitting, engineers needed to manually outline packages. Extra particularly, our packaging map was a large 6,000+ line dictionary that specified which modules have been included during which package deal.

As you may think about, this turned extremely advanced to keep up over time. To keep away from sub-optimal packaging, we enforced a rigorous set of assessments—the packager assessments—which turned dreaded by engineers since they might typically require a handbook reshuffling of modules with every change.

This additionally resulted in much more code than what was wanted by sure pages. As an illustration, assume now we have the next package deal map:

If a web page relies on modules a, b, and c, the browser would solely must make two HTTP calls (i.e. to fetch pkg-a and pkg-b) as a substitute of three separate calls, as soon as per module. Whereas this would scale back the HTTP name overhead, it could typically end in having to load pointless modules—on this case, module d. Not solely have been we loading pointless code attributable to an absence of tree shaking, however we have been additionally loading whole modules that weren’t mandatory for a web page, leading to an general slower person expertise. 

Drawback #3: No tree shaking
Tree shaking is a bundle-optimization approach to scale back bundle sizes by eliminating unused code. Let’s assume your app imports a third-party library that comprises a number of modules. With out tree shaking, a lot of the bundled code is unused.

With tree shaking, the static construction of the code is analyzed and any code that isn’t instantly referenced by different code is eliminated. This ends in a remaining bundle that’s a lot leaner.

Since our present bundler was barebones, there wasn’t any tree shaking performance both. The ensuing packages would typically include giant swaths of unused code, particularly from third-party libraries, which translated to unnecessarily longer wait instances for web page hundreds. Additionally, since we used Protobuf definitions for environment friendly knowledge switch from the front-end to the back-end, instrumenting sure observability metrics would typically find yourself introducing a number of further megabytes of unused code!

Though we thought-about many options through the years, we realized that our main requirement was having sure options like automated code-splitting, tree shaking, and, optionally, some plugins for additional optimizing the bundling pipeline. Rollup was probably the most mature on the time and most versatile to include into our present construct pipeline, which is principally why we settled on it. 

Another excuse: much less engineering overhead. Since we have been already utilizing Rollup for bundling our NPM modules (albeit with out lots of its helpful options), increasing our adoption of Rollup would require much less engineering overhead than integrating a wholly overseas device in our construct course of. Moreover, this meant that we had extra engineering experience with Rollup’s quirks in our codebase versus that of different bundlers, lowering the the chance of so-called unknown unknowns. Additionally, replicating Rollup’s options inside our present module bundler would require considerably extra engineering time than if we simply built-in Rollup extra deeply in our construct course of.

We knew that rolling out a module bundler safely and progressively could be no straightforward feat, particularly since we’d must reliably assist two module bundlers (and consequently, two totally different units of generated bundles) on the similar time. Our main issues included making certain secure and bug-free bundled code, the elevated load on our construct methods and CI, and the way we might incentivize groups to opt-in to utilizing Rollup bundles for the pages they owned. 

With reliability and scalability in thoughts, we divided the rollout course of to 4 levels:

• The developer preview stage allowed engineers to opt-in to Rollup bundles of their dev setting. This allowed us to successfully crowdsource QA testing by having builders floor any surprising utility conduct launched by Rollup bundles early on, giving us loads of time to handle bugs and scope modifications. 
• The Dropboxer preview stage concerned serving Rollup bundles to all inner Dropbox staff, which allowed us to assemble early efficiency knowledge and additional collect suggestions on any utility behavioral modifications.
• The final availability stage concerned progressively rolling out to all Dropbox customers, each inner and exterior. This solely occurred as soon as our Rollup packaging was totally examined and deemed secure sufficient for customers. 
• The upkeep stage concerned addressing any tech debt left over within the undertaking and iterating on our use of Rollup to additional optimize efficiency and the developer expertise. We realized that initiatives of such a large scale will inevitably find yourself accumulating some tech debt, and we should always proactively plan to handle it at some stage as a substitute of sweeping it beneath the rug.

To assist every of those levels, we used a mixture of cookie-based gating and our in-house feature-gating system. Traditionally, most rollouts at Dropbox are completely carried out utilizing our in-house characteristic gating system; nonetheless, we determined to permit cookie-based gating to shortly toggle between Rollup and legacy packages, which sped up debugging. Nested inside every of those rollout levels have been gradual rollouts, which concerned ramping up from 1%, 10%, 25%, 50%, to 100%. This gave us the flexibleness to gather early efficiency and stability outcomes—and to seamlessly roll-back any breaking modifications in the event that they occurred—whereas minimizing influence to each inner and exterior customers.

See Also

2023 in evaluation – Ruffle

Due to the massive variety of pages we needed to migrate, we not solely wanted a method to change pages over to Rollup safely, but in addition to incentivize web page house owners to change within the first place. Since our net stack was about to bear a significant renovation with Edison, we realized that piggybacking on Edison’s rollout may resolve each our issues. If Rollup was an Edison-only characteristic, developer groups would have better incentive emigrate to each Rollup and Edison, and we may tightly couple our migration technique with Edison’s too. 

Edison was additionally anticipated to have its personal efficiency and growth velocity enhancements. We figured that coupling Edison and Rollup collectively would have a transformational synergy strongly felt all through the corporate.

Whereas we did anticipate to run into some surprising challenges, we realized that daisy-chaining one construct system (Rollup) with one other (our present Bazel-based infrastructure) proved to be more difficult than anticipated.

Firstly, working two totally different module bundlers on the similar time proved to be extra resource-intensive than we estimated. Rollup’s tree-shaking algorithm, whereas fairly mature, nonetheless needed to load all modules into reminiscence and generate the summary syntax bushes wanted to investigate relationships and shake code out. Additionally, our integration of Rollup into Bazel restricted us in having the ability to cache middleman construct outcomes, requiring our CI to rebuild and re-minify all Rollup chunks on every construct. This precipitated our CI builds to time-out attributable to reminiscence exhaustion, and delayed the rollout considerably. 

We additionally discovered a number of bugs with Rollup’s tree-shaking algorithm which resulted in overly aggressive tree shaking. Fortunately, this solely resulted in minor bugs that have been caught and glued throughout the developer preview part with out ever impacting our customers. Moreover, we discovered that our legacy bundler was serving some code from third-party libraries that was incompatible with JavaScript’s strict mode. Serving this similar code through the brand new bundler with strict mode enabled resulted in fail-hard runtime errors within the browser. This required us to conduct a one-time audit of our whole codebase and patch code that was incompatible with strict mode.

Lastly, throughout the Dropboxer preview part, we discovered that our A/B telemetry metrics between Rollup and the legacy bundler weren’t exhibiting as a lot of a TTVC improvement as we anticipated. We finally narrowed this right down to Rollup producing much more chunks than what our legacy packager produced. Though we initially hypothesized that HTTP2’s multiplexing would negate any efficiency degradations from a better variety of chunks, we discovered that too many chunks would consequence within the browser spending considerably extra time in discovering all of the modules wanted for the web page. Growing the variety of chunks additionally resulted in decrease compression effectivity, since compression algorithms resembling Zlib use a sliding-window strategy to compression, which ends up in better compression effectivity for one giant file vs. many smaller information.

After rolling out Rollup to all Dropbox customers, we discovered that this undertaking diminished our JavaScript bundle sizes by 33%, our complete JavaScript script rely by 15%, and yielded modest TTVC enhancements. We additionally considerably improved entrance finish growth velocity by means of automated code-splitting, which eradicated the necessity for builders to manually shuffle round bundle definitions with every change. Lastly and maybe most significantly, we introduced our bundling infrastructure into modernity and slashed years of tech debt accrued since 2014, lowering our upkeep burden going ahead.

Along with having a extremely impactful rollout, the Rollup undertaking revealed a number of bottlenecks in our present structure—for instance, a number of render-blocking RPCs, extreme perform calls to third-party libraries, and inefficiencies in how the browser hundreds our module dependency graph. Given Rollup’s wealthy plugin ecosystem, addressing such bottlenecks has by no means been simpler in our codebase. 

Total, adopting Rollup totally as our module bundler has not solely resulted in instant efficiency and productiveness positive factors, however will even unlock important efficiency enhancements down the highway.

~ ~ ~

If constructing progressive merchandise, experiences, and infrastructure excites you, come construct the long run with us! Go to dropbox.com/jobs to see our open roles, and comply with @LifeInsideDropbox on Instagram and Facebook to see what it is prefer to create a extra enlightened means of working.