How we organise our very giant Python monolith

2023-07-18 14:56:07

By David Seddon from Kraken Applied sciences.

Hello, I’m David, a Python developer at Kraken Applied sciences. I work on Kraken: a Python utility which has, eventually depend, 27,637 modules. Sure, you learn that proper: practically 28k separate Python information – not together with exams. I do that together with 400 different builders worldwide, continually merging in code. And all anybody must make a change – and kick begin a deployment of the software program that runs 17 completely different power and utility firms, with many tens of millions of consumers – is one single approval from a colleague on Github.

Now you could be considering this seems like a recipe for chaos. Truthfully, I’d have mentioned the identical. But it surely seems that giant numbers of builders can, no less than within the area we work in, work successfully on a big Python monolith. There are many the reason why that is doable, a lot of them cultural quite than technical, however on this weblog put up I wish to clarify about how the organisation of our code helps to make this doable.

Layering our code base

In the event you’ve labored on a code base for any size of time, you’ll have felt the drift in the direction of disagreeable complexity. Strands of logic tangle collectively throughout your utility, and it turns into more and more tough to consider elements of it in isolation. That is what began taking place to our younger code base, and so we determined to undertake what is called a ‘layered structure’ the place there are constraints about what elements of the code base can find out about one another.

Layering is a widely known software program structure sample by which elements are organized, conceptually, right into a stack. A element shouldn’t be allowed to rely upon any elements increased up the stack.

Layered Structure the place dependencies circulation downward

For instance, within the above diagram, C could be allowed to rely upon B and A, however not D.

The thought of a layered structure is broad: it could be utilized to completely different sorts of elements. For instance, you may layer a number of independently-deployable companies; or alternatively your elements may simply be a set of supply code information.

What constitutes a dependency can also be broad. On the whole, if a element has direct information of one other element (even when purely at a conceptual stage) then it is determined by it. Oblique interplay (e.g. through configuration) shouldn’t be normally seen as a dependency.

Layers in Python

In a Python code base, the layers are greatest regarded as Python modules, and dependencies as import statements.

Take the next code base:

myproject
    __init__.py
    funds/
        __init__.py
        api.py
        vendor.py
    merchandise.py
    shopping_cart.py

The highest-level modules and subpackages are good candidates for layers. Let’s say we resolve the layers ought to be on this order:

shopping_cart
funds
merchandise

Our structure would thus forbid, for instance, any of the modules inside funds from importing from shopping_cart. They may, nonetheless, import from merchandise.

Layering will also be nested, so we may select to layer inside our funds module like so:

api
vendor

There’s no single, appropriate method of selecting which layers exist, and by which order – that’s an act of design. However layering like this results in a much less tangled code base, making it simpler to grasp and alter.

How we’ve layered Kraken

On the time of writing, 17 completely different power and utility firms license Kraken. We name these firms purchasers, and run a separate occasion for every. Now, one in every of Kraken’s major traits is that completely different cases are ‘the identical, however completely different’. In different phrases, there’s loads of shared habits, but in addition each consumer has bespoke code that defines their particular wants. That is additionally true on the territory stage: there are commonalities between all of the purchasers that run in Britain (they combine with the identical power business) that aren’t shared with, say, Octopus Power Japan.

As Kraken grew right into a multi-client platform, we developed our layering to assist with this. Broadly talking, it now seems like this on the prime stage:

kraken/
    __init__.py
    purchasers/
        __init__.py
        oede/
        oegb/
        oejp/
        ...
	territories/
    	__init__.py
        deu/
        gbr/
        jpn/
        ...
    core/

The purchasers layer is on the prime. Every consumer will get a subpackage inside that layer (for instance, oede corresponds to Octopus Power Germany). Beneath that’s territories, for all of the country-specific behaviour, once more with territory-specific subpackages. The underside layer is core, which accommodates code that’s utilized by all purchasers. There may be an extra rule, which is that consumer subpackages have to be impartial (i.e. not import from different purchasers), and the identical goes for territories.

Layering Kraken like this permits us to make adjustments with a restricted ‘blast radius’. As a result of the purchasers layer is on the prime, nothing is determined by it straight, making it simpler to alter one thing that pertains to a selected consumer with out by accident affecting habits on a distinct consumer. Likewise, adjustments that relate solely to at least one territory gained’t have an effect on something in a distinct one. This enables us to maneuver rapidly and independently throughout groups, particularly after we are making adjustments that solely have an effect on a small variety of Kraken cases.

Imposing layering with Import Linter

Once we launched layering, we rapidly discovered that simply speaking in regards to the layering was not sufficient. Builders would typically by accident introduce layering violations. We would have liked to implement it by some means, and we do that utilizing Import Linter.

Import Linter is an open supply device for checking that you’re following layered architectures. First, in an INI file you outline a contract describing your layering – one thing like this

[importlinter:contract:top-level]

title = High stage layers
sort = layers
layers =
    kraken.purchasers
    kraken.territories
    Kraken.core

We will additionally implement the independence of the completely different purchasers and territories, utilizing two extra contracts (this time `independence` contracts)

[importlinter:contract:client-independence]
title = Shopper independence
sort = independence
layers =
    kraken.purchasers.oede
    kraken.purchasers.oegb
    kraken.purchasers.oejp
    ...

[importlinter:contract:territory-independence]
title = Territory independence
sort = independence
layers =
    kraken.territories.deu
    kraken.territories.gbr
    kraken.territories.jpn
    ...

Then you possibly can run lint-imports on the command line and it’ll inform you whether or not or not there are any imports that break our contracts. We run this within the automated checks on each pull request, so if somebody introduces an unlawful import, the checks will fail they usually gained’t be capable to merge it.

These will not be the one contracts. Groups can add their very own layering deeper within the utility: kraken.territories.jpn, for instance, is itself layered. We at the moment have over 40 contracts in place.

Burning down technical debt

Once we launched the layered structure, we weren’t capable of adhere to it from day one. So we used a characteristic in Import Linter which lets you ignore sure imports earlier than checking the contract.

[importlinter:contract:my-layers-contract]
title = My contract
sort = layers
layers =
    kraken.purchasers
    kraken.territories
    kraken.core
ignore_imports =
    kraken.core.prospects ->
    kraken.territories.gbr.prospects.views
    kraken.territories.jpn.funds -> kraken.utils.urls
    (and so forth...)

We then used the variety of ignored imports as a metric for monitoring technical debt. This allowed us to watch whether or not issues have been bettering, and at what price.

Right here’s our graph of how we’ve been working by means of ignored imports over the past yr or so. Periodically I share this to point out individuals how we’re doing and encourage them to work in the direction of full adherence. We use this burndown method for a number of different technical debt metrics too.

Downsides, there are all the time downsides

Native complexity

Sooner or later after adopting a layered structure, you’ll run right into a state of affairs the place you wish to break the layers. Actual life is complicated, there are interdependencies in all places, and you can see your self eager to, say, name a perform that’s in the next layer.

Thankfully, there’s all the time a method round this. It’s referred to as inversion of control and it’s simple to do in Python, it simply requires a mindset shift. But it surely does result in a rise in ‘native’ complexity (i.e. in just a little a part of your code base). Nevertheless, it’s a worth price paying for a less complicated system general.

An excessive amount of code in increased layers

The upper the layer, the simpler the change. We intentionally made it simple to alter code for particular purchasers or territories. Code within the core, which every thing is determined by, is extra pricey and dangerous to make adjustments to.

Because of this, there was a design stress, caused partly by the layering we selected, to jot down extra consumer and territory-specific quite than introduce deeper, extra globally helpful code into the core. Because of this, there’s extra code within the increased layers than we would ideally like. We’re nonetheless studying about tips on how to sort out this.

We’re nonetheless not completed

Keep in mind these ignored imports? Effectively, years on, we nonetheless have some! Ultimately depend, 15. These previous couple of imports are the stubbornest, most tangled ones of all.

It could take critical effort to retrospectively layer a code base. However the sooner you do it, the much less tangling you’ll have to deal with.

In abstract

Layering Kraken has stored our very giant code base wholesome and comparatively simple to work with, particularly contemplating its dimension. With out imposing constraints on the relationships between the tens of 1000’s of modules, our code base would in all probability have tangled into an infinite plate of spaghetti. However the giant scale construction we selected – and developed together with the enterprise – has helped us work in giant numbers on a single Python code base. It shouldn’t be doable, however it’s!

In the event you’re engaged on a big Python codebase – or perhaps a comparatively small one – give layering a strive. The earlier you do, the simpler will probably be.