Constructing a world deployment platform is tough, right here is why

Multi-region or not, whenever you want to deploy an application on our platform, it all begins with a POST API call against our API with the desired deployment definition. A deployment definition describes how your app should be deployed and roughly looks like this:

{
  "name": "my-cool-website", 
  "type": "WEB",
  "routes": [
    {
      "port": 3000, 
      "path": "/" 
    }
  ],
  "ports": [
    {
      "port": 3000,
      "protocol": "http"
    }
  ],
  "docker": {
    "image": "docker.io/koyeb/demo", 
    "command": "",
    "args": []
  }
  
}

Our API server stores this in a database and a Golang worker starts an elaborate boot process. The schema belows describes the components at play.

Authentic structure we ran to handle apps

Major architectural changes like this have a long-lasting impact: these decisions can be carried over during 10 years.
We needed a future-proof architecture that would hold its ground for at least two or three years to come and support at least 25 locations, actually up to 100 locations.

We first laid down our requirements and wishes. Our three main wishes were to:

1. Provision new regions fast: our goal is to have dozens of locations available for our users, so it should be business as usual to spawn new locations
2. Sustain partial outages:

• A failing region should not bring down our infrastructure as a whole
• A failure in Koyeb components should not affect the workloads of our users

3. Reach our target architecture gradually: we wanted to start deploying new regions in the coming months, not years. Hence, we wanted to be able to ship a first version fast but that we could iterate on to improve over time

Given these requirements, we started exploring different ideas.

• Decentralized architecture: We have plans to deploy components all around the world. The physical distance between these components will lead to high latencies, which would make this architecture challenging to maintain.
• Dedicated virtual machines per region: We thought about having dedicated virtual machines which would host just some core services like Kuma, nearby the bare metal machines. It was tempting because of the low cost. However, we ruled it out too because we thought that we probably would need to enlarge those VMs to host more and more services over time and that this wouldn’t be flexible with a non-negligeable cost of maintenance.

Some early concepts that we had

In the end, we settled on a federated-like hierarchy to distribute our systems around the world: one global component, to which are attached a few smaller components, to each are attached a few smaller components, and so on.

The great thing about federation is that it is simple. The problem is that your global component does not scale well. We tried to shoot for something “in-between” that would allow us to move the stuff that does not scale well from the top-level components to the lower-level ones.

We defined 4 kinds of components scopes: global, continental, regional, data center-level and settled down on the following topology:

Goal topology

The way we answered the reliability problem was to consider each region as an independent satellite.

Engineering is all about trade-offs and when we settled on this design, we had to make some:

1. There is a global cluster.

   Our global cluster is there to host… global resources. Sounds like a weak link, right? It might be, but we believe that it is way easier to manage a global cluster and that we can greatly mitigate the impact of outages on this component.

   First, the risks of an outage of that Kubernetes cluster are low because it can be distributed across multiple availability zones. Then, the target architecture just described allows regions (and continents) to run independently in case of a global cluster outage.

2. We decided to run one Nomad cluster per region.

   Nomad allows us to natively reschedule Instances to some other servers if one of them crashes. Having one Nomad cluster per region effectively prevents us from leveraging its native failover policies to reschedule jobs across regions in case of regional outages. So, we have to handle that failure scenario by ourselves; that is more work.

   On the other hand, we believe that it is key to achieving our vision of independent, satellite regions. Theoretically, a single Nomad cluster is supposed to be able to orchestrate thousands of tasks, all over the globe. However, by splitting the Koyeb World into multiple regions orchestrated by multiple Nomad clusters, we reduce the impact of a Nomad cluster failing.

This design allows us to iterate quickly and to progressively improve availability.
Continental clusters have a privileged latency to bare metal machines (they are physically closer). We aim to move a lot of stuff there to improve performance and reduce costs.

For now, we settled on moving only the strictly necessary sotware on continental clusters (Kuma and Nomad). Over time, we will port more components there as we need it.

After all of this thinking, it was finally time to get our hands dirty! We laid down our specifications for the first version of our multi-regions deployment engine: keep it simple and migrate only the strictly necessary stuff over to continental clusters.

Before deploying a new region in the US, we decided to first make our European region comply with this new architecture.

At the time we had one single region. We decided to start from scratch with a new region, transparently migrate all of our users over there, and then bid farewell to the old setup. This was simpler than trying to build the target architecture while maintaining the original setup; a probably painful experience that we avoided.

We made very few changes to our original global cluster:

We provisioned a new cluster, our European continental cluster. Then, we dedicated a Kubernetes namespace for the new Frankfurt region (fra).

In there, we put:

• A zonal kuma control plane, that we plugged to the global control plane
• A set of nomad servers for the region

We put a load balancer in front of nomad-server and kuma-cp because our bare metal machines would need to talk to them over the Internet. We protected those services with mutual TLS.

Design of management planes

This design makes it tremendous straightforward to bootstrap a management aircraft. A management aircraft is just a set of YAML manifests. We will then apply it to an present continental Kubernetes cluster. It may be packaged in a Helm chart or a Kustomize configuration tree, for simplicity.

Lastly, we deployed our knowledge aircraft: a handful of naked metallic machines situated in Frankfurt. We configured the companies (e.g. nomad-agent) on these hosts to focus on the model new regional management aircraft and voilà, the area was prepared! We simply needed to make our employee conscious of it and launch it… wait. That’s harder than it sounds.

So, we had to make the code changes in our APIs and workers to handle multi-region deployments. When users deploy an app on the platform, they push to us a deployment definition to describe the desired deployment: how much RAM should we allocate, how many instances should run, what is the image or GitHub repo to use…

First of all, at that time, a Service could only be deployed in a single region. We now wanted users to deploy the same service in different regions and potentially override some values for some regions. For example, if they want bigger instances of a service in a given region because this is where most of their users are, they should be able to.

We split the concept of deployment definition into two: regional deployment definition and deployment definition.

A deployment definition would hold the Service definition for all of the regions and all of the overrides. We added a mechanism to derive, for each region defined in a deployment definition, a regional deployment definition, which is the view of the deployment for a given region. In that way, the regional deployment definition is very close to what the original deployment definition was.

This allowed us to perform minimal changes to the existing worker. It just needs a regional deployment definition, and the right Nomad and Kuma clients:

type RegionalDeploymentDefinition struct {
    Region  string 
    Scaling uint   
    MemMB   uint   
    Image   string 
    
}

var nomadClients map[string]*nomad.Client

func init() {
    nomadClients = map[string]*nomad.Client{
        
        
        
        
        "fra": nomad.NewClient("http://nomad-api-access.fra:4646"),
        "was": nomad.NewClient("http://nomad-api-access.was:4646"),
        
    }
}


func DeployService(ctx context.Context, req *RegionalDeploymentDefinition) error {
    
    nomadClient, ok := nomadClients[req.Region]
    if !ok {
        return errors.New("this region is not available")
    }

    
    spec := req.ToNomadSpec()
    res, err := nomadClient.RegisterJob(ctx, spec)
    if err != nil {
        return errors.Join(err, errors.New("cannot create Nomad Job"))
    }

    
    
}

We worked for months to deploy Frankfurt. Truth is, from a user point of view, nothing much had changed: we still only offered a single location!

However, we set up pretty much all of the machinery to onboard new locations easily. And boy the architecture design paid off: we then deployed a new region in Washington in less than a month! All we needed to do was to do the same thing over again:

• Provision a new continental cluster in the US
• Provision a regional control plane in there
• Provision new bare metal machines in Washington, DC

Then, we once again ran automated tests until we slowly opened the region to our users.

With these two regions live, we were able to validate some features (and ship bugfixes 🤫) we were willing to offer for multi-region apps:

• All of your services can privately reach each other via DNS. In practice, it means that you can curl http://my-other-service.koyeb:8080 from your code and reach your other service. Traffic is transparently encrypted with mutual TLS and we take care of routing requests to the closest healthy instance where your code is running
• Inbound HTTP requests take the fastest path to reach your service. Once again, in practice, it means that when someone reaches the public URL of your app, our load balancing stack will pick up the request at the closest edge location to the user and route it to the closest healthy instance where your code is running

Those are features that you get out-of-the-box when deploying an app replicated on more than one region on us – we believe that they are great for global workloads.

We have a ton of ideas to improve our multi-region engine.

Now, if you are attentive to details, you might have noticed that we did not mention where some of our components like our container image registry, Harbor, were in this new architecture. For example, we retrieve metrics and logs from the services running on bare metal machines to display this beautiful view:

This telemetry needs to be queryable and hence, stored somewhere. It is first crafted on bare metal machines but it needs to make its way to some database. Where is that database? For now, these kinds of components live in our global cluster.

    😠 But we said that the bare metal machines should communicate directly to the global cluster!

Correct. That is suboptimal and we know it.

As stated before, we plan on porting more and more components from our global cluster to continental clusters. It should boost performance, improve reliability and reduce our costs.

We would like to make it a no-op for us to introduce a new region: it would be great if we could prepare end-to-end continuous deployment strategies to provision new regions, run automated tests and slowly roll them out. The same goes for rolling out configuration changes; we would love to make this frictionless and safe.