Kubernetes The Onerous Means – Developer Pleasant Weblog
You would possibly’ve solved this problem approach prior to I tried it. Nonetheless, I at all times wished to undergo the method because it has many angles and studying the main points intrigues me.
This model, nonetheless, doesn’t use any cloud supplier. Particularly, the issues I’m utilizing in another way from the unique problem are:
- Vagrant & VirtualBox: For the nodes of the cluster
- Ansible: For configuring every little thing till the cluster is prepared
- Cilium: For the community CNI and as a alternative for the kube-proxy
So, right here is my story and the way I solved the well-known “Kubernetes The Onerous Means” by the good Kelsey Hightower. Keep tuned should you’re within the particulars.
Introduction
Kubernetes the Onerous Means is a good train for any system administrator to actually get into the nit and grit of Kubernetes and work out how completely different elements work collectively and what makes it as such.
In case you have solely used a managed Kubernetes cluster, or used kubeadm to spin up one, that is your likelihood to actually perceive the inside workings of Kubernetes. As a result of these instruments summary numerous the main points away from you, which isn’t serving to to know the implementation particulars when you have a knack for it.
Goal
The entire level of this train is to construct a Kubernetes cluster from scratch, downloading the binaries, issuing and passing the certificates to the completely different elements, configuring the community CNI, and eventually, having a working Kubernetes cluster.
With that introduction, let’s get began.
Conditions
First issues first, let’s make certain all the mandatory instruments are put in on our system earlier than we begin.
Instruments
All of the instruments talked about beneath are the newest variations on the time of writing, February 2024.
Alright, with the instruments put in, it is time to get our fingers soiled and actually get into it.
The Vagrantfile
Data
The Vagrantfile configuration language is a Ruby DSL. If you’re not a Ruby developer, fret not, as I am not both. I simply know sufficient to get by.
The preliminary step is to have three nodes up and operating for the Kubernetes cluster, and one for the Load Balancer utilized in entrance of the API server. We can be utilizing Vagrant on high of VirtualBox to create all these nodes.
These can be Digital Machines hosted in your native machine. As such, there isn’t any cloud supplier wanted on this model of the problem and all of the configurations are carried out regionally.
The configuration for our Vagrantfile appears as beneath.
field = "ubuntu/jammy64"
N = 2
common_script = <<~SHELL
export DEBIAN_FRONTEND=noninteractive
sudo apt replace
sudo apt improve -y
SHELL
Vagrant.configure("2") do |config|
config.vm.outline "lb" do |node|
node.vm.field = field
node.vm.community :private_network, ip: "192.168.56.100", hostname: true
node.vm.community "forwarded_port", visitor: 6443, host: 6443
node.vm.hostname = "lb.native"
node.vm.supplier "virtualbox" do |vb|
vb.title = "k8s-the-hard-way-lb"
vb.reminiscence = "1024"
vb.cpus = 1
vb.linked_clone = true
finish
node.vm.synced_folder "share/dl", "/downloads", create: true
node.vm.provision "shell", inline: common_script
node.vm.provision "ansible" do |ansible|
ansible.verbose = "vv"
ansible.playbook = "bootstrap.yml"
ansible.compatibility_mode = "2.0"
finish
finish
(0..N).every do |machine_id|
config.vm.outline "node#{machine_id}" do |node|
node.vm.field = field
node.vm.hostname = "node#{machine_id}.native"
node.vm.community :private_network, ip: "192.168.56.#{machine_id+2}", hostname: true
node.vm.supplier "virtualbox" do |vb|
vb.title = "k8s-the-hard-way-node#{machine_id}"
vb.reminiscence = "1024"
vb.cpus = 1
vb.linked_clone = true
finish
# To carry the downloaded gadgets and survive VM restarts
node.vm.synced_folder "share/dl", "/downloads", create: true
node.vm.provision "shell", inline: common_script
if machine_id == N
node.vm.provision :ansible do |ansible|
ansible.restrict = "all"
ansible.verbose = "vv"
ansible.playbook = "bootstrap.yml"
finish
finish
finish
finish
finish
Non-public Community Configuration
There are a few vital notes price mentioning about this config, highlighted within the snippet above and the next listing.
The community configuration, as you see above, is a non-public community with hard-coded IP addresses. This isn’t a tough requirement, nevertheless it makes numerous the upcoming assumptions loads simpler.
Dynamic IP addresses will want extra cautious dealing with with regards to configuring the nodes, their TLS certificates, and the way they impart total.
And tackling craziness on this problem is a certain approach to not go down the rabbit gap of despair 
.
Load Balancer Port Forwarding
For some purpose, I wasn’t capable of immediately name 192.168.56.100:6443, which is the deal with pair obtainable for the HAProxy. That is accessible from inside the Vagrant VMs, however not from the host machine.
Utilizing firewall strategies similar to ufw solely made issues worse; I used to be locked out of the VM. I do know now that I needed to allow SSH entry first, however that is behind me now.
Having the port-forwarding configured, I used to be capable of name the localhost:6443 from my machine and immediately get entry to the HAProxy.
On Vagrant Networking
Normally, I’ve discovered many networking points whereas engaged on this problem. For some purpose, the obtain velocity contained in the VMs was horrible (I’m not the one complainer right here should you search by means of the online). That is the primary driver for mounting the identical obtain listing for all of the VMs to cease re-downloading each time the Ansible playbook runs.
CPU and Reminiscence Allocation
Whereas not strictly required, I discovered profit restraining the CPU and reminiscence utilization on the VMs. This ensures that no further sources is getting used.
Frankly talking they should not even transcend this. That is an emtpy cluster with simply the control-plane elements and no heavy workload is operating on it.
Mounting the Obtain Listing
The obtain listing is mounted to all of the VMs to keep away from re-downloading the binaries from the web each time the playbook is operating, both as a result of a machine restart, or just to start out from scratch.
The trick, nonetheless, is that in Ansible get_url, as you will see shortly, you’ll have to specify absolutely the path to the vacation spot file to profit from this optimization and solely specifying a listing will re-download the file.
Ansible Provisioner
node.vm.provision "ansible" do |ansible|
ansible.verbose = "vv"
ansible.playbook = "bootstrap.yml"
ansible.compatibility_mode = "2.0"
finish
if machine_id == N
node.vm.provision :ansible do |ansible|
ansible.restrict = "all"
ansible.verbose = "vv"
ansible.playbook = "bootstrap.yml"
finish
finish
The final and most vital a part of the Vagrantfile is the Ansible provisioner part which, as you may see, is for each the Load Balancer VM in addition to all of the three nodes of the Kubernetes cluster.
The distinction, nonetheless, is that for the Kubernetes nodes, we wish the playbook to run for all of them on the similar time to profit from parallel execution of Ansible playbook. The choice could be to spin up the nodes one after the other and run the playbook on every of them, which isn’t environment friendly and consumes extra time.
Ansible Playbook
After provisioning the VMs, it is time to try what the Ansible playbook does to configure the nodes and the Load Balancer.
The primary configuration of all the Kubernetes cluster is completed through this playbook and as such, you may count on a hefty quantity of configurations to be carried out right here.
First, let’s check out the playbook itself to get a sense of what to anticipate.
- title: Configure the Load Balancer
hosts: lb
turn into: true
gather_facts: true
vars_files:
- vars/apt.yml
- vars/k8s.yml
- vars/lb.yml
- vars/etcd.yml
roles:
- stipulations
- haproxy
- etcd-gateway
- title: Bootstrap the Kubernetes Cluster
hosts:
- node0
- node1
- node2
turn into: true
gather_facts: true
vars_files:
- vars/tls.yml
- vars/apt.yml
- vars/lb.yml
- vars/k8s.yml
- vars/etcd.yml
atmosphere:
KUBECONFIG: /var/lib/kubernetes/admin.kubeconfig
roles:
- stipulations
- function: tls-ca
run_once: true
- tls
- kubeconfig
- encryption
- etcd
- k8s
- employee
- function: coredns
run_once: true
- function: cilium
run_once: true
For those who discover there are two performs operating on this playbook, one for the Load Balancer and the opposite for the Kubernetes nodes. This distinction is vital as a result of not all of the configurations would be the similar for all of the VMs. That is the logic behind having completely different hosts in every.
One other vital spotlight is that the Load Balancer is being configured first, solely as a result of that is the entrypoint for our Kubernetes API server and we want that to be prepared earlier than the upstream servers.
Listing Format
This playbook you see above is on the root of our listing construction, proper subsequent to all of the roles you see included within the roles part.
To get a greater understanding, here is what the listing construction appears like:
.
├── ansible.cfg
├── bootstrap.yml
├── cilium/
├── coredns/
├── encryption/
├── etcd/
├── etcd-gateway/
├── haproxy/
├── k8s/
├── kubeconfig/
├── prerequisites/
├── tls/
├── tls-ca/
├── Vagrantfile
├── vars/
└── worker/
Beside the playbook itself, the Ansible playbook and the Vagrantfile the other pieces are roles, initialized with ansible-galaxy init <role-name> command and modified as per the specification in the originial challenge.
We will take a closer look at each role shortly.
Ansible Configuration
Before jumping into the roles, one last impotant piece of information is the ansible.cfg file, which holds the modifications we make to the Ansible default behavior.
The content is as below.
[defaults]
stock=.vagrant/provisioners/ansible/stock/
turn into=false
log_path=/tmp/ansible.log
gather_facts=false
host_key_checking=false
gathering = good
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 86400
[inventory]
enable_plugins = 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml'
cache = sure
cache_connection = /tmp/ansible_inventory
The significance of details caching is to get performant execution of the playbooks throughout subsequent runs.
Let’s Start the Actual Work
To this point, we have solely been making ready every little thing. The rest of this put up will give attention to the true problem itself, creating the items and elements that make up the Kubernetes cluster, one after the other.
For the sake of brevity, and since I do not need this weblog put up be lengthy, or worse, damaged into a number of components, I’ll solely spotlight crucial duties, take away duplicates from the dialogue, and customarily undergo the core side of the duty at hand. You might be greater than welcome to go to the supply code for your self and dig deeper.
Step 0: Conditions
In right here we are going to allow port-forwarding, create the mandatory directories that can be utilized by later steps, and optionally, add the DNS information to each node’s /and so on/hosts file.
The vital traces are highlighted within the snippet beneath.
---
- title: Allow IP Forwarding completely
ansible.builtin.copy:
content material: web.ipv4.ip_forward = 1
dest: /and so on/sysctl.d/20-ipforward.conf
mode: "0644"
notify: Reload sysctl
- title: Guarantee python3-pip package deal is put in
ansible.builtin.apt:
title: python3-pip
state: current
update_cache: true
cache_valid_time: "{{ cache_valid_time }}"
- title: Add present person to adm group
ansible.builtin.person:
title: "{{ ansible_user }}"
teams: adm
append: true
- title: Guarantee CA listing exists
ansible.builtin.file:
path: /and so on/kubernetes/pki
state: listing
proprietor: root
group: root
mode: "0700"
- title: Create private-network DNS information
ansible.builtin.lineinfile:
path: /and so on/hosts
line: "{{ merchandise }}"
state: current
with_items:
- 192.168.56.2 node0 etcd-node0
- 192.168.56.3 node1 etcd-node1
- 192.168.56.4 node2 etcd-node2
tags:
- dns
As you may see, after the sysctl modification, we’re notifying our handler for a reload and re-read of the sysctl configurations. The handler definition is as beneath.
---
- title: Reload sysctl
ansible.builtin.shell: sysctl --system
changed_when: false
Ansible Linting
If you’re working with Ansible, I extremely advocate utilizing ansible-lint as it’s going to enable you refine your playbooks a lot quicker in the course of the growth section of your undertaking. It is not simply “good” and “linting” that issues. A number of the suggestions are actually vital from different features, similar to safety and efficiency.
Step 1: TLS CA certificates
For all of the workloads we can be deploying, we are going to want a CA signing TLS certificates for us. For those who’re not a TLS professional, simply know two major issues:
- TLS enforces encrypted and safe communication between the events (shopper and server on this case however can alse be friends). You’ll be able to attempt it out for your self and sniff among the knowledge utilizing Wireshark to see that not one of the knowledge is readable. They’re solely decipherable by the events concerned within the communication.
- A minimum of within the case of Kubernetes, TLS certificates are used for authentication and authorization of the completely different elements and customers. This may, in impact, imply that if a TLS certificates was signed by the trusted CA of the cluster, and the topic of that TLS has elevated privileges, then that topic can ship similar to the API server and no additional authentication is required.
The TLS key and certificates generations are a ache within the butt IMO. However, with the ability of Ansible, we take numerous the ache away, as you may see within the snippet beneath.
---
- title: Generate an empty temp file for CSR
ansible.builtin.file:
path: /tmp/ca.csr
state: contact
proprietor: root
group: root
mode: "0400"
register: temp_file
- title: Generate CA non-public key
group.crypto.openssl_privatekey:
path: /vagrant/share/ca.key
kind: "{{ k8s_privatekey_type }}"
state: current
- title: Generate CA CSR to offer ALT names and different choices
group.crypto.openssl_csr:
basicConstraints_critical: true
basic_constraints:
- CA:TRUE
common_name: kubernetes-ca
keyUsage_critical: true
key_usage:
- keyCertSign
- cRLSign
path: "{{ temp_file.dest }}"
privatekey_path: /vagrant/share/ca.key
state: current
- title: Generate CA certificates
group.crypto.x509_certificate:
path: /vagrant/share/ca.crt
privatekey_path: /vagrant/share/ca.key
csr_path: "{{ temp_file.dest }}"
supplier: selfsigned
state: current
- title: Copy cert to kubernetes PKI dir
ansible.builtin.copy:
src: "{{ merchandise }}"
dest: /and so on/kubernetes/pki/
remote_src: true
proprietor: root
group: root
mode: "0400"
loop:
- /vagrant/share/ca.crt
- /vagrant/share/ca.key
I am not a TLS professional, however from my understanding, crucial a part of the CSR creation is the CA: TRUE flag. I really do not even know if any of the constraints or usages are wanted, used and revered by any software!
Additionally, the supplier: selfsigned, as self-explanatory as it’s, is used to instruct that we’re creating a brand new root CA certificates and never a subordinate one.
Lastly, we copy each the CA key and its certificates to a shared listing that can be utilized by all the opposite elements when producing their very own certificates.
Etcd CA
We may use the identical CA for etcd communications as nicely, however I made a decision to separate them out to verify no part aside from the API server and the friends of etcd can be allowed to ship any requests to the etcd server.
In the identical Ansible function, we additionally generate a key and certificates for the admin/ operator of the cluster. On this case, that’ll be me, the one who’s provisioning and configuring the cluster.
The thought is that we are going to not use the TLS certificates of different elements to speak to the API server, however slightly use those explicitly created for this function.
Here is what it’s going to appear to be:
- title: Generate an empty temp file for CSR
ansible.builtin.file:
path: /tmp/admin.csr
state: contact
proprietor: root
group: root
mode: "0400"
register: temp_file
- title: Generate Admin Operator non-public key
group.crypto.openssl_privatekey:
path: /vagrant/share/admin.key
kind: "{{ k8s_privatekey_type }}"
- title: Generate Admin Operator CSR
group.crypto.openssl_csr:
path: "{{ temp_file.dest }}"
privatekey_path: /vagrant/share/admin.key
common_name: 'admin'
topic:
O: 'system:masters'
OU: 'Kubernetes The Onerous Means'
- title: Create Admin Operator TLS certificates utilizing CA key and cert
group.crypto.x509_certificate:
path: /vagrant/share/admin.crt
csr_path: "{{ temp_file.dest }}"
privatekey_path: /vagrant/share/admin.key
ownca_path: /vagrant/share/ca.crt
ownca_privatekey_path: /vagrant/share/ca.key
supplier: ownca
ownca_not_after: +365d
The topic you see on line 19, is the group system:masters. This group contained in the Kubernetes cluster has the very best privileges. It will not require any RBAC to carry out the requests, as all can be granted by default.
As for the certificates creation, you see on line 26-28 that we specify to the Ansible process that the CA can be of kind selfsigned and we’re passing the identical key and certificates we created within the final step.
TLS CA Execution
To wrap this step up, two vital issues price mentioning are:
- Each of the snippets talked about right here and those not talked about, can be imported into the basis of the Ansible function with the next
major.yml.tls-ca/duties/major.yml- title: CA ansible.builtin.import_tasks: file: ca.yml - title: Etcd CA ansible.builtin.import_tasks: file: etcd-ca.yml - title: Etcd Admin ansible.builtin.import_tasks: file: etcd-admin.yml - title: Create admin operator TLS certificates ansible.builtin.import_tasks: file: admin.yml - You may need seen within the
bootstrap.ymlroot playbook that this CA function will solely run as soon as, the primary Ansible stock that will get up to now. This may guarantee we do not eat further CPU energy or overwrite the at present current CA key and certificates. A few of our roles are designed this fashion, e.g., the set up ofciliumis one other a kind of circumstances.
Step 2: TLS Certificates for Kubernetes Elements
The variety of certificates we have to generate for the Kubernetes elements is eight in whole, however we’ll convey just one on this dialogue. Probably the most impotant one, the API server certificates.
All of the others are comparable with a potential minor tweak.
Let’s first check out what the Ansible function will appear to be:
- title: Generate API Server non-public key
group.crypto.openssl_privatekey:
path: /and so on/kubernetes/pki/kube-apiserver.key
kind: "{{ k8s_privatekey_type }}"
- title: Generate API Server CSR
group.crypto.openssl_csr:
basicConstraints_critical: true
basic_constraints:
- CA:FALSE
common_name: kube-apiserver
extKeyUsage_critical: false
extended_key_usage:
- clientAuth
- serverAuth
keyUsage:
- keyEncipherment
- dataEncipherment
keyUsage_critical: true
path: /and so on/kubernetes/pki/kube-apiserver.csr
privatekey_path: /and so on/kubernetes/pki/kube-apiserver.key
topic:
O: system:kubernetes
OU: Kubernetes The Onerous Means
subject_alt_name: "{ from_yaml }"
- title: Create API Server TLS certificates utilizing CA key and cert
group.crypto.x509_certificate:
path: /and so on/kubernetes/pki/kube-apiserver.crt
csr_path: /and so on/kubernetes/pki/kube-apiserver.csr
privatekey_path: /and so on/kubernetes/pki/kube-apiserver.key
ownca_path: /vagrant/share/ca.crt
ownca_privatekey_path: /vagrant/share/ca.key
ownca_not_after: +365d
supplier: ownca
From the highlights within the snippet above, you may see at the very least 3 piece of vital info:
- This certificates will not be for the CA:
CA: FALSE. - The topic is in
system:kubernetesgroup. That is simply an identifier actually and serves to particular function. - The identical properties as with the
admin.ymlwas used to generate the TLS certificates. Particularly thesupplierand all of theownca_*properties.
Step 3: KubeConfig Recordsdata
On this step, for each part that can speak to the API server, we are going to create a KubeConfig file, specifying the server deal with, the CA certificates, and the important thing and certificates of the shopper.
The format of the KubeConfig is identical as you’ve in your filesystem beneath ~/.kube/config. That, for the aim of our cluster, can be a Jinja2 template that can take the variables we simply talked about.
Here is what that Jinja2 template will appear to be:
apiVersion: v1
clusters:
- cluster:
certificate-authority: {{ ca_cert_path }}
server: https://{{ kube_apiserver_address }}:{{ kube_apiserver_port }}
title: {{ kube_context }}
contexts:
- context:
cluster: {{ kube_context }}
person: {{ kube_context }}
title: {{ kube_context }}
current-context: {{ kube_context }}
sort: Config
preferences: {}
customers:
- title: {{ kube_context }}
person:
client-certificate: {{ client_cert_path }}
client-key: {{ client_key_path }}
And with that template, we are able to generate a number of KubeConfigs for every part. This is likely one of the examples to create one for the Kubelet part.
---
- title: Generate KubeConfig for kubelet
ansible.builtin.template:
src: kubeconfig.yml.j2
dest: /var/lib/kubernetes/kubelet.kubeconfig
mode: "0640"
proprietor: root
group: root
vars:
kube_apiserver_address: localhost
kube_apiserver_port: 6443
client_cert_path: /and so on/kubernetes/pki/kubelet.crt
client_key_path: /and so on/kubernetes/pki/kubelet.key
Kubernetes API Server
The present setup is to deploy three API server, one on every of the three VM nodes. Meaning in any node, we may have localhost entry to the management airplane, if and provided that the important thing and certificates are handed accurately.
As you discover, the Kubelet is speaking to the localhost:6443. A greater various is to speak to the Load Balancer in case one of many API servers goes down. However, that is an academic setup and never a manufacturing one!
The values that aren’t immediately handed with vars property, are being handed by the defaults variables:
---
ca_cert_path: /and so on/kubernetes/pki/ca.crt
kube_context: kubernetes-the-hard-way
kube_apiserver_address: "{{ load_balancer_ip }}"
kube_apiserver_port: "{{ load_balancer_port }}"
They can be handed from mother and father of the function, or the recordsdata being handed to the playbook.
Ansible Variables
As you’ll have seen, inside each vars file, we are able to use the values from different variables. That is one of many many issues that make Ansible so highly effective!
Step 4: Encryption Configuration
The target of this step is to create an encryption key that can be used to encrypt and decrypt the Kubernetes Secrets and techniques saved within the etcd database.
For this process, we use one template, and a set of Ansible duties.
sort: EncryptionConfig
apiVersion: v1
sources:
- sources:
- secrets and techniques
suppliers:
- aescbc:
keys:
- title: {{ key_name }}
secret: {{ key_secret }}
- id: {}
---
- title: Learn the contents of /vagrant/share/encryption-secret
ansible.builtin.slurp:
src: '/vagrant/share/encryption-secret'
register: encryption_secret_file
failed_when: false
- title: Generate random string
ansible.builtin.set_fact:
key_secret: "{{ lookup('ansible.builtin.password', '/dev/null size=32 chars=ascii_letters,digits,special_characters') }}"
no_log: true
when: key_secret will not be outlined
- title: Guarantee key_secret is populated
when: encryption_secret_file.content material will not be outlined
block:
- title: Write secret to file
ansible.builtin.copy:
content material: '{{ key_secret }}'
dest: '/vagrant/share/encryption-secret'
mode: '0400'
- title: Learn current key_secret
ansible.builtin.set_fact:
key_secret: "{{ encryption_secret_file.content material }}"
no_log: true
when: encryption_secret_file.content material is outlined
- title: Create encryption config
ansible.builtin.template:
src: config.yml.j2
dest: /and so on/kubernetes/encryption-config.yml
mode: '0400'
proprietor: root
group: root
no_log: true
As you see within the process definition, we are going to solely generate one secret for all the following runs, and reuse the file that maintain that password.
That encryption configuration will later be handed to the cluster for storing encrypted Secret sources.
Step 5: Etcd Cluster
On this step we are going to obtain the compiled etcd binary, create the configuration, create the systemd service, difficulty the certificates for the etcd friends in addition to one for the API server speaking to the etcd cluster as a shopper.
The set up will like beneath.
---
- title: Obtain etcd launch tarball
ansible.builtin.get_url:
url: "{{ etcd_download_url }}"
dest: "{{ downloads_dir }}/{ basename }"
mode: "0644"
proprietor: root
group: root
checksum: sha256:{{ etcd_checksum }}
tags:
- obtain
register: etcd_download
- title: Guarantee gzip is put in
ansible.builtin.package deal:
title: gzip
state: current
- title: Extract etcd from the tarball to /usr/native/bin/
ansible.builtin.unarchive:
src: "{{ etcd_download.dest }}"
dest: /usr/native/bin/
remote_src: true
mode: "0755"
extra_opts:
- --strip-components=1
- --wildcards
- "**/etcd"
- "**/etcdctl"
- "**/etcdutl"
notify: Reload etcd systemd
Just a few vital be aware to say for this playbook:
- We specify the
destas an absolute path to theget_urlprocess to keep away from re-downloading the file for subsequent runs. - The
checksumensures that we do not get any nasty binary from the web. - The
registerfor the obtain step will permit us to make use of theetcd_download.destwhen later attempting to extract the tarball. - Contained in the tarball might or will not be multiple file. We’re solely curious about extracting those we specify within the
extra_optsproperty. Be conscious of the--strip-componentsand the--wildcardschoices.
The variables for the above process will appear to be beneath:
---
etcd_initial_cluster: etcd-node0=https://192.168.56.2:2380,etcd-node1=https://192.168.56.3:2380,etcd-node2=https://192.168.56.4:2380
etcd_advertise_ip: "0.0.0.0"
etcd_privatekey_type: Ed25519
etcd_version: v3.5.12
etcd_checksum: f2ff0cb43ce119f55a85012255609b61c64263baea83aa7c8e6846c0938adca5
etcd_download_url: https://github.com/etcd-io/etcd/releases/obtain/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-amd64.tar.gz
k8s_etcd_servers: https://192.168.56.2:2379,https://192.168.56.3:2379,https://192.168.56.4:2379
As soon as the set up is completed, we are able to proceed with the configuration as beneath:
- title: Make sure the etcd directories exist
ansible.builtin.file:
path: '{{ merchandise }}'
state: listing
proprietor: 'root'
group: 'root'
mode: '0750'
loop:
- /and so on/etcd
- title: Copy CA TLS certificates to /and so on/kubernetes/pki/etcd/
ansible.builtin.copy:
src: /vagrant/share/etcd-ca.crt
dest: /and so on/kubernetes/pki/etcd/ca.crt
mode: '0640'
remote_src: true
notify: Reload etcd systemd
- title: Create systemd service
ansible.builtin.template:
src: systemd.service.j2
dest: '/and so on/systemd/system/etcd.service'
mode: '0644'
proprietor: root
group: root
tags:
- systemd
notify: Reload etcd systemd
- title: Begin etcd service
ansible.builtin.systemd_service:
title: etcd.service
state: began
enabled: true
daemon_reload: true
The handler for restarting the etcd will not be a lot completely different from what we have seen beforehand. However the systemd Jinja2 template is an attention-grabbing one:
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd
[Service]
Sort=notify
ExecStart=/usr/native/bin/etcd
--advertise-client-urls=https://{{ host_ip }}:2379
--cert-file=/and so on/kubernetes/pki/etcd/server.crt
--client-cert-auth
--data-dir=/var/lib/etcd
--initial-advertise-peer-urls=https://{{ host_ip }}:2380
--initial-cluster-state={{ etcd_initial_cluster_state }}
--initial-cluster-token={{ etcd_initial_cluster_token }}
--initial-cluster={{ etcd_initial_cluster }}
--key-file=/and so on/kubernetes/pki/etcd/server.key
--listen-client-urls=https://{{ bind_address }}:2379
--listen-peer-urls=https://{{ bind_address }}:2380
--log-level data
--log-outputs stderr
--logger zap
--name={{ etcd_peer_name }}
--peer-cert-file=/and so on/kubernetes/pki/etcd/server.crt
--peer-client-cert-auth
--peer-key-file=/and so on/kubernetes/pki/etcd/server.key
--peer-trusted-ca-file=/and so on/kubernetes/pki/etcd/ca.crt
--trusted-ca-file=/and so on/kubernetes/pki/etcd/ca.crt
LimitNOFILE=40000
Restart=at all times
RestartSec=5
TimeoutStartSec=0
[Install]
WantedBy=multi-user.goal
The 2 variables you see within the above template are handed from the next vars file:
---
host_ip: "{ first }"
k8s_version: v1.29.2
apiserver_port: 6443
apiserver_ips:
- 192.168.56.2
- 192.168.56.3
- 192.168.56.4
cilium_version: 1.15.1
k8s_static_pods_dir: /and so on/kubernetes/manifests
bind_address: "0.0.0.0"
You’ll notice that the etcd is instructed to confirm the authentication of its requests utilizing the TLS CA. No request shall be allowed until its TLS is signed by the verified and trusted CA.
That is achieved by the --client-cert-auth and --trusted-ca-file choices for purchasers of the etcd cluster, and the --peer-client-cert-auth and --peer-trusted-ca-file for the friends of the etcd cluster.
Additionally, you will discover that this can be a 3-node etcd cluster, and the friends are statically configured by the values given within the vars/etcd.yml file. That is precisely one of many circumstances the place having static IP addresses make numerous our assumptions simpler and the configurations less complicated. One can solely think about what could be required for dynamic environments the place DHCP is concerned.
Step 6: Kubernetes Elements
There are a number of elements, as you recognize, however here is a pattern, being the Kubernetes API server.
---
- title: Obtain the Kubernetes binaries
ansible.builtin.get_url:
url: "https://dl.k8s.io/{{ k8s_version }}/kubernetes-server-linux-amd64.tar.gz"
dest: "{{ downloads_dir }}/kubernetes-server-{{ k8s_version }}-linux-amd64.tar.gz"
mode: "0444"
proprietor: root
group: root
tags:
- obtain
register: k8s_download
- title: Extract binaries to system path
ansible.builtin.unarchive:
src: "{{ k8s_download.dest }}"
dest: /usr/native/bin/
remote_src: true
proprietor: root
group: root
mode: "0755"
extra_opts:
- --strip-components=3
- kubernetes/server/bin/kube-apiserver
- kubernetes/server/bin/kube-controller-manager
- kubernetes/server/bin/kube-scheduler
- kubernetes/server/bin/kubectl
- kubernetes/server/bin/kubelet
- kubernetes/server/bin/kube-proxy
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/native/bin/kube-apiserver
--allow-privileged=true
--audit-log-maxage=30
--audit-log-maxbackup=3
--audit-log-maxsize=100
--audit-log-path=/var/log/audit.log
--authorization-mode=Node,RBAC
--bind-address={{ bind_address }}
--external-hostname={{ external_hostname }}
--client-ca-file=/and so on/kubernetes/pki/ca.crt
--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
--etcd-cafile=/and so on/kubernetes/pki/etcd/ca.crt
--etcd-certfile=/and so on/kubernetes/pki/etcd/kube-apiserver.crt
--etcd-keyfile=/and so on/kubernetes/pki/etcd/kube-apiserver.key
--etcd-servers={{ k8s_etcd_servers }}
--event-ttl=1h
--encryption-provider-config=/and so on/kubernetes/encryption-config.yml
--kubelet-certificate-authority=/and so on/kubernetes/pki/ca.crt
--kubelet-client-certificate=/and so on/kubernetes/pki/kube-apiserver.crt
--kubelet-client-key=/and so on/kubernetes/pki/kube-apiserver.key
--runtime-config='api/all=true'
--service-account-key-file=/and so on/kubernetes/pki/serviceaccount.key
--service-account-signing-key-file=/and so on/kubernetes/pki/serviceaccount.key
--service-account-issuer=https://{{ kubernetes_public_ip }}:6443
--service-cluster-ip-range=10.0.0.0/16,fd00:10:96::/112
--service-node-port-range=30000-32767
--tls-cert-file=/and so on/kubernetes/pki/kube-apiserver.crt
--tls-private-key-file=/and so on/kubernetes/pki/kube-apiserver.key
--proxy-client-cert-file=/and so on/kubernetes/pki/kube-apiserver.crt
--proxy-client-key-file=/and so on/kubernetes/pki/kube-apiserver.key
--peer-advertise-ip={{ bind_address }}
--peer-ca-file=/and so on/kubernetes/pki/ca.crt
--feature-gates=UnknownVersionInteroperabilityProxy=true,StorageVersionAPI=true
--v=4
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.goal
And the default vaules are fetched from its personal function defaults:
---
k8s_version: "{{ lookup('url', 'https://dl.k8s.io/launch/secure.txt').cut up('n')[0] }}"
kubernetes_public_ip: "{{ host_ip }}"
cluster_cidr: 10.200.0.0/16
service_cluster_ip_range: 10.32.0.0/24
external_hostname: 192.168.56.100
You’ll discover the next factors on this systemd setup:
- The exterior host deal with is of the Load Balancer.
- The certificates recordsdata for each Kubernetes API server and the Etcd server are being handed from the placement beforehand generated.
- The encryption config is being fetched from a file generated in step 4.
What makes this setup HA then?
To begin with, there are a few locations that aren’t pointing to the Load Balancer IP deal with so this would not rely as as an entire HA setup. However, having an LB in-place already qualifies this setup as such.
However, you’ll not see any peer deal with configuration within the API server because it was the case for etcd with --initial-cluster flag and also you would possibly marvel how do the completely different cases of the API server know of each other and the way can they coordinate between one another when a number of requests hit the API server?
The reply to this query doesn’t lie within the Kubernetes itself, however within the storage layer, being the etcd cluster. The etcd cluster, on the time of writing, makes use of the Raft protocol for consensus and coordination between the friends.
And that’s what makes the Kubernetes cluster HA, not the API server itself. Every occasion will speak to the etcd cluster to know the state of the cluster and the elements inside.
Step 7: Employee Nodes
This is likely one of the final steps earlier than we now have a non-Prepared Kubernetes cluster.
The duty consists of downloading among the binaries, passing in among the TLS certificates generated earlier, and beginning the systemd companies.
- title: Guarantee CNI listing exists
ansible.builtin.file:
path: /and so on/cni/web.d/
state: listing
proprietor: root
group: root
mode: "0755"
tags:
- by no means
- title: Configure CNI Networking
ansible.builtin.copy:
content material: |
{
"cniVersion": "1.0.0",
"title": "containerd-net",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"promiscMode": true,
"ipam": {
"type": "host-local",
"ranges": [
[{
"subnet": "{{ pod_subnet_cidr_v4 }}"
}]
],
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
},
{
"kind": "portmap",
"capabilities": {"portMappings": true}
}
]
}
dest: /and so on/cni/web.d/10-containerd-net.conf
proprietor: root
group: root
mode: "0640"
tags:
- by no means
- title: Guarantee containerd listing exists
ansible.builtin.file:
path: /and so on/containerd
state: listing
proprietor: root
group: root
mode: "0755"
- title: Get containerd default config
ansible.builtin.command: containerd config default
changed_when: false
register: containerd_default_config
tags:
- config
- title: Configure containerd
ansible.builtin.copy:
content material: "{{ containerd_default_config.stdout }}"
dest: /and so on/containerd/config.toml
proprietor: root
group: root
mode: "0640"
tags:
- config
notify: Restart containerd
The CNI config for containerd is documented of their repository should you really feel curious.
Employee Position Default Variables
---
cluster_cidr: 10.200.0.0/16
cluster_dns: 10.32.0.10
cluster_domain: cluster.native
cni_plugins_checksum_url: https://github.com/containernetworking/plugins/releases/obtain/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz.sha256
cni_plugins_url: https://github.com/containernetworking/plugins/releases/obtain/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
containerd_checksum_url: https://github.com/containerd/containerd/releases/obtain/v1.7.13/containerd-1.7.13-linux-amd64.tar.gz.sha256sum
containerd_service_url: https://github.com/containerd/containerd/uncooked/v1.7.13/containerd.service
containerd_url: https://github.com/containerd/containerd/releases/obtain/v1.7.13/containerd-1.7.13-linux-amd64.tar.gz
crictl_checksum_url: https://github.com/kubernetes-sigs/cri-tools/releases/obtain/v1.29.0/crictl-v1.29.0-linux-amd64.tar.gz.sha256
crictl_url: https://github.com/kubernetes-sigs/cri-tools/releases/obtain/v1.29.0/crictl-v1.29.0-linux-amd64.tar.gz
kubectl_checksum_url: https://dl.k8s.io/launch/{{ k8s_version }}/bin/linux/amd64/kubectl.sha256
kubectl_url: https://dl.k8s.io/launch/{{ k8s_version }}/bin/linux/amd64/kubectl
kubelet_checksum_url: https://dl.k8s.io/{{ k8s_version }}/bin/linux/amd64/kubelet.sha256
kubelet_config_path: /var/lib/kubelet/config.yml
kubelet_url: https://dl.k8s.io/launch/{{ k8s_version }}/bin/linux/amd64/kubelet
pod_subnet_cidr_v4: 10.88.0.0/16
pod_subnet_cidr_v6: 2001:4860:4860::/64
runc_url: https://github.com/opencontainers/runc/releases/obtain/v1.1.12/runc.amd64
runc_checksum: aadeef400b8f05645768c1476d1023f7875b78f52c7ff1967a6dbce236b8cbd8
- title: Obtain containerd
ansible.builtin.get_url:
url: "{{ containerd_url }}"
dest: "{{ downloads_dir }}/{ basename }"
checksum: "sha256:{ cut up }"
mode: "0444"
register: containerd_download
tags:
- obtain
- title: Create /tmp/containerd listing
ansible.builtin.file:
path: /tmp/containerd
state: listing
mode: "0755"
- title: Extract containerd
ansible.builtin.unarchive:
src: "{{ containerd_download.dest }}"
dest: /tmp/containerd
mode: "0755"
remote_src: true
- title: Glob recordsdata in unarchived bin listing
ansible.builtin.discover:
paths: /tmp/containerd
file_type: file
recurse: true
mode: "0755"
register: containerd_bin_files
- title: Set up containerd
ansible.builtin.copy:
src: "{{ merchandise }}"
dest: /usr/native/bin/
proprietor: root
group: root
mode: "0755"
remote_src: true
loop: "{ map(attribute='path') }"
- title: Obtain containerd service
ansible.builtin.get_url:
url: "{{ containerd_service_url }}"
dest: /and so on/systemd/system/{ basename }
mode: "0644"
proprietor: root
group: root
tags:
- obtain
Step 8: CoreDNS & Cilium
The final step is simple.
We plan to run the CoreDNS as Kubernetes Deployment with affinity, and set up the Cilium utilizing its CLI.
---
- title: Take away CoreDNS as static pod
ansible.builtin.file:
path: "{{ k8s_static_pods_dir }}/coredns.yml"
state: absent
- title: Slurp CoreDNS TLS certificates
ansible.builtin.slurp:
src: /and so on/kubernetes/pki/coredns.crt
register: coredns_cert
- title: Slurp CoreDNS TLS key
ansible.builtin.slurp:
src: /and so on/kubernetes/pki/coredns.key
register: coredns_key
- title: Slurp CoreDNS CA certificates
ansible.builtin.slurp:
src: /and so on/kubernetes/pki/ca.crt
register: coredns_ca
- title: Apply CoreDNS manifest
kubernetes.core.k8s:
definition: "{ from_yaml }"
state: current
Discover the slurp duties as a result of they are going to be used the go the TLS certificates to the CoreDNS occasion.
CoreDNS Kubernetes Manifests
apiVersion: v1
sort: Checklist
gadgets:
- apiVersion: v1
knowledge:
Corefile: |
.:53 {
errors
well being {
lameduck 5s
}
prepared
kubernetes cluster.native in-addr.arpa ip6.arpa {
endpoint https://{{ host_ip }}:{{ apiserver_port }}
tls /cert/coredns.crt /cert/coredns.key /cert/ca.crt
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
ahead . /and so on/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
sort: ConfigMap
metadata:
title: coredns-config
namespace: kube-system
- apiVersion: v1
sort: Secret
metadata:
title: coredns-tls
namespace: kube-system
knowledge:
tls.crt: "{{ coredns_cert.content material }}"
tls.key: "{{ coredns_key.content material }}"
ca.crt: "{{ coredns_ca.content material }}"
- apiVersion: v1
sort: ServiceAccount
metadata:
title: coredns
namespace: kube-system
- apiVersion: rbac.authorization.k8s.io/v1
sort: ClusterRole
metadata:
title: coredns
guidelines:
- apiGroups:
- ""
sources:
- endpoints
- companies
- pods
- namespaces
verbs:
- listing
- watch
- apiGroups:
- discovery.k8s.io
sources:
- endpointslices
verbs:
- listing
- watch
- apiVersion: rbac.authorization.k8s.io/v1
sort: ClusterRoleBinding
metadata:
title: coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
sort: ClusterRole
title: coredns
topics:
- sort: ServiceAccount
title: coredns
namespace: kube-system
- apiVersion: v1
sort: Service
metadata:
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
title: kube-dns
namespace: kube-system
spec:
ports:
- title: dns
port: 53
protocol: UDP
targetPort: 53
- title: dns-tcp
port: 53
protocol: TCP
targetPort: 53
- title: metrics
port: 9153
protocol: TCP
targetPort: 9153
selector:
k8s-app: coredns
kind: ClusterIP
- apiVersion: apps/v1
sort: Deployment
metadata:
title: coredns
namespace: kube-system
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: coredns
technique:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
kind: RollingUpdate
template:
metadata:
labels:
k8s-app: coredns
spec:
containers:
- args:
- -conf
- /and so on/coredns/Corefile
picture: coredns/coredns:1.11.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 5
httpGet:
path: /well being
port: 8080
scheme: HTTP
initialDelaySeconds: 60
successThreshold: 1
timeoutSeconds: 5
title: coredns
ports:
- containerPort: 53
title: dns
protocol: UDP
- containerPort: 53
title: dns-tcp
protocol: TCP
- containerPort: 9153
title: metrics
protocol: TCP
readinessProbe:
httpGet:
path: /prepared
port: 8181
scheme: HTTP
sources:
limits:
reminiscence: 170Mi
requests:
cpu: 100m
reminiscence: 70Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
volumeMounts:
- mountPath: /and so on/coredns
title: config-volume
readOnly: true
- mountPath: /cert
title: coredns-tls
readOnly: true
dnsPolicy: Default
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- impact: NoSchedule
key: node-role.kubernetes.io/control-plane
volumes:
- configMap:
gadgets:
- key: Corefile
path: Corefile
title: coredns-config
title: config-volume
- secret:
defaultMode: 0444
gadgets:
- key: tls.crt
path: coredns.crt
- key: tls.key
path: coredns.key
- key: ca.crt
path: ca.crt
secretName: coredns-tls
title: coredns-tls
And at last the Cilium.
- title: Obtain cilium-cli
ansible.builtin.get_url:
url: "{{ cilium_cli_url }}"
dest: "{{ downloads_dir }}/{ basename }"
proprietor: root
group: root
mode: "0644"
checksum: "sha256:{{ cilium_cli_checksum }}"
register: cilium_cli_download
- title: Extract cilium bin to /usr/native/bin
ansible.builtin.unarchive:
src: "{{ cilium_cli_download.dest }}"
dest: /usr/native/bin/
remote_src: true
proprietor: root
group: root
mode: "0755"
extra_opts:
- cilium
- title: Set up cilium
ansible.builtin.command: cilium set up
failed_when: false
Cilium Position Default Variables
That is it. Consider it or not, the Kubernetes cluster is now prepared and should you run the next command, you will notice three nodes within the Prepared state.
Find out how to run it?
For those who clone the repository, you’ll solely want vagrant up to construct every little thing from scratch. It would take a while for all of the elements to be up and prepared, however it’s going to set issues up with none additional guide intervention.
Conclusion
This process took me numerous time to get proper. I needed to undergo numerous iterations to make it work. One of the crucial time-consuming components was how the etcd cluster was misbehaving, resulting in the Kubernetes API server hitting timeout errors and being inaccessible for the remainder of the cluster’s elements.
I realized loads from this problem. I realized the way to write environment friendly Ansible playbooks, the way to create the proper psychological mannequin for the goal host the place the Ansible executes a command, the way to take care of all these TLS certificates, and total, the way to arrange a Kubernetes cluster from scratch.
I could not be happier reaching the ultimate end result, having spent numerous hours debugging and banging my head towards the wall.
I like to recommend everybody giving the problem a attempt. You by no means understand how a lot you do not know in regards to the inside workings of Kubernetes till you attempt to set it up from scratch.
Thanks for studying to this point. I hope you loved the journey as a lot as I did 
.
Supply Code
As talked about earlier than, you could find the supply code for this problem on the GitHub repository.
FAQ
Why Cilium?
Cilium has emerged as a cloud-native CNI software that occurs to have numerous the options and traits of a production-grade CNI. To call a couple of, efficiency, safety, and observability are the highest ones. I’ve used Linkerd prior to now however I’m utilizing Cilium for any of the present and upcoming initiatives I’m engaged on. It would proceed to show itself as a terrific CNI for Kubernetes clusters.
Why use Vagrant?
I am low cost 
and I do not need to pay for cloud sources, even for studying functions. I’ve lively subscription on O’Reilly and A Cloud Guru and I’d’ve gone for his or her sandboxes, however I initially began this problem simply with Vagrant and I resisted the urge to vary that, even after numerous hours was spent on the horrible community efficiency of the VirtualBox VMs 
.