6 Raspberry Pis, 6 SSDs on a Mini ITX Motherboard

A number of months in the past somebody advised me a few new Raspberry Pi Compute Module 4 cluster board, the DeskPi Super6c.

You could have heard of one other Pi CM4 cluster board, the Turing Pi 2, however that board isn’t but delivery. It had a very successful Kickstarter campaign, however manufacturing has been delayed as a consequence of components shortages.

The Turing Pi 2 gives some distinctive options, like Jetson Nano compatibility, distant administration, a totally managed Ethernet change (able to VLAN help and hyperlink aggregation). However if you happen to simply need to slap a bunch of Raspberry Pis inside a tiny type issue, the Super6c is about as trim as you will get—and it is accessible right now!

On the highest, there are slots for as much as six Compute Module 4s, and every slot exposes IO pins (although not the complete Pi GPIO), a Micro USB port for flashing eMMC CM4 modules, and a few standing LEDs.

On the underside, there are six NVMe SSD slots (M.2 2280 M-key), and 6 microSD card slots, so you possibly can boot Lite CM4 modules (these with out built-in eMMC).

On the IO facet, there are a bunch of ports tied to CM4 slot 1—twin HDMI, two USB 2.0 (plus an inner USB 2.0 header), and micro USB, so you possibly can handle your complete cluster self-contained. You possibly can plug a keyboard, monitor, and mouse into the primary node, and use it to arrange every thing else if you’d like. (That was one criticism I had with the Turing Pi 2—there was no possibility of managing the cluster with out utilizing one other laptop.)

There’s additionally two energy inputs: a barrel jack accepting 19-24V DC (the board comes with a 100W energy adapter), and a 4-pin ATX 12V energy header if you wish to use an inner PSU.

Lastly, there are six little exercise LEDs protruding the again, one for every Pi. Watching the cluster as Ceph was operating made me consider WOPR, from the movie War Games—simply on a a lot smaller scale!

Ceph storage cluster

Since this board exposes a lot storage immediately on the underside, I made a decision to put in Ceph on it utilizing cephadm following this guide.

Ceph is an open-source storage cluster resolution that manages object/block/file-level storage throughout a number of computer systems and drives, much like RAID on one laptop—besides you possibly can mixture a number of storage gadgets throughout a number of computer systems.

I needed to do a pair further steps through the set up since I made a decision to run Raspberry Pi OS (a Debian by-product) as a substitute of operating Fedora just like the information urged:

1. I needed to allow the backports repo by including the road deb http://deb.debian.org/debian unstable fundamental contrib non-free to my /and many others/apt/sources.listing file after which operating sudo apt replace.

2. I added a file at /and many others/apt/preferences.d/99pin-unstable with the traces:

    Bundle: *
    Pin: launch a=secure
    Pin-Precedence: 900
   
    Bundle: *
    Pin: launch a=secure
    Pin-Precedence: 10
   

3. After that, I might set up cephadm with sudo apt set up -y cephadm

As soon as cephadm was put in, I might arrange the Ceph cluster utilizing the next command (inserting the primary node’s IP handle):

# cephadm bootstrap --mon-ip 10.0.100.149

This bootstraps a Ceph cluster, however you continue to want so as to add particular person hosts to the cluster, and I elected to try this by way of Ceph’s internet UI. The bootstrap command outputs an admin login and the dashboard URL, so I went there, logged in, up to date my password, and began including hosts:

I constructed this Ansible playbook to assist with the setup, since there are a pair different steps that should be run on all the nodes, like copying the Ceph pubkey to every node’s root consumer authorized_keys file, and putting in Podman and LVM2 on every node.

However as soon as that was achieved, and all my hosts had been added, I used to be capable of arrange a storage pool on the cluster, utilizing the three.3 TiB of built-in NVMe storage (distributed throughout 5 nodes). I used Ceph’s built-in benchmarking software rados to run some sequential write and skim exams:

See Also

Feds discover “substantial’’ security challenge at SC nuclear plant

I used to be capable of get 70-80 MB/sec write speeds on the cluster, and 110 MB/sec learn speeds. Not too dangerous, contemplating your complete factor’s operating over a 1 Gbps community. You possibly can’t actually enhance throughput because of the Pi’s IO limitations—perhaps within the subsequent technology of Pi, we are able to get sooner networking!

Throw in different options like encryption, although, and the speeds are positive to fall off a bit extra. I additionally needed to check an NFS mount throughout the Pis from my Mac, however I kept getting errors when I tried adding the NFS service to the cluster.

If you wish to persist with an ARM construct, a devoted Ceph storage equipment just like the Mars 400 remains to be going to obliterate a hobbyist board like this, by way of community bandwidth and IOPS—regardless of its slower CPUs. After all, that efficiency comes at a price; the Mars 400 is a $12,000 server!

Video

I produced a full video of the cluster construct, with extra details about the {hardware} and the way I set it up, and that is embedded beneath:

For this challenge, I printed the Ansible automation playbooks I arrange within the DeskPi Super6c Cluster repo on GitHub, and I’ve additionally uploaded the custom IO shield I designed to Thingiverse.

Conclusion

This board appears ultimate for experimentation—assuming you can find a bunch of CM4s for list price. Particularly if you happen to’re dipping your toes into the waters of K3s, Kubernetes, or Ceph, this board enables you to throw collectively as much as six bodily nodes, with out having to handle six energy adapters, six community cables, and a separate community change.

Many individuals will say “simply purchase one PC and run VMs on it!”, however to that, I say “phooey.” Coping with bodily nodes is a good way to be taught extra about networking, distributed storage, and multi-node utility efficiency—far more so than operating VMs inside a sooner single node!

When it comes to manufacturing utilization, the Super6c could possibly be helpful in some edge computing eventualities, particularly contemplating it makes use of simply 17W of energy for six nodes at idle, or about 24W most, however actually, utilizing different extra highly effective mini/SFF PCs can be less expensive proper now.