ssh whoami.filippo.io
I up to date the whoami.filippo.io dataset over the vacations, so it needs to be fairly correct no less than for a short while. In case you already know what I’m speaking about, under are some tidbits about how I fetched the brand new dataset and the way it’s saved.
In case you don’t, cease studying, and run this. I’ll wait.
$ ssh whoami.filippo.io
Right here’s an image of my grandmother’s cat, to keep away from spoilers.
What?!
There are two issues happening right here that is likely to be surprising.
The primary is that the SSH protocol supplies to the server all public keys the shopper is keen to provide signatures for, which by default are all the general public keys in your ssh-agent and in your ~/.ssh/id_*. That is considerably unavoidable. You possibly can technically make an authentication protocol that doesn’t disclose public keys to a peer that didn’t already know them, however it will be annoying: for instance, you couldn’t use ECDSA or RSA with out first having the server show information of the general public keys, as a result of it’s normally doable to recuperate the general public key from these signatures. Having the server show information of the general public keys earlier than the shopper makes use of them can also be kinda annoying, since you additionally don’t need the shopper to be taught concerning the public keys accepted by the server. You find yourself with a posh interactive protocol that wastes round-trips.
(That is additionally why we are able to do git clone git@github.com as a substitute of git clone FiloSottile@github.com. GitHub depends on the shopper sharing its public keys to know who’s attempting to authenticate. Which can also be why you possibly can’t use the identical SSH key for 2 accounts.)
The way it technically works is that the shopper sends public keys to the server till the server solutions that it likes one in all them, after which the shopper sends a signature from that key. The shopper is allowed to skip the primary half and begin sending signatures instantly, however it doesn’t as a result of producing a signature would possibly require person interplay (for instance to decrypt the personal key or to enter the PIN of a {hardware} token) and it’s unhealthy UX to require that for a key the server will reject. That is additionally why age ciphertexts encrypted to SSH keys carry a hash of the general public key: to let the shopper know if it ought to hassle the person to decrypt an encrypted personal key.
(A neat consequence is which you could take a look at what public keys a server accepts even with out having the corresponding personal key.)
The second factor happening that is likely to be surprising is that your GitHub SSH keys are, effectively, public. For instance, you possibly can see mine at https://github.com/FiloSottile.keys.
I knew this, what’s new?
whoami.filippo.io has been operating since 2015, initially on a dataset collected by Ben Cox. What’s new is that I now have a quicker strategy to refresh its keys database, and that it runs on new structure.
The GitHub GraphQL API now contains customers public keys, and because it permits fetching 100 customers per request and 5000 requests per hour it’s considerably quicker than utilizing the REST API and the .keys endpoint. What it lacks although is the flexibility to iterate by all customers.
You can also make a seek for all customers, which can let you know there are 97,616,627 customers on the time of this writing, however you possibly can solely fetch at most 1000 outcomes from a search, they usually don’t are available any clear order, so you possibly can’t simply make the following search begin the place the earlier one left off (or I didn’t determine how).
What you are able to do although is request accounts created in a sure time vary. In case you get the time vary proper, in order that it has lower than 1000 entries, you possibly can paginate by it, after which request the following time vary. This was just a little simpler stated than achieved, due to course registrations are available waves and the speed modifications through the years, however I finally constructed a simple adaptive algorithm that rarely overshot, and that went by all customers in lower than a pair weeks with out ever hitting the rate-limits. (Meaning it may have been just a little quicker with some concurrency, however ok.) That is how the ultimate GraphQL question appeared like:
{
search(
kind: USER
question: "kind:person created:{{ .From }}..{{ .To }}"
first: 100
{{ if .After }}after: "{{ .After }}"{{ finish }}
) {
userCount
pageInfo {
hasNextPage
endCursor
}
edges {
node {
... on Person {
databaseId
publicKeys(first: 100) {
nodes {
key
}
}
}
}
}
}
}
As soon as I had all of the keys as a pleasant ~5GB JSON Traces file, I needed to discover a strategy to deploy this that was easier than the earlier PostgreSQL database. I performed with some extra advanced concepts, however ultimately I attempted making a two column SQLite database from a SHA-256(key)[:16] PRIMARY KEY to the person ID and it was lower than 400MB, sufficiently small to only embed in a Docker picture deployed on Fly.io. (Hell, some base photographs are that enormous.)
That’s how this runs now. No shifting elements, and no want for me to sysadmin something. Bliss. You may see all of the supply at https://github.com/FiloSottile/whoami.filippo.io, though I’ve not printed the dataset. In case you have a compelling evaluation you wish to run, be at liberty to achieve out. I’m not too involved about sharing it as a result of it’s all simply fetched public knowledge anyway.
Have enjoyable, and take into account following me at @filippo@abyssdomain.expert.