Studying DNS in 10 years
Right here’s a keynote I gave at RubyConf Mini final 12 months: Studying DNS in 10 years.
It’s about methods I exploit to study exhausting issues. I simply seen that they’d
launched the video the opposite day, so I’m simply posting it now regardless that I gave
the speak 6 months in the past.
Right here’s the video, in addition to the slides and a transcript of (roughly) what I
stated within the speak.
the video
the transcript
However this speak is
probably not about DNS. I imply, this can be a Ruby convention, proper? So this speak is
actually about studying exhausting issues, and DNS is an instance of one thing that was
exhausting for me to study.
It took me perhaps 16 years from the primary time
that like I purchased a website title and arrange my DNS data to once I actually
felt like I understood how the system labored.
And one factor I wish to say in the beginning of this speak, is that I feel that
taking like 16 years to study one thing like DNS is type of regular. The concept
that “I ought to perceive this already” is a bit foolish. For me, I used to be doing
different stuff for a lot of the 16 years! There was different stuff I needed to study.
And so, this speak shouldn’t be about how you must study any explicit
factor. I do not care in the event you learn the way DNS works! It is actually about how you can
method studying one thing exhausting that is a precedence so that you can study.
So, we’ll speak about studying by means of
a sequence of tiny deep dives. My favourite manner of studying issues is to do
nothing, more often than not.
That is why it takes 10 years.
So for six months I am going to do nothing after which like I am going to furiously study one thing
for perhaps half-hour or three hours or a day. After which I am going to declare
success and return to doing nothing for months. I discover this works very well
for me.
Listed here are a number of the methods we’ll speak about for doing these tiny deep dives
First, we’ll begin briefly by speaking about what DNS is.
Subsequent, we’ll speak about spying on DNS.
Then we’re gonna speak about being confused, which is my essential mode. (I am at all times confused about one thing!)
Then we’ll speak about studying the specification, we’ll going to
do some experiments, and we’ll implement our personal horrible model
of DNS.
And so what’s DNS actually briefly? DNS stands for the Area Identify System. And
each time you go to a web site like www.instance.com, your browser
must search for that web site’s IP handle. So DNS interprets
domains into IP addresses. It appears to be like up different details about area
names too, however we’re principally simply going to speak about IP addresses right now.
I wish to briefly promote why I feel DNS is cool, as a result of we’ll be
speaking about it so much.
One cool factor about DNS is that it is this invisible system that controls the
complete web.
For instance, you are in your telephone, you are utilizing Google Maps, it must know,
the place is maps.google.com, proper? Or in your
laptop, the place’s reddit.com? What is the IP handle? And if we
did not have DNS, all the web
would collapse.
I feel it is enjoyable to learn the way this behind the scenes stuff works.
The opposite factor about DNS I discover attention-grabbing is that it is actually previous. There’s
this doc (RFC
1035) which defines how DNS works, that was written in 1987. And if
you are taking that doc and also you write a program that works
the best way that paperwork says to work, your program will work. And I feel
that is type of wild, proper?
The fundamentals have not modified since earlier than I used to be born. So in the event you’re a bit of sluggish
about studying about it, that is okay: it is not going to alter out from underneath
you.
Subsequent I wish to speak about spying on DNS, which is certainly one of my favourite methods to
study issues.
I will speak about two spy instruments for DNS: dig and wireshark.
dig is a software for making DNS queries. We talked about you recognize, how your
browser must search for the IP handle for maps.google.com. We
can try this in dig!
After we run dig maps.google.com, it prints out 5 fields. Let’s
speak about what these 5 fields are.
I’ve used instance.com as a substitute of maps.google.com on this slide, however the fields
are the identical. Let’s speak about 4 of them:
We’ve the area title, no huge deal
The Time To Reside, which is how lengthy to cache that report for therefore this can be a at some point
You’ve the report kind, A stands for handle as a result of that is an IP handle
And you’ve got the content material, which is the IP handle
However I feel that the funniest discipline in a DNS report
is that this discipline within the center, IN, which stands for INternet. I assume in 1987, they thought that we could be on a variety of
totally different networks. So that they made an choice for it. In actuality, we’re all on the
web. And each DNS question has class set to “web”. There are a few
others question courses (CHAOS and HESIOD), which actually nearly no one makes use of.
We are able to additionally type of poke round on the
web with Dig. We have talked about A data to search for IP addresses.
However there are
different kinds of data like TXT data. So we’ll take a look at a TXT report
actually rapidly simply because I feel that is very enjoyable. We will take a look at twitter.com’s TXT data.
So TXT data are one thing that folks use for area verification, for
instance to show to Google that you just personal twitter.com.
So what you are able to do is you’ll be able to set this DNS
report google-site-verification. Google will inform you what to set
it to, you will set it, after which Google will imagine you.
I feel it is type of enjoyable you could
like type of poke round with DNS and see that Twitter is utilizing
Miro or Canva or Mixpanel, that is all public. It is like a bit of peek into what
persons are doing inside their firms
Oh, the opposite factor about dig is that by default, dig’s output appears to be like like
this, which may be very ugly and unreadable. There’s a variety of nonsense right here.
So dig has a configuration file, the place you’ll be able to put +noall +reply and
then your dig responses look a lot nicer (like they did within the screenshots
above) as a substitute of getting a variety of nonsense in them. At any time when attainable, I attempt to
make my instruments behave in a extra human manner.
The opposite factor I wish to speak about is Wireshark, which
is my favourite laptop networking software within the universe for spying on
all issues laptop networks. On this case, DNS queries. So let’s go take a look at
Wireshark.
After we make a DNS question like this and search for instance.com, Wireshark can seize it.
While you
begin wanting within the guts of issues, I feel it may be a bit scary at first. Like
what do all these numbers? It type of appears
like so much. So once I’m taking a look at one thing new, I attempt to begin by taking a look at stuff
that I perceive.
For instance, I do know that instance.com is a website title,
proper? So we should always ready to make use of Wireshark to go discover that area title within the
DNS question. If we click on into the “question” a part of the DNS packet, we are able to see 3
fields that we acknowledge. First, the area title.
We are able to additionally see the kind (“A”)
And the third one is the category which
is INternet, which is at all times the identical. What I discover comforting right here is that in
the question, there are actually solely 2 vital fields: a DNS question is simply saying “I need
the IP handle for instance.com”. There’s simply two fields. And that that at all times
makes me really feel a bit of bit higher about understanding one thing.
A fast caveat: your browser could be utilizing encrypted DNS and spying in your
DNS queries with Wireshark is not going to work in case your DNS is encrypted. However there’s
a number of non-encrypted DNS to spy on.
The second factor I wish to speak about for studying new issues is to
discover while you’re confused about one thing.
I wish to inform you a narrative, “the case of the mysterious caching”, of one thing
that occurred to me with DNS that basically confused me.
First, I wish to speak to you a bit of
bit about how DNS works a bit of bit extra. So on the left right here, you will have your
browser. And when your browser makes a DNS question, it asks a server known as a
resolver. And all you might want to know in regards to the resolver is that it is cache, which
as we all know is just like the worst factor in laptop science. So the resolver is a cache,
and it will get its data from the supply of fact, which has the actual solutions.
So your browser talks to a resolver, which is a cache.
On the time of this story, I had this psychological mannequin for like how I considered
DNS, which is that if I set a TTL (the cache time) of 5 minutes when configuring my DNS data,
then I’d by no means have to attend greater than 5 minutes. One thing you might want to
find out about me is that I am a really impatient individual. And I hate ready. So this
mannequin was principally working for me on the time, although there are a number of different very
vital caveats that we’re not going to get into.
However at some point I used to be organising a brand new subdomain for some new venture. For instance it
was new.jvns.ca. So I set it up. I made its DNS data, and I refreshed the
web page. And it wasn’t working. So I figured, that is tremendous, my mannequin says, I solely
have to attend 5 minutes, proper? As a result of that is what I used to be used to. However I
waited 5 minutes and nonetheless did not work.
And I used to be like, oh, no. My psychological mannequin was damaged! I didn’t really feel good.
And infrequently when this occurs to me, and I feel for many of us, if one thing
bizarre occurs with a pc, you let it go, proper? You would possibly determine okay, I
do not have time to enter a deep investigation right here. I am going to simply wait longer.
However generally I
have a variety of vitality, and perhaps I am feeling mad, like “the pc
cannot beat me right now”! As a result of there is a cause that that is taking place, proper? And I
wish to discover out what it’s. So at the present time for some cause. I had so much
of vitality.
So I began Googling furiously. And I discovered a helpful touch upon Stack
Overflow.
The Stack Overflow remark talked about one thing known as detrimental caching.
What’s that?
And so here is what it stated could be happening. The primary time I opened the
web site (earlier than the DNS data had been arrange), the DNS servers returned a
detrimental reply, saying hey,this area would not exist but. The code for that’s
NXDOMAIN, which is sort of a 404 for DNS.
And the resolver cached that detrimental NXDOMAIN response. So the truth that it
did not exist was cached.
So my subsequent query was: how lengthy do I’ve to attend for the cache to run out?
This brings us to a one other studying method.
I feel like perhaps the
most upsetting studying method to me is to learn a really boring
technical doc. I am like very impatient. I type of hate
studying boring issues. And so once I learn one thing very boring, I wish to
deliver a particular query. So on this case, I had a particular query, which is
how lengthy do I’ve to attend for the cache to run out?
In networking, all the pieces has a specification. The boring technical paperwork
are known as RFC is for request for feedback. I discover this title a bit humorous,
as a result of for DNS, a number of the essential RFCs are RFC 1034 and 1035. These have been written in 1987,
and the remark interval led to 1987. You possibly can undoubtedly now not make
feedback. However anyway, that is what they’re known as.
I personally type of love
RFCs as a result of they’re like the final word reply to many questions. There is a
nice sequence of HTTP RFCs, 9110 to 9114. DNS really has 1,000,000
totally different RFCs, it’s extremely upsetting, however the solutions are sometimes there. So I went
wanting. And I feel I went wanting as a result of once I learn feedback on
StackOverflow, I do not at all times belief them. How do I do know in the event that they’re correct? So
I needed to go to an authoritative supply.
So I discovered this doc known as RFC 2308. In part 3, it has this very boring
sentence, the TTL of this report is about to the minimal of the minimal discipline of the
SOA report and the TTL of the SOA itself. It signifies how lengthy a resolver might
cache the detrimental reply.
So, um, okay, cool. What does that imply, proper? Fortunately, we solely have one
query: I needn’t learn all the boring doc. I simply want to love
analyze this one sentence and determine it out.
So it is saying that the cache time is determined by two fields. I wish to present you
the precise knowledge it is speaking about, the SOA report.
Let us take a look at what occurs once we run dig +all asdfasdfasdfasdfasdf.jvns.ca
It says that the area would not exist, NXDOMAIN. However it additionally returns this
report known as the SOA report, which has some area metadata. And there are two
fields right here which are related.
Right here. I put this on a slide to attempt to make it a bit of bit clearer. This slide
is a bit tousled, however there’s this discipline on the finish that is known as the MINIMUM
discipline, and there is the TTL, time to dwell of the report, that I’ve tried to
circle.
And what it is saying is that if a report would not exist, the period of time the
resolver ought to cache “it would not exist” for is the minimal of these two numbers.
On this case, each of these numbers are 10800. In order that’s how lengthy should
wait. We’ve to attend 10,800 seconds. That is 3 hours.
And so I waited three hours after which all the pieces labored. And I discovered this
type of enjoyable to know as a result of usually like in the event you search for DNS recommendation it is going to
say one thing like, if one thing has gone improper, you might want to wait 48 hours. And I
don’t wish to wait 48 hours! I hate ready. So I adore it once I
can like use my mind to determine that I can await much less time.
Typically once I discover my psychological mannequin is damaged, it seems like I do not know
something
However on this case, and I feel in a variety of circumstances, there’s usually just some
issues I am lacking? Like this detrimental caching factor is like type of bizarre, however
it actually was the one factor I used to be lacking. There are a number of extra vital details about how
DNS caching works that I have not talked about, however I have not run into extra
issues I did not perceive since then. Although I am positive there’s one thing I
do not know.
So generally studying one small factor actually can clear up all of your issues.
I wish to say briefly that there is a answer to this detrimental caching drawback.
We talked about how like in the event you go to a website that is nonexistent, it will get
cached. The answer is that if you have not arrange your area’s DNS, do not go to
the area! Solely go to it after you set it up. So I’ve realized to do this and
now I nearly by no means have this drawback anymore. It is nice.
The subsequent factor I wish to speak about is doing experiments.
So as an instance we wish to do some experiments with caching.
I feel most individuals do not wish to make experimental adjustments to their area
names, as a result of they’re apprehensive about breaking one thing. Which I feel may be very comprehensible.
As a result of I used to be actually into DNS, I needed to experiment with DNS. And I additionally
needed different individuals to experiment with DNS with out having to fret about
breaking one thing. So I made this little web site with my good friend, Marie, known as
Mess with DNS
The concept is, in the event you do not wish to try this DNS experiments in your area, you
can do them on my area. And in the event you mess one thing up, it is my drawback, it is
not your drawback. And there have been no issues, in order that’s
tremendous.
So let’s use Mess With DNS to perform a little DNS experimentation
The way in which this works is you get a bit of subdomain. This one is
chair131.messwithdns.com. After which you can also make DNS data on it and take a look at
issues out. Right here we’re making a report for take a look at.char131.messwithdns.web, with
kind A, the IP 7.7.7.7, and TTL 3000 seconds.
What we’d anticipate to see is that if we make a question to the resolver, then it
asks type of just like the supply of fact, which we management. And we should always anticipate
the resolver to make just one question, as a result of it is cached. So I wish to do an
experiment and see if it is true that we get just one question.
So I will make a number of queries for it, with dig @1.1.1.1 take a look at.chair131.messwithdns.com.
I’ve queried it a bunch of occasions, perhaps 10 or 20.
Oh, cool. This is not what I anticipated to see. That is enjoyable, although, that is nice.
We made about 20 queries for that DNS report. The server logs all queries it
receives, so we are able to rely them.
Our server acquired 1, 2, 3, 4, 5, 6, 7, 8 queries. That is type of enjoyable. 8 is lower than 20.
One cause I love to do demos dwell on stage is that generally what I what
occurs is not precisely what I feel will occur. After I do that precise experiment
at residence, I simply get 1 question to the resolver.
So we solely noticed like eight queries right here. And I assume that that is
as a result of the resolver, 1.1.1.1, we’re speaking to has a couple of
unbiased cache, I assume there are 8 caches. This is smart to me as a result of
Cloudflare’s community is distributed — the precise machines I am speaking to right here
in Windfall should not the identical as those in Montreal.
That is attention-grabbing as a result of it complicates your concept about how caching works a
little bit, proper? Like perhaps a given DNS resolver really has like eight
caches and which one you get is random, and you are not at all times speaking
to the identical one. I feel that is what is going on on right here.
We are able to additionally do the identical experiment, however ask Google’s resolver, 8.8.8.8, as a substitute
of Cloudflare’s resolver.
And we’re seeing the same factor right here to what we noticed with Cloudflare, there are
perhaps 4 unbiased caches.
We might additionally do an experiment with detrimental caching, however no, I am not going to
do that demo. Sorry. I might simply see it going downhill. The issue is that
there’s too many alternative caches, and I really need there to be one cache, however
there’s like seven. That is tremendous, let’s transfer on.
Now I will speak
about my favourite technique for studying about stuff, which is to
write my very own very unhealthy model of the factor. And I wish to say that writing my
very unhealthy implementation provides me a extremely unreasonable quantity of confidence.
So that you would possibly suppose that writing DNS software program is difficult, proper? However it’s
simpler than you would possibly suppose, so long as you retain your expectations low.
To make the DNS queries, the very first thing we have to do is we have to
make a community connection. Let’s try this.
These 4 traces of Ruby join to eight.8.8.8, the Google DNS resolver, on UDP
port 53. Now we’re like midway there. So after we have made a connection,
we have to ship Google a DNS question. You could be pondering, Julia, I
do not know how you can write a DNS question.
However there isn’t any drawback. We are able to copy one from one thing else that is aware of what a
DNS question appears to be like like. AKA Wireshark.
So if I proper click on on this DNS question, it’s extremely small, however I am clicking on
“copy”, after which “copy as hex stream”.
You may not know what this implies but, however this can be a DNS question. And
you would possibly suppose that like, Hey, you’ll be able to’t simply copy and paste one thing and
then ship the very same factor and it will reply, however you’ll be able to. And it really works.
This is what the code appears to be like wish to ship this hex string we copied and pasted to eight.8.8.8.
So we take this like hex string that we copy and pasted, and paste it into our
tiny Ruby program, and use `.pack` to transform right into a string of bytes and ship it.
Now we run the Ruby program.
Let’s go to Wireshark and search for the packet we simply despatched. And we are able to see it there! There’s another noise in between, so I am going to cease the seize.
We are able to see that it is the similar packet as a result of the question ID matches, B962.
So we despatched a question to Google the reply server and we acquired a response proper? It
was like that is completely professional. There isn’t any drawback. It would not know that we copied and pasted it and that we do not know what it means!
However we do wish to know what this implies, proper? And so we’ll take this hex string and break up it into 2 elements.
The primary half is the header. And the second half is the
query, which comprises the precise area title we’re wanting up.
We will see how you can assemble these in Ruby, however first
I wish to speak about what a byte is for
one second. So this (b9) is the hexadecimal illustration of a byte. The way in which
I like to take a look at determine what meaning is simply kind it into IRB, if
you kind in 0xB9 it will print out, that is the quantity 184.
So the query is 12 bytes
These 12 bytes correspond
six numbers, that are two bytes every. So the primary quantity is the factor
b962 which is the question ID. The subsequent quantity is the flags, which
mainly on this case, means like this can be a question like hi there, I’ve a
query. After which there’s 4 extra sections, the variety of questions after which
the variety of solutions. We shouldn’t have any solutions. We solely have a query. So
we’re saying, hi there, I’ve one query. That is what the header means.
And the best way that we are able to do that in Ruby, is we are able to make a bit of array that has the
question ID, after which these numbers which
correspond to the opposite the opposite header fields, the flags after which 1 for 1
query, after which three zeroes for every of the three sections of solutions.
After which we have to inform Ruby how you can take these like six numbers and
then characterize them as bytes. So n right here means every
of those is meant to characterize it as two bytes, and it additionally means to make use of huge endian byte order.
Now let’s speak in regards to the query.
I broke up the query part right here. There are two elements
you would possibly acknowledge from instance.com: there’s instance, and com.
The way in which it really works is that first you will have a quantity (like 7), after which a
7-character string, like “instance”. The quantity tells you what number of characters to
anticipate in every a part of the area title. So it is 7, instance, 3, com, 0.
After which on the finish, you
have two extra fields for the kind and the category. Class 1 is code for
“web”. And kind 1 is code for “IP handle”, as a result of we wish to search for the
IP handle. is
So we are able to write a bit of little bit of code to do that. If we wish to translate
instance.com into seven instance three column zero, can like break up the area on
a dot after which like get its size and concatenate that collectively and put a 0 on
the tip. It is just a bit little bit of Ruby. how you can encode a website title.
After which we are able to wrap all this up
collectively the place we make a random question ID. And you then make
the header, encode the area title, after which we add the kind
and the category, 1 and 1, after which we are able to simply
concatenate all the pieces collectively and that is our question.
There’s undoubtedly extra work to do right here to print out the response, however I wrote
a 120-line Ruby script that parses the response too, and I wish to present you a fast demo of it working.
What area ought to we glance up>. rubyconfmini.com. All proper, let’s do it. Hey, it really works!
We’re on the finish! Let’s do a recap.
Okay. Let’s go over the methods we have talked about studying issues!
First, spy on it. I discover that once I take a look at issues like
to see like actually what’s taking place underneath the hood, and once I take a look at like,
what’s within the bytes, you recognize what is going on on? It is usually like not as
difficult as I feel. Like, oh, there’s simply the area title and the
kind. It actually makes me really feel way more assured that I perceive that factor.
I attempt to discover once I’m confused, and I wish to say once more, that
noticing while you’re confused is one thing that like we do not
at all times have time for proper? It is one thing to do when you will have the vitality. For
instance there’s this bizarre DNS question I noticed in one of many demos right now that I
do not perceive, however I ignored it as a result of, nicely, I am giving a chat. However perhaps at some point I am going to really feel like taking a look at it.
We talked about studying the specification, which, there are few occasions I really feel
like extra highly effective than once I’m in like a dialogue with somebody, and I KNOW that I’ve the appropriate reply as a result of, nicely, I learn the specification!
It is a very nice method to really feel sure.
I like to do experiments to verify that my understanding of stuff is true. And
usually I study that my understanding of one thing is improper! I had an instance in
this speak that I used to be going to incorporate and I did an experiment to verify that
that instance was true, and it wasn’t! And now I do know that. I like that
experiments on computer systems are very quick and low-cost and often haven’t any
penalties.
After which the very last thing we talked about and actually my favourite, however probably the most
work is like implementing your individual horrible model. For me,
the arrogance I get from writing like a horrible DNS implementation that works
on 11 totally different domains is unmatched. If my factor works in any respect, I really feel like,
wow, you’ll be able to’t inform me that I do not understand how DNS works! I carried out it! And
it would not matter if my implementation is “unhealthy” as a result of I do know that it really works!
I’ve examined it. I’ve seen it with my very own eyes. And I feel that simply feels
wonderful. And there are additionally no penalties since you’re by no means going to run
it in manufacturing. So it would not matter if it is horrible. It simply exists to offer
you large quantities of confidence in your self. And I feel that is very nice.
That is all for me. Thanks for listening.
due to the organizers!
Because of the RubyConf Mini organizers for doing such an important job with the
convention – it was the primary convention I’d been to since 2019, and I had a
nice time.