The little ssh that (typically) could not

2024-02-05 17:14:51

Preface

It is a technical article chronicling one of the vital attention-grabbing bug hunts I’ve had the pleasure of chasing down.

At AdGear Technologies Inc. the place I work, ssh is king. We use it for administration, monitoring, deployments, log file harvesting, even some occasion streaming. It’s stable, dependable, has all of the predictability of a local unix software, and simply works.

Till in the future, random cron emails began flowing about it not working.

The timeout

The machines in our London information heart had been randomly failing to ship their occasion log information to our information machines in our Montreal information heart. This job is initiated periodically from cron, and the failure manifested itself as:

cron emails stating that the ssh was unsuccessful
- Generally hangs
- Generally exits with a timeout error
monitoring warnings down the road for in-house sanity checks detecting the lacking information in Montreal

We logged into the London machines, manually ran the push command, and it labored efficiently. We brushed it off as short-term community partitions.

The timeouts

However the failures saved popping up randomly. As soon as a day, a few instances a day, then one Friday morning, a number of instances an hour. It was clear one thing’s getting worse. We saved up with manually pushing the information till we work out what the issue was.

There have been 17 hops between London and Montreal. We constructed a profile of latency and packet loss for them, and located {that a} couple had been dropping 1-3% of packets. We filed a ticket with our London DC ops to route away from them.

Whereas London DC ops had been verifying the packet loss, we began seeing random timeouts from London to our SECOND information heart in Montreal, and hops to that information heart didn’t share the identical routes we noticed the packet loss at. We concluded packet loss just isn’t the primary downside across the identical time London DC ops replied saying they’re not capable of replicate the packet loss or timeouts and that every part regarded wholesome on their finish.

The revelation

Whereas manually maintaining with failed cron uploads, we seen an attention-grabbing sample. A file switch both succeeded at a excessive pace, or didn’t succeed in any respect and hung/timed out. There have been no cases of a file importing slowly and ending efficiently.

Eradicating the massive quantity of information from the equation, we had been capable of recreate the situation by way of easy vanilla ssh. On a London machine an “ssh mtl-machine” would both work instantly, or hold and by no means set up a connection. Eyebrows began going up.

The place the wild packets are

We triple-checked the ssh server configs and well being in Montreal:

The servers appeared wholesome by all measures
SSHd DNS reverse lookup was not enabled
SSHd Most shopper connections was excessive sufficient
We weren’t beneath assault
Bandwidth utilization was nowhere close to saturation

Moreover, even when one thing was off, we had been observing the hangs speaking to 2 utterly distinct information facilities in Montreal. Moreover, our different information facilities (non-London) had been speaking fortunately to Montreal. One thing about London was off.

We fired up tcpdump and began wanting on the packets, each in abstract and in captured pcaps loaded into wireshark. We noticed telltale indicators of packet loss and retransmission, but it surely was minimal and never notably worrisome.

We then captured full connections from instances the place ssh established efficiently, and full connections from instances the place the ssh connection hung.

Right here’s what we logically noticed when a connection from London to Montreal hung:

Regular TCP handshake
Bunch of ssh-specific backwards and forwards, with regular TCP ACK packets the place they need to be
A specific packet despatched from London and obtained in Montreal
The identical packet re-sent (and re-sent, a number of instances) from London and obtained in Montreal
Montreal’s simply not responding to it!

It didn’t make sense why Montreal was not responding (therefore London re-transmitting it). The connection was stalled at this level, because the layer 4 protocol was at a stalemate. Extra infuriatingly, in the event you kill the ssh try in London and re-launched it instantly, odds are it labored efficiently. When it did, tcpdump confirmed Montreal receiving the packet however responding to it, and issues moved on.

We enabled verbose debugging (-vvv) on the ssh shopper in London, and the hold occurred after it logged:

debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: discovered hmac-md5
debug1: kex: server->shopper aes128-ctr hmac-md5 none
debug2: mac_setup: discovered hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) despatched
debug1: anticipating SSH2_MSG_KEX_DH_GEX_GROUP

Googling “ssh hold SSH2_MSG_KEX_DH_GEX_GROUP” has many outcomes – from dangerous WiFi, to home windows TCP bugs, to buggy routers discarding TCP fragments. One resolution for LANs was to determine the trail’s MSS and set that because the MTU on each ends.

I saved decrementing the MTU on a London server down from 1500 – it didn’t assist till I hit the magic worth 576. At that time, I used to be not capable of get the ssh hanging conduct replicated. I had an ssh loop script working, and it was on-demand that I might trigger timeouts by bringing the MTU again as much as 1500, or make them disappear by setting it to 576.

Sadly these are public net servers and globally setting the MTU to 576 gained’t minimize it, however the above did recommend that maybe packet fragmentation or reassembly is damaged someplace.

Going again to test the obtained packets with tcpdump, there was no proof of fragmentation. The obtained packet dimension matched precisely the packet dimension despatched. If one thing did fragment the packet at byte 576+, one thing else reassembled it efficiently.

Twinkle twinkle little mis-shapen star

Digging in some extra, I used to be now full packet dumps (tcpdump -s 0 -X) as an alternative of simply the headers. Evaluating that magic packet in cases of ssh success vs ssh hold confirmed little or no distinction apart from TCP/IP header variations. It was nonetheless clear that that is the primary packet within the TCP connection that had sufficient information to bypass the 576-byte mark – all earlier packets had been a lot smaller.

Evaluating the identical packet, throughout a dangling occasion, because it left London, and as captured in Montreal, one thing caught my eye. One thing very refined, and I brushed it off as fatigue (it was late Friday at this level), however positive sufficient after just a few refreshes and comparisons, I wasn’t imagining issues.

Right here’s the packet because it left London (minus the primary few bytes figuring out the IP addresses):

0x0040:  0b7c aecc 1774 b770 ad92 0000 00b7 6563  .|...t.p......ec
0x0050:  6468 2d73 6861 322d 6e69 7374 7032 3536  dh-sha2-nistp256
0x0060:  2c65 6364 682d 7368 6132 2d6e 6973 7470  ,ecdh-sha2-nistp
0x0070:  3338 342c 6563 6468 2d73 6861 322d 6e69  384,ecdh-sha2-ni
0x0080:  7374 7035 3231 2c64 6966 6669 652d 6865  stp521,diffie-he
0x0090:  6c6c 6d61 6e2d 6772 6f75 702d 6578 6368  llman-group-exch
0x00a0:  616e 6765 2d73 6861 3235 362c 6469 6666  ange-sha256,diff
0x00b0:  6965 2d68 656c 6c6d 616e 2d67 726f 7570  ie-hellman-group
0x00c0:  2d65 7863 6861 6e67 652d 7368 6131 2c64  -exchange-sha1,d
0x00d0:  6966 6669 652d 6865 6c6c 6d61 6e2d 6772  iffie-hellman-gr
0x00e0:  6f75 7031 342d 7368 6131 2c64 6966 6669  oup14-sha1,diffi
0x00f0:  652d 6865 6c6c 6d61 6e2d 6772 6f75 7031  e-hellman-group1
0x0100:  2d73 6861 3100 0000 2373 7368 2d72 7361  -sha1...#ssh-rsa
0x0110:  2c73 7368 2d64 7373 2c65 6364 7361 2d73  ,ssh-dss,ecdsa-s
0x0120:  6861 322d 6e69 7374 7032 3536 0000 009d  ha2-nistp256....
0x0130:  6165 7331 3238 2d63 7472 2c61 6573 3139  aes128-ctr,aes19
0x0140:  322d 6374 722c 6165 7332 3536 2d63 7472  2-ctr,aes256-ctr
0x0150:  2c61 7263 666f 7572 3235 362c 6172 6366  ,arcfour256,arcf
0x0160:  6f75 7231 3238 2c61 6573 3132 382d 6362  our128,aes128-cb
0x0170:  632c 3364 6573 2d63 6263 2c62 6c6f 7766  c,3des-cbc,blowf
0x0180:  6973 682d 6362 632c 6361 7374 3132 382d  ish-cbc,cast128-
0x0190:  6362 632c 6165 7331 3932 2d63 6263 2c61  cbc,aes192-cbc,a
0x01a0:  6573 3235 362d 6362 632c 6172 6366 6f75  es256-cbc,arcfou
0x01b0:  722c 7269 6a6e 6461 656c 2d63 6263 406c  r,rijndael-cbc@l
0x01c0:  7973 6174 6f72 2e6c 6975 2e73 6500 0000  ysator.liu.se...
0x01d0:  9d61 6573 3132 382d 6374 722c 6165 7331  .aes128-ctr,aes1
0x01e0:  3932 2d63 7472 2c61 6573 3235 362d 6374  92-ctr,aes256-ct
0x01f0:  722c 6172 6366 6f75 7232 3536 2c61 7263  r,arcfour256,arc
0x0200:  666f 7572 3132 382c 6165 7331 3238 2d63  four128,aes128-c
0x0210:  6263 2c33 6465 732d 6362 632c 626c 6f77  bc,3des-cbc,blow
0x0220:  6669 7368 2d63 6263 2c63 6173 7431 3238  fish-cbc,cast128
0x0230:  2d63 6263 2c61 6573 3139 322d 6362 632c  -cbc,aes192-cbc,
0x0240:  6165 7332 3536 2d63 6263 2c61 7263 666f  aes256-cbc,arcfo
0x0250:  7572 2c72 696a 6e64 6165 6c2d 6362 6340  ur,rijndael-cbc@
0x0260:  6c79 7361 746f 722e 6c69 752e 7365 0000  lysator.liu.se..
0x0270:  00a7 686d 6163 2d6d 6435 2c68 6d61 632d  ..hmac-md5,hmac-
0x0280:  7368 6131 2c75 6d61 632d 3634 406f 7065  sha1,umac-64@ope
0x0290:  6e73 7368 2e63 6f6d 2c68 6d61 632d 7368  nssh.com,hmac-sh
0x02a0:  6132 2d32 3536 2c68 6d61 632d 7368 6132  a2-256,hmac-sha2
0x02b0:  2d32 3536 2d39 362c 686d 6163 2d73 6861  -256-96,hmac-sha
0x02c0:  322d 3531 322c 686d 6163 2d73 6861 322d  2-512,hmac-sha2-
0x02d0:  3531 322d 3936 2c68 6d61 632d 7269 7065  512-96,hmac-ripe
0x02e0:  6d64 3136 302c 686d 6163 2d72 6970 656d  md160,hmac-ripem
0x02f0:  6431 3630 406f 7065 6e73 7368 2e63 6f6d  [email protected]
0x0300:  2c68 6d61 632d 7368 6131 2d39 362c 686d  ,hmac-sha1-96,hm
0x0310:  6163 2d6d 6435 2d39 3600 0000 a768 6d61  ac-md5-96....hma
0x0320:  632d 6d64 352c 686d 6163 2d73 6861 312c  c-md5,hmac-sha1,
0x0330:  756d 6163 2d36 3440 6f70 656e 7373 682e  umac-64@openssh.
0x0340:  636f 6d2c 686d 6163 2d73 6861 322d 3235  com,hmac-sha2-25
0x0350:  362c 686d 6163 2d73 6861 322d 3235 362d  6,hmac-sha2-256-
0x0360:  3936 2c68 6d61 632d 7368 6132 2d35 3132  96,hmac-sha2-512
0x0370:  2c68 6d61 632d 7368 6132 2d35 3132 2d39  ,hmac-sha2-512-9
0x0380:  362c 686d 6163 2d72 6970 656d 6431 3630  6,hmac-ripemd160
0x0390:  2c68 6d61 632d 7269 7065 6d64 3136 3040  ,hmac-ripemd160@
0x03a0:  6f70 656e 7373 682e 636f 6d2c 686d 6163  openssh.com,hmac
0x03b0:  2d73 6861 312d 3936 2c68 6d61 632d 6d64  -sha1-96,hmac-md
0x03c0:  352d 3936 0000 0015 6e6f 6e65 2c7a 6c69  5-96....none,zli
0x03d0:  6240 6f70 656e 7373 682e 636f 6d00 0000  [email protected]...
0x03e0:  156e 6f6e 652c 7a6c 6962 406f 7065 6e73  .none,zlib@opens
0x03f0:  7368 2e63 6f6d 0000 0000 0000 0000 0000  sh.com..........
0x0400:  0000 0000 0000 0000 0000 0000            ............

And right here’s the identical packet because it arrived in Montreal:

0x0040:  0b7c aecc 1774 b770 ad92 0000 00b7 6563  .|...t.p......ec
0x0050:  6468 2d73 6861 322d 6e69 7374 7032 3536  dh-sha2-nistp256
0x0060:  2c65 6364 682d 7368 6132 2d6e 6973 7470  ,ecdh-sha2-nistp
0x0070:  3338 342c 6563 6468 2d73 6861 322d 6e69  384,ecdh-sha2-ni
0x0080:  7374 7035 3231 2c64 6966 6669 652d 6865  stp521,diffie-he
0x0090:  6c6c 6d61 6e2d 6772 6f75 702d 6578 6368  llman-group-exch
0x00a0:  616e 6765 2d73 6861 3235 362c 6469 6666  ange-sha256,diff
0x00b0:  6965 2d68 656c 6c6d 616e 2d67 726f 7570  ie-hellman-group
0x00c0:  2d65 7863 6861 6e67 652d 7368 6131 2c64  -exchange-sha1,d
0x00d0:  6966 6669 652d 6865 6c6c 6d61 6e2d 6772  iffie-hellman-gr
0x00e0:  6f75 7031 342d 7368 6131 2c64 6966 6669  oup14-sha1,diffi
0x00f0:  652d 6865 6c6c 6d61 6e2d 6772 6f75 7031  e-hellman-group1
0x0100:  2d73 6861 3100 0000 2373 7368 2d72 7361  -sha1...#ssh-rsa
0x0110:  2c73 7368 2d64 7373 2c65 6364 7361 2d73  ,ssh-dss,ecdsa-s
0x0120:  6861 322d 6e69 7374 7032 3536 0000 009d  ha2-nistp256....
0x0130:  6165 7331 3238 2d63 7472 2c61 6573 3139  aes128-ctr,aes19
0x0140:  322d 6374 722c 6165 7332 3536 2d63 7472  2-ctr,aes256-ctr
0x0150:  2c61 7263 666f 7572 3235 362c 6172 6366  ,arcfour256,arcf
0x0160:  6f75 7231 3238 2c61 6573 3132 382d 6362  our128,aes128-cb
0x0170:  632c 3364 6573 2d63 6263 2c62 6c6f 7766  c,3des-cbc,blowf
0x0180:  6973 682d 6362 632c 6361 7374 3132 382d  ish-cbc,cast128-
0x0190:  6362 632c 6165 7331 3932 2d63 6263 2c61  cbc,aes192-cbc,a
0x01a0:  6573 3235 362d 6362 632c 6172 6366 6f75  es256-cbc,arcfou
0x01b0:  722c 7269 6a6e 6461 656c 2d63 6263 406c  r,rijndael-cbc@l
0x01c0:  7973 6174 6f72 2e6c 6975 2e73 6500 0000  ysator.liu.se...
0x01d0:  9d61 6573 3132 382d 6374 722c 6165 7331  .aes128-ctr,aes1
0x01e0:  3932 2d63 7472 2c61 6573 3235 362d 6374  92-ctr,aes256-ct
0x01f0:  722c 6172 6366 6f75 7232 3536 2c61 7263  r,arcfour256,arc
0x0200:  666f 7572 3132 382c 6165 7331 3238 2d63  four128,aes128-c
0x0210:  6263 2c33 6465 732d 6362 632c 626c 6f77  bc,3des-cbc,blow
0x0220:  6669 7368 2d63 6263 2c63 6173 7431 3238  fish-cbc,cast128
0x0230:  2d63 6263 2c61 6573 3139 322d 6362 632c  -cbc,aes192-cbc,
0x0240:  6165 7332 3536 2d63 6263 2c61 7263 666f  aes256-cbc,arcfo
0x0250:  7572 2c72 696a 6e64 6165 6c2d 6362 7340  ur,rijndael-cbs@
0x0260:  6c79 7361 746f 722e 6c69 752e 7365 1000  lysator.liu.se..
0x0270:  00a7 686d 6163 2d6d 6435 2c68 6d61 732d  ..hmac-md5,hmas-
0x0280:  7368 6131 2c75 6d61 632d 3634 406f 7065  sha1,umac-64@ope
0x0290:  6e73 7368 2e63 6f6d 2c68 6d61 632d 7368  nssh.com,hmac-sh
0x02a0:  6132 2d32 3536 2c68 6d61 632d 7368 7132  a2-256,hmac-shq2
0x02b0:  2d32 3536 2d39 362c 686d 6163 2d73 7861  -256-96,hmac-sxa
0x02c0:  322d 3531 322c 686d 6163 2d73 6861 322d  2-512,hmac-sha2-
0x02d0:  3531 322d 3936 2c68 6d61 632d 7269 7065  512-96,hmac-ripe
0x02e0:  6d64 3136 302c 686d 6163 2d72 6970 756d  md160,hmac-ripum
0x02f0:  6431 3630 406f 7065 6e73 7368 2e63 7f6d  [email protected]
0x0300:  2c68 6d61 632d 7368 6131 2d39 362c 786d  ,hmac-sha1-96,xm
0x0310:  6163 2d6d 6435 2d39 3600 0000 a768 7d61  ac-md5-96....h}a
0x0320:  632d 6d64 352c 686d 6163 2d73 6861 312c  c-md5,hmac-sha1,
0x0330:  756d 6163 2d36 3440 6f70 656e 7373 782e  umac-64@openssx.
0x0340:  636f 6d2c 686d 6163 2d73 6861 322d 3235  com,hmac-sha2-25
0x0350:  362c 686d 6163 2d73 6861 322d 3235 362d  6,hmac-sha2-256-
0x0360:  3936 2c68 6d61 632d 7368 6132 2d35 3132  96,hmac-sha2-512
0x0370:  2c68 6d61 632d 7368 6132 2d35 3132 3d39  ,hmac-sha2-512=9
0x0380:  362c 686d 6163 2d72 6970 656d 6431 3630  6,hmac-ripemd160
0x0390:  2c68 6d61 632d 7269 7065 6d64 3136 3040  ,hmac-ripemd160@
0x03a0:  6f70 656e 7373 682e 636f 6d2c 686d 7163  openssh.com,hmqc
0x03b0:  2d73 6861 312d 3936 2c68 6d61 632d 7d64  -sha1-96,hmac-}d
0x03c0:  352d 3936 0000 0015 6e6f 6e65 2c7a 7c69  5-96....none,z|i
0x03d0:  6240 6f70 656e 7373 682e 636f 6d00 0000  [email protected]...
0x03e0:  156e 6f6e 652c 7a6c 6962 406f 7065 6e73  .none,zlib@opens
0x03f0:  7368 2e63 6f6d 0000 0000 0000 0000 0000  sh.com..........
0x0400:  0000 0000 0000 0000 0000 0000            ............

Did one thing there catch your eye ? If not, I don’t blame you. Be at liberty to repeat every right into a textual content editor and quickly change back-and-forth to see some characters dance. Right here’s what it appears to be like like after they’re positioned in vimdiff:

Nicely properly properly. It’s not packet loss, it’s packet corruption! Very refined, very predictable packet corruption.

Some attention-grabbing notes:

The decrease a part of the packet (<576 bytes) is unaffected
The affected portion is predictably corrupted on the fifteenth byte of each 16
The corruption is predictable. All cases of “h” change into “x”, all cases of “c” change into “s”

Some readers might need already checked ASCII charts and reached the conclusion: There’s a single bit statically caught at “1” someplace. Flipping the 4th bit in a byte to 1 would reliably corrupt the above letters on the left aspect to the worth on the suitable aspect.

The plain culprits inside our management (NIC playing cards, receiving machines) are usually not suspect as a result of sample of failure noticed (a number of London machines -> A number of Montreal information facilities and machines). It’s acquired to be one thing upstream and near London.

Going again to validate, issues began to make sense. I additionally seen a bit trace in tcpdump verbose mode (tcp cksum dangerous) which was missed earlier than. A Montreal machine receiving this packet discarded it on the kernel degree after realizing it’s corrupt, by no means passing it to the userland ssh daemon. London then re-transmitted it, going via the identical corruption, getting the identical silent remedy. From ssh and sshd’s perspective, the connection was at a stalemate. From tcpdump’s perspective, there was no loss, and Montreal machines seemed to be simply ignoring information.

We despatched these findings to our London DC ops, and inside a couple of minutes they modified outbound routes dramatically. The primary router hop, and most hops afterwards, had been totally different. The hanging downside disappeared.

Late Friday evening fixes are good as a result of you’ll be able to calm down and never carry issues and assist workers into the weekend 🙂

The place’s Waldo

Blissful that we had been not affected by this downside and that our methods are caught up with the backlog, I made a decision I’d strive my hand at truly discovering the machine inflicting the corruption.

Having the London routes up to date to not undergo the outdated path meant that I couldn’t reproduce the issue simply. I requested round till I discovered a good friend with a FreeBSD field in Montreal I might use, which was nonetheless accessed via the outdated routes from London.

Subsequent, I needed to guarantee that the corruption is predictable even with out ssh involvement. This was trivially confirmed with just a few pipes.

In Montreal:

nc -l -p 4000 > /dev/null

Then in London:

cat /dev/zero | nc mtl 4000

Once more, accounting for the randomness issue and settings issues up in a retry loop, I acquired just a few packets which take away any doubt in regards to the earlier conclusions. Right here’s a part of one – do not forget that we’re sending only a stream of nulls(zeroes):

0x0210  .....
0x0220  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0230  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0240  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0250  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0260  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0270  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0280  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0290  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02a0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02b0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02c0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02d0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02e0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02f0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0300  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0310  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0320  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0330  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0340  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0350  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0360  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0370  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0380  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0390  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03a0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03b0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03c0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03d0  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x03e0  .....

With the bug replicated, I wanted to discover a solution to isolate which of the 17 hops alongside that path trigger the corruption. There was merely no solution to name up the supplier of every cluster to ask them to test their methods.

I made a decision pinging every router, incrementally, could be the best way to go. I crafted particular ICMP packets which can be giant sufficient to go over the 576 security margin, and crammed fully with NULLs. Then pinged the Montreal machine with them from London.

They got here again completely regular. There was no corruption.

I attempted all variations of pace, padding, dimension – to no avail. I merely couldn’t observe corruption within the returned ICMP ping packets.

I changed the netcat pipes with UDP as an alternative of TCP. Once more there was no corruption.

The corruption wanted TCP to be reproducible – and TCP wants 2 cooperating endpoints. I attempted in useless to see if all 17 router hops had an open TCP port I can speak to immediately, to no avail.

It appeared there was no straightforward means an exterior occasion can pinpoint the dangerous apple. Or was there ?

Mirror mirror on the wall

To detect whether or not corruption occurred or not, we want one in every of these eventualities:

Management over the TCP peer we’re speaking to examine the packet on the vacation spot
- Not simply in userland, the place the packet wouldn’t get delivered if the TCP checksum failed, however root + tcpdump to examine it because it arrives
A TCP peer that acts as an echo server to reflect again the information it obtained, so we get to examine it on the sending node and detect corruption there

It immediately occurred to me that the second information level is obtainable to us. Not per-se, however contemplate this: In our very first style of the issue, we noticed ssh shoppers hanging when speaking to ssh servers over the corrupting hop. It is a good passive sign that we will use as an alternative of the energetic “echo” sign.

… and there are many open ssh servers on the market on the web to assist us out.

We don’t want precise accounts on these servers – we simply must kickstart the ssh connection and see if the cipher trade part succeeds or hangs (with an inexpensive variety of retries to account for corruption randomness).

So this plan was hatched:

Use the great nmap software – particularly – its “random IP” mode – to make an inventory of geographically distributed open ssh servers
Take a look at every server to find out whether or not it’s:
- Unresponsive/unpredictable/firewalled -> Ignore it
- Negotiates efficiently after being retried N instances -> mark as “good”
- Negotiates with hangs on the telltale part after being retried N instances -> mark as “dangerous”
For each “good” and “dangerous” servers, bear in mind the traceroute to them

The thought was this: All servers marked as “dangerous” will share just a few hops of their traceroute. We are able to then take that set of suspect hops, and subtract from it any that seem within the traceroutes of the “good” servers. Hopefully what’s left is just one or two.

After spending an hour manually doing the above train, I finished to examine the information. I had categorized 16 servers as “BAD” and 25 servers as “GOOD”.

The primary train was to seek out the checklist of hops that seem in all of the traceroutes of the “BAD” servers. As I cleaned and trimmed the checklist, I spotted I gained’t even must get to the “GOOD” checklist to take away false positives. Throughout the “BAD” lists alone, there remained only one that was widespread to all of them.

For what it’s value, it was 2 suppliers away: London -> N hops upstream1 -> Y hops upstream2

It was the primary in Y hops of upstream2 – proper on the edge between upstream1 and upstream2, corrupting random TCP packets, inflicting many retries, and, relying on the protocol’s logical back-and-forth, hangs, or diminished transmission charges. You could have been a telephony supplier who sufferred dropped calls, a retailer who misplaced just a few clients or gross sales, the probabilities actually are infinite.

I adopted up with our London DC ops with the only hop’s IP tackle. Hopefully with their direct relationship with upstream1 they’ll escalate via there and get it fastened.

/filed beneath loopy devops warfare tales

Replace

By way of upstream1, I acquired affirmation that the hop I identified (first in upstream2) had an inside “administration module failure” which affected BGP and routing between two inside networks. It’s nonetheless down (they’ve routed round it) till they obtain a alternative for the defective module.

Thanks for the type phrases and nice feedback right here on Disqus, Reddit (/r/linux & /r/sysadmin) and hacker news