Curious case of Martian visitors

At FastNetMon we work with UDP based mostly protocols such us Netflow, IPFIX and sFlow on day by day foundation.

These protocols are utilized by nice number of community tools distributors to ship details about visitors noticed by community gadget to monitoring software program.

As we speak our technical assist staff acquired report from buyer about problem with receiving Netflow visitors.

Because the very first step our assist determined to examine that VM with FastNetMon truly receives Netflow visitors and tcpdump helps us with such circumstances:

sudo tcpdump -n -i eth0  ‘port 2055’ -c 1
18:27:28.186333 IP 0.0.0.0.2055 > a.b.c.d.2055: UDP, size 1400

We clearly can see that machine receives visitors over port 2055 and FastNetMon should obtain it.

Sadly FastNetMon doesn’t observe any UDP visitors over port 2055:

sudo fcli present system_counters |grep netflow_all_protocols_total_flows_speed
netflow_all_protocols_total_flows_speed                    0

We clearly can see that it listens on port:

sudo netstat -lnpu
Lively Web connections (solely servers)
Proto Recv-Q Ship-Q Native Deal with           Overseas Deal with         State       PID/Program identify
udp6       0      0 :::2055                 :::*                                44283/fastnetmon

What’s the problem?

For all dropped UDP visitors Linux kernel has counters in netstat -su:

After working this command a number of instances we can not see any counters growing and clearly our dropped UDP visitors in direction of port 2055 just isn’t counted right here.

The problem with tcpdump that it may possibly see a number of visitors which will probably be discarded by Linux kernel on subsequent levels of packet processing pipeline and it was positively the case.

For instance if in case you have iptables rule which drop particular visitors tcpdump will see it however software won’t obtain it.

We have checked that machine has no iptables and final remaining suspect is Linux community stack. Let’s dig deeper.

You could discover that visitors is coming from fairly unusual IP tackle which is 0.0.0.0 and gadget which generates it clearly positioned someplace exterior because it’s separate bodily field.

This IP belong to special purpose range 0.0.0.0/8 which even has identify “This community”:

In response to RFC 1812 any visitors coming from this vary have to be dropped until it is coming from localhost / loop interface:

That is precisely what Linux kernel does.

Much more attention-grabbing that in subsequent part of RFC now we have point out that implementation should present choice to log such so referred to as martian packets (or just martians):

Clearly Linux is RFC compliant and now we have flag which controls behaviour of this logic. We are able to allow logic to print all discarded martians to dmesg this manner:

sudo sysctl internet.ipv4.conf.all.log_martians=1

After making this configuration change we instantly see details about dropped visitors in dmesg:

Do now we have any higher methods to seek out such visitors with out making any adjustments to Linux kernel configuration?

I discovered such approach throughout my Linux kernel supply code evaluation:

RT_CACHE-STAT_INC clearly will increase counter with identify in_martian_src when it observes martian visitors.

And you’ll simply learn worth of this counter utilizing software lnstat:

lnstat -k in_martian_dst,in_martian_src -w 20 -c 1
rt_cache|            rt_cache|
in_martian_dst|      in_martian_src|
0|           198940723|

It is comparatively difficult to learn as textual content and you could choose to look on screenshot of my console:

That clearly pinpoints root trigger and offers cause why hundreds of thousands of UDP packets to port 2055 had been dropped.

What’s the repair? One of the best ways to repair this problem is to alter Netflow / IPFIX / sFlow agent configuration on gadget to make use of one other reputable IP tackle as a substitute of 0.0.0.0.