Now Reading
A Dive into Mikrotik’s Bizarre SmartNIC (CCR2004-1G-2XS-PCIe) // Alyx Wijers

A Dive into Mikrotik’s Bizarre SmartNIC (CCR2004-1G-2XS-PCIe) // Alyx Wijers

2023-12-02 01:21:52

The Mikrotik CCR2004-1G-2XS-PCIe

an addendum has been posted, learn it here

2023/12/02 – Hi there, Hacker Information! Take a look at these photos of my cat: https://emma.pet

Again in February, Mikrotik introduced the CCR2004-1G-2XS-PCIe, a peculiar 25gbps NIC.
They’ve thrown a complete router right into a PCIe NIC. However why?

Mikrotik’s product itemizing has illustrations that counsel utilizing it in a file server and a workstation, with a copper DAC in between. For one thing like a primary NIC, positive, that is smart. However why throw in all the routing and firewall capabilities? All of it appears wildly extreme for that usecase (nevertheless it does appear to be one of many least expensive model new 25Gbe NICs at the moment accessible).

I can’t actually see it becoming into giant infrastructure deployments aside from getting used as an affordable 25Gbps NIC within the default passthrough configuration, and appears to be focused largely in direction of small companies.

I’m undecided if this product will ever really take off, because it’s been over 3 months because it was introduced (or launched?) and is seemingly nowhere to be seen. I can’t discover any correct evaluations for it, and just about all the data on it’s simply commentary on or immediately parroting the official announcement (so I suppose I’m first?).
The lifecycle state of this can be very unclear: Mikrotik’s web site simply says “ship buy questions”, there are 34 pages complete that point out it on their web site (a overwhelming majority of that are individuals asking why in discussion board threads, the remaining are a handful of manuals and product matrixes), and virtually all of the Mikrotik licensed retailers both say “Pre-Order” or “Out of Inventory”. I’m not even positive if I’m imagined to have this factor.

Whatever the bizarre lifecycle void this falls into, I consider I’ve discovered the best use case for it: single server co-location.

Once you lease co-location area in sufficiently small portions to lease when it comes to “rack items” as a substitute of “racks”, you’re going to be paying further for each single area you employ.
They usually present no answer for firewalls and no customer-configurable routers and simply hand you an uplink that’s wide-open to any site visitors that might presumably head your means. The standard configuration for customized routing and firewalls is to connect the WAN port to a digital machine working one thing like Vyos or pfSense after which making a LAN vSwitch and attaching it to all your VMs, which burns further CPU cycles on all the site visitors processing.

Apart from the bizarre lifecycle state of affairs this card has, there’s additionally an inconsistency between what’s listed on their primary product web page which says that it runs a 1.5GHz Annapurna Labs AL52400 CPU and the handbook and spec sheet, which says it makes use of a 2GHz Annapurna Labs AL32400 CPU.
The 2 completely different CPU speeds along with the completely different mannequin numbers leads me to consider they’re completely different chips however with out taking it aside I can’t verify. I’m unable to seek out any precise documentation on the AL52400, so it may both be an error or one thing from an earlier “beta” card. Sadly, I’m not in a position to get any good diagnostic info on what it really is from the machine itself.
Mikrotik’s documentation lists system useful resource print as additionally outputting a CPU subject that reveals what the mannequin is, however this doesn’t output one. I’m inclined to consider that it’s the AL32400 because it’s the identical one which seems in its cousin, the CCR2004-1G-12S+2XS.

[admin@MikroTik] /system/useful resource> print
                   uptime: 1h8m25s
                  model: 7.1.4 (secure)
               build-time: Mar/21/2022 11:23:09
         factory-software: 7.1.4
              free-memory: 3875.9MiB
             total-memory: 4032.0MiB
                cpu-count: 4
                 cpu-load: 0%
           free-hdd-space: 106.9MiB
          total-hdd-space: 129.0MiB
  write-sect-since-reboot: 174
         write-sect-total: 174
               bad-blocks: 0%
        architecture-name: arm64
               board-name: CCR2004-1G-2XS-PCIe
                 platform: MikroTik

The Mikrotik CCR2004-1G-2XS-PCIe

The CCR2004-1G-2XS-PCIe is a smooth, black, single-slot card with a single blower fan in direction of the again. It has two SFP28 cages for uplinks, and one RJ-45 connector.
Mikrotik’s specs listing it as 170mm x 69mm x 18mm, sufficiently small to suit comfortably in my half-depth HP ProLiant DL20 G9. It does partially block the SATA port that’s beneath it, so if you wish to entry the DL20’s SATA port you’ll have to interchange the DL20’s unique SATA cable with one which has a 90° connector.
It doesn’t require any energy sources aside from the PCIe slot it’s put in in.

The Mikrotik CCR2004-1G-2XS-PCIe in a server viewed from above with no chassis lid

The Mikrotik CCR2004-1G-2XS-PCIe in my HP ProLiant DL20 G9.


The handbook suggests including a PCIe initialization delay in your BIOS settings as a result of the cardboard must be up earlier than the host system tries to initialize it. I couldn’t discover any apparent methods to try this on my HP ProLiant DL20 G9 after going by way of each menu, however ultimately it didn’t matter as a result of the cardboard was all the time up lengthy earlier than the host tried to initialize it.
The brochure (however not the handbook? Severely, the handbook has virtually nothing. Its largest part tells you the best way to use the reset button.) recommends utilizing the next script to reinitialize PCIe units in Linux techniques if the host boots earlier than the cardboard:

echo "1" > /sys/bus/pci/units/0000:03:00.0/take away # substitue your precise machine ID right here
sleep 2
echo "1" > /sys/bus/pci/rescan

The cardboard is offered to the host working system as a Qualcomm Atheros AR8151 (for the complete verbose output of lspci, test here), so the system you put in it in will want drivers that help it.
I examined Ubuntu 22.04 (nevertheless it says something with kernel 5.15.25 or larger ought to work), pfSense 2.6.0, and the most recent Vyos 1.4 Rolling construct that was accessible on the time (vyos-1.4-rolling-202205210217-amd64.iso).

I tried to load drivers for it into VMware 7.0 Replace 3d, however the group VIB file (net-atl1e-1.0.1.14-1.x86_64.vib) for the Atheros chipset isn’t supported in VMware 6.7+ and newer as a result of VMware deprecated the flexibility to load legacy drivers. It could run in VMware 5.0-6.5, however I’ve not examined it they usually’re sufficiently old that I’m undecided why you’d need to.

In my testing, I used FS.com’s SFP+ DACs and was in a position to get a hyperlink up at 10Gbps to my NAS working an Intel X520-DA2 and throughout the SFP+ ports on my Dell Force10 S4810P. (ethtool output for the interfaces the cardboard offered is here (ens2f0 has the SFP+ module hooked up; ens2f2 and ens2f3 are bridged to the administration interface))

It really works precisely as you’d anticipate, it’s only a Mikrotik router that siphons energy from and may current digital interfaces to its host, there’s not likely any surprises.

I don’t have the gear to construct elaborate topologies to push it to its limits and don’t have {hardware} to check it towards that goes any sooner than 10Gbps, so my efficiency testing was extraordinarily restricted.

See Also

In testing, I used the default MTU of 1500.
{Hardware}-wise I used:

  • HP ProLiant DL20 G9
    • Intel Xeon E3-1240 v6
    • 32 GB DDR4 ECC (2×16)
    • Mikrotik CCR2004-1G-2XS-PCIe
    • Nitrokey HSM 2
  • Customized Dice NAS
    • Supermicro X11SCL-F
    • Intel Celeron G4930 (consider it or not, this really works fantastically for a NAS)
    • 16GB DDR4 ECC (2×8)
    • Intel X520-DA2
  • Dell S4180P
  • FS.com CBL-10GSFP-DAC-1M Copper DACs

Bi-directional iPerf3

I ran a bidirectional iPerf3 check within the default configuration for a period of 10 minutes, and didn’t observe any efficiency or pace degradations throughout that point. Throughout the check the cardboard reported a CPU load of roughly 20% for your entire period. I’ve hooked up the primary ten seconds under, however you may view your entire check here.

Throughout the check the RX charge is considerably decrease than the TX, which I believe could also be as a consequence of a bottleneck from the Celeron-powered NAS.

[  5][TX-C]   0.00-1.00   sec  1.06 GBytes  9.08 Gbits/sec   37   1.01 MBytes
[  7][RX-C]   0.00-1.00   sec   676 MBytes  5.67 Gbits/sec
[  5][TX-C]   1.00-2.00   sec  1.08 GBytes  9.31 Gbits/sec   10   1.37 MBytes
[  7][RX-C]   1.00-2.00   sec   661 MBytes  5.55 Gbits/sec
[  5][TX-C]   2.00-3.00   sec  1.07 GBytes  9.16 Gbits/sec   14   1.43 MBytes
[  7][RX-C]   2.00-3.00   sec   670 MBytes  5.62 Gbits/sec
[  5][TX-C]   3.00-4.00   sec  1.09 GBytes  9.36 Gbits/sec    0   1.46 MBytes
[  7][RX-C]   3.00-4.00   sec   684 MBytes  5.74 Gbits/sec
[  5][TX-C]   4.00-5.00   sec  1.09 GBytes  9.32 Gbits/sec    0   1.48 MBytes
[  7][RX-C]   4.00-5.00   sec   676 MBytes  5.67 Gbits/sec
[  5][TX-C]   5.00-6.00   sec  1.09 GBytes  9.32 Gbits/sec    0   1.49 MBytes
[  7][RX-C]   5.00-6.00   sec   675 MBytes  5.66 Gbits/sec
[  5][TX-C]   6.00-7.00   sec  1.08 GBytes  9.31 Gbits/sec    0   1.51 MBytes
[  7][RX-C]   6.00-7.00   sec   667 MBytes  5.59 Gbits/sec
[  5][TX-C]   7.00-8.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.52 MBytes
[  7][RX-C]   7.00-8.00   sec   688 MBytes  5.77 Gbits/sec
[  5][TX-C]   8.00-9.00   sec  1.08 GBytes  9.29 Gbits/sec    0   1.54 MBytes
[  7][RX-C]   8.00-9.00   sec   671 MBytes  5.63 Gbits/sec
[  5][TX-C]   9.00-10.00  sec  1.09 GBytes  9.34 Gbits/sec   20   1.57 MBytes
[  7][RX-C]   9.00-10.00  sec   676 MBytes  5.67 Gbits/sec

The low RX pace endured in a separate unidirectional check with the NAS TXing and the DL20+Mikrotik RXing, ruling out it being because of the load from the bidirectional check:

[ ID] Interval           Switch     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   505 MBytes  4.24 Gbits/sec   42    617 KBytes
[  5]   1.00-2.00   sec   540 MBytes  4.53 Gbits/sec   62    609 KBytes
[  5]   2.00-3.00   sec   545 MBytes  4.57 Gbits/sec   66    658 KBytes
[  5]   3.00-4.00   sec   545 MBytes  4.57 Gbits/sec   52    609 KBytes
[  5]   4.00-5.00   sec   544 MBytes  4.56 Gbits/sec   79    629 KBytes
[  5]   5.00-6.00   sec   546 MBytes  4.58 Gbits/sec   75    626 KBytes
[  5]   6.00-7.00   sec   546 MBytes  4.58 Gbits/sec   40    683 KBytes
[  5]   7.00-8.00   sec   549 MBytes  4.60 Gbits/sec   63    624 KBytes
[  5]   8.00-9.00   sec   548 MBytes  4.59 Gbits/sec   48    598 KBytes
[  5]   9.00-10.00  sec   542 MBytes  4.55 Gbits/sec   73    508 KBytes

The CCR2004-1G-2XS-PCIe appears to be a stable card to be used in small deployments, particularly single-server co-location setups the place one thing like a devoted router is probably not sensible.
It’s a completely featured router and is cheaper than most of the 25Gbps succesful playing cards which can be accessible new available on the market.
It has no or restricted help in VMware, so in case your infrastructure makes use of VMware it’s solidly a no-go.
Drivers must be accessible in many of the newest releases of most main Linux distros, and it really works with the present releases of Vyos, pfSense and OPNsense (if you wish to put a router in your router).
For the $200 value level Mikrotik has set for it, it’s onerous to go. The most cost effective and most elementary dual-port SFP28 25Gbps card at the moment accessible on CDW, the Supermicro AOC-S25G-b2S, begins at $313 USD, greater than $100 greater than Mikrotik’s CCR2004-1G-2XS-PCIe.
It matches my area of interest completely, and within the subsequent few months I’m going to be deploying my CCR2004-1G-2XS-PCIe in addition to my HP ProLiant DL20 G9 to a co-location facility as a part of one other upcoming mission.

Should you can handle to discover a retailer that carries it (severely, has this been formally launched? virtually nobody has it) and have a use-case that it matches (or simply need a actually low cost 25Gbps NIC and don’t care concerning the routing options), it’s undoubtedly price it.

an addendum has been posted, learn it here

Source Link

What's Your Reaction?
Excited
0
Happy
0
In Love
0
Not Sure
0
Silly
0
View Comments (0)

Leave a Reply

Your email address will not be published.

2022 Blinking Robots.
WordPress by Doejo

Scroll To Top