Received my 2nd 10G Ethernet card, and successfully inserted it to the router. 3rd card is on order...
I've started testing the cards, and can get 9.3 Gbits/s speed! That does feel fast. This number is from iperf. Using SSH to copy files I get a smaller number, however, around 1.2 Gbits/s to 1.8 Gbit/s depending on which crypto is being used. The smaller number is on chacha20, the faster on aes128-ctr. Still investigating what the bottlenecks here are, trying to understand what iperf measures, for instance. Preliminary results seem to indicate that a CPU core is operating at a high load when it is doing encryption for SSH, but that disks are not the source of the delay.
More research needed... but this is already a 12-18 fold increase from my earlier servers who were only able to do about 100 Mbit/s while using SSH. In this case that speed was very clearly due to the CPU being unable to do crypto at a faster speed.
With regards to getting these cards to work, my only complaint is that it is difficult to manage Linux devices when the number of type of interfaces change. The interface names change... and for some reason I don't get accurate information about link status from ethtool, and some of my interfaces seem to not work well with a /etc/network/interfaces-based definition, but rather need explicit commands to be brought up. Odd. Maybe I've misconfigured something, or maybe there's some issue with these specific cards.
Photos (c) 2017 by Jari Arkko