lauantai 29. huhtikuuta 2017

More 10G cards

Received my 2nd 10G Ethernet card, and successfully inserted it to the router. 3rd card is on order...

I've started testing the cards, and can get 9.3 Gbits/s speed! That does feel fast. This number is from iperf. Using SSH to copy files I get a smaller number, however, around 1.2 Gbits/s to 1.8 Gbit/s depending on which crypto is being used. The smaller number is on chacha20, the faster on aes128-ctr. Still investigating what the bottlenecks here are, trying to understand what iperf measures, for instance. Preliminary results seem to indicate that a CPU core is operating at a high load when it is doing encryption for SSH, but that disks are not the source of the delay.

More research needed... but this is already a 12-18 fold increase from my earlier servers who were only able to do about 100 Mbit/s while using SSH. In this case that speed was very clearly due to the CPU being unable to do crypto at a faster speed.

With regards to getting these cards to work, my only complaint is that it is difficult to manage Linux devices when the number of type of interfaces change. The interface names change... and for some reason I don't get accurate information about link status from ethtool, and some of my interfaces seem to not work well with a /etc/network/interfaces-based definition, but rather need explicit commands to be brought up. Odd. Maybe I've misconfigured something, or maybe there's some issue with these specific cards.

Photos (c) 2017 by Jari Arkko

tiistai 18. huhtikuuta 2017

Native IPv6!

I have once again full IPv6 connectivity on ALL of my uplinks, and this time natively! Thanks to my great ADSL ISP, Nebula, who offers IPv6 as standard service for everyone, and also my great mobile network provider, DNA, who do the same. IPv6 life is good in Finland!

Nebula gives you a /56 for your own networks, and a /96 for the router-to-router interface.

Their default sales and support guys understand and know IPv6 well. All you have to do is to ask for the /56 address and it will be given to you.

DNA IPv6 comes on, completely automatically, for every user with a capable device. Which is most devices by now. Both DNA and Nebula have been providing this service for many years.

My previous setup was through a tunnel service, interrupted due to addressing changes, and now gone forever; good riddance :-) Native is the way to go! I'm now fully dual stacked natively for all my networks, be it ADSL or LTE.

A couple of observations:

  • I was so happy to find out that while on my previous setup I had to resort to hacks to do firewalling on IPv6, all functionality is now there for even dynamic connection tracking. Great!
  • Once again, the simplification of my network to offer the bare essential services only has made things like firewall configuration much easier.
  • The router advertisement daemon, RADVD, has a bug on Ubuntu 16.04. The installation scripts do not create the pid directory /var/run/radvd, and this causes the startup scripts to fail. Silently... but you can do "sudo mkdir /var/run/radvd" and everything works after that.

Copyright (c) 2017 by Jari Arkko

lauantai 15. huhtikuuta 2017

New Router

I have finally replaced my trusty old main router with a new one. The old one has been going strong, but it was running on an old Pentium II platform from 1997 that I paid 10€ sometime in the early 2000s... for that investment, it has paid off phenomenally!

Not only that, but the old machine was badly maintained and stopped accepting updates without complete reinstall which I never found time to do. I was running kernel from 2005 for twelve years! Not just bad, this was a security nightmare!

I guess I don't have a lot of attack surface even if you get past the router/firewall, but I have been at least the target of DoS attacks on the router. Here is the old router, with its proud Pentium II CPU:

But, the hardware for the new router arrived earlier this year, and now I had time to set it up properly, and disconnect the old router. While the new router is not the newest gear still either, it has a modern architecture, hopefully better settings and maintenance, and much simplified configuration. The new machine is running an ASUS CS-B motherboard and the Intel Celeron G1850 CPU in a stylish but simple Bitphoenix Phenom micro-ATX case. There's a medium-sized SSD for the machine but no other disks. The machine will not run any other services than forwarding packets, firewalling, and DHCP for the internal network. And my OS is still Ubuntu, but this time version 16, not 4.

At the same time, I've reorganised my entire network around the following principles:
  • Right things in the cloud: Keep as much of the functionality in the cloud as possible. But do not lose control of your own systems or materials. I rent my own space in the cloud and keep file storage in my own servers at home.
  • Just make it fast: Build a fast, general-purpose and simple network that supports any new service that might come up in  the future.
  • Keep it simple: No unnecessary services, no extra complications, no complex architectures.
More specifically, what I have done is this:
  • Move all external-facing web services to the cloud. With one exception, all my websites -- such as -- are now hosted by Linode, and provide TLS certificates via Letsencrypt. I have yet to move, because that is the only domain that handles e-mail, and I haven't found a reasonable, free alternative to hosting that outside our lab server at work.
  • Simplify internal network organisation. I've disabled much of the old hardware and special purpose networks. I won't be needing NAT64 any longer, and I will work with a simpler network that doesn't require the HOMENET automatic routing setup. I will still maintain two special networks, for internal and visitor networks. But I've divided the two networks to use the two redundant uplinks that I have, on ADSL and LTE Advanced. This also allows easy (but manual) switching from one uplink and router to another when something breaks.
  • Turn off dozens of services for which I had no use, or which were only partially functional.
  • Upgrade the internal network to 10G. This is still in progress, as only one of my file servers has the necessary network card. Other cards have been ordered, but I'm still searching for a reasonably priced 10G switch with at least 3 but preferably 8 10GBase-T connectors. Pointers welcome.
  • Employ IPv6 as a means to access individual services from elsewhere in the Internet.
  • Employ smaller number but larger file servers. In my case it is still beneficial to have multiple physically separate devices for safety, but they need to be appropriately dimensioned. I.e., n * 10TB rather than measly 2-4 TB each as previously.

    The primary new file server runs on a similar new computer as the router, but with the MSI A88XM-E45 motherboard and the AMD Athlon x4 760K black edition CPU. This particular CPU unit is by the way a world record holder for the Athlon x4 760Ks; it used to be overclocked up to 7.1 GHz with liquid nitrogen, but it is now enjoying retirement at a more relaxed 3.7 GHz.

  • Employ redundant disk clusters. I've turned on ZFS on my new file server, running currently 2x10TB disks in mirroring mode, so being able to provide 10TB of storage. The really excellent thing with this is that I can add more storage on the go while keeping the same logical disk structure for users, even if I run out of the 10TB. Of course, redundancy within the same case is not sufficient for problems, so in addition to having manual backups I'm also considering hosting backup servers at alternate locations, with automatic network sync.

Not everything is quite up and running yet, in particular I spent five hours last night just getting the router to work. Turned out that the mere existence of a DHCP client package affected network interfaces that had been defined as static ones.

Setting up IPv6 to work with my ADSL connection to Nebula is the next step. The LTE side of the network already has it. There's also a couple of old laptops still running something that I need to figure out what it exactly is :-) as well. One of those laptops also drives the display to the sauna and its broken display... that needs replacement.

Here's the communications closet. The new router, file server, and old computations server are sitting side by side at the far end (this whole space is under a staircase), next to the new small rack that I had built earlier.

Photos (c) 2017 by Jari Arkko

10 Gb Ethernet Cards

Visit the US, buy hardware from Fry's. Wanted to play with 10G Ethernet for my servers. Very expensive in Europe, sometimes significantly cheaper here. This one was 200$. Sadly they had only one. Where could I buy cards cheaply online/Europe?? But funny with the heatsink on a network card.

They also had a 200$ 10G switch but only with 2 10G ports, not so useful for my purpose, even with a extra 1G ports :-) Any suggestions for low-cost switches?

Also, this is 200-400x faster than my Internet connection, so obviously not so useful for external connections. But maybe I can increase the speed of my backup processes from one file server to another :-)

Photo credits (c) 2017 by Jari Arkko

19" Wine rack

A 3U wine bottle storage for your rack system! But does it come with RJ-45?

Photo credits by Canford (the supplier for this product)

AMD Ryzen

Janne is today building the first computer ever bought for him (previous ones were from Olli's scrapyard). AMD Ryzen 7 1700 CPU, Asus Prime X370-PRO motherboard, Define R5 case.

Photo credits (c) 2017 by Jari Arkko

Gaming museum in Vapriikki

Se on moro! I'm spending the winter vacation weekend in Tampere with Janne. First stop the gaming museum in Vapriikki, with ping pong, C64 in real-life settings, and the fall-the-stairs-make-insurance-claim game Stair Dismount.

Photo credits (c) 2017 by Jari Arkko


We are starting to build the new file server today. I'm still missing the intended CPU and a better cooler, but it turns out Olli the overclocker has a box of spareparts...

Photo credits (c) 2017 by Jari Arkko


I am in California for a quick 2 day visit this February, on IETF business. But I'm not so busy that I couldn't stop on the way from the airport to pick up some storage from Fry's :-)

Photo credits (c) 2017 by Jari Arkko