maanantai 5. kesäkuuta 2017

Human rights and IOT

This week I'm at the EuroDIG conference, discussing policy issues related to the Internet. I will be on a panel focusing human rights and the Internet of Things (IOT).

And that's an interesting topic! At the IETF, we've had plenty of debates about the general topic of whether human rights should be a consideration when designing Internet technology. If you are interested, read the Human Rights Research Group's draft on the topic.

But back the more specific question of IOT. The panel is hosted by the Dynamic Coalition on IOT. This is a group of people who have looked at the role of ethics in IOT systems. I've been an occasional contributor in that group as well, and their document also makes good reading: it covers things like meaningful transparency and user control.

But, to be honest, I'm not the human rights or ethics expert. I know a few things about the tech though. Amidst various IOT discussions I find that it is useful to set of few things straight, so that we at least have a good basis for understanding the technology. And then we can have a more accurate discussion of the ethics or human rights.

Done right, the Internet of Things can bring great benefit and support our societies and human rights: the environment, energy efficiency, quantity and quality of food production, safety and many other things stand to benefit. But it takes effort to ensure that we can enjoy these benefits, and to avoid side effects. And it takes education for all of us to understand how IT is shaping our lives, and how it can be managed and used.

I wanted to highlight four issues:

1. It is not about the gadgets, dammit!
Many IOT discussions focus on the efficiency, security issues or other characteristics of the devices. While that's important, that is far from the full picture. We'd be far better off to consider cloud servers as an even more important component in most systems; that's where the most of the interesting functionality usually resides. And that's what you also want to be under the user's control. 
Similarly, we are often focused on the gadgets and servers, but from my perspective the true value of IOT systems in the data produced or consumed by them. Having user control of the data is very important. How that data is used and by whom is important. It needs to be put to good use by or with the consent of those whose data is being used. It should not be used to violate privacy or in a discriminatory manner.
Also, the architecture of IOT systems as a whole matters a lot. The IRTF Thing-to-Thing Research Group, for instance, is looking at various designs where the devices are talking to each other, rather than (for instance) connecting through a centralised cloud entity. A classic example of where this is the right way to design the system are light control systems; you don't want your ability to turn on lights be dependent on your Internet connection :-) 
 2. Collateral damage
When we talk about security of the IOT systems, we need to understand that security is not merely about protecting the devices or even the data. 
The attacks that caused some common Internet services to fail last year were launched from compromised IOT devices, but the target of the attack was not the devices themselves. It was the other parties, this time the Internet naming infrastructure. (For a discussion of this incident, see the video from IETF-97 technical plenary.) 
The friendly neighbour principle: You cannot design Internet-connected systems without having to consider the effect of your systems to others in the Internet. 
 3. Interoperability
Interoperability is a key issue in creating a large market of useful applications and enabling user control. With more and more Internet-based smart devices, I believe we are on a good path with regards to interoperable devices being able to use the same networks and run over Internet protocols. However, this not enough. We also need applications that are interoperable. Otherwise it will not be possible to plug light switches from one manufacturer to light bulbs from another. 
We also need interoperability for the sake of driving competition, and to ensure that the market supports these systems on a long-term basis regardless of individual manufacturer's decisions. Application level interoperability was discussed in the 2016 IAB workshop on semantic interoperability
 4. Rights of the user
The ability of the user to be in the driver's seat with regards to information concerning him or her is important. I wanted to highlight one additional issue that is important: the right to tinker
This isn't just an issue for hobbyists, it also important for our ability to update products that may be used for decades after they have been manufactured and long after support for them has ceased. I also believe the ability to build new things and modify various consumer systems is important for a healthy, innovative ecosystem.
And as for the opening picture above, that was the message waiting for me this morning on my Inbox. My IOT devices, such as the weight scale, telling how I'm doing. I think the machines have something to learn still from instilling confidence and positive attitude! ūüėÄ Then again, maybe the weight scale would be more efficient, if it slapped me on the face for my failure to have a more healthy diet. Would the positive attitude or the slapping IOT be more ethical?

Jari Arkko

Screenshot (c) 2017 by Jari Arkko. I'd like to acknowledge Ari Keränen, Anna Larmo and Francisco Alcoba for interesting discussions in this problem space.

Have an idea, buy components at midnight

Have an idea, fetch parts at midnight to implement ❤️

Photos (c) 2017 by Jari Arkko

torstai 25. toukokuuta 2017

Access point recommendations?

With some upgrade of my Internet connection, it seems that my trusty WRT-54GL wireless network is now a bottleneck. I would love to get a recommendation for 802.11ac etc access points. I'll be operating them strictly in access point bridged mode, and I'll need several so cost is a factor. But the WRT-54GLs have been spectacularly reliable and did not get confused and need reboots like many other products. That is something that I absolutely need.

lauantai 20. toukokuuta 2017

More disk!

I am in Canada, and have bought once again more hard drives! C-Ordinateurs Canada, the local Fry's equivalent, supplied the goods!

tiistai 16. toukokuuta 2017

If you think about IOT security, think broad enough!

Internet of Things security issues are serious, and are often the focus of discussions. The discussion is much needed. How can we make our IOT devices safe?

How can we prevent attacks similar to those that last year caused many popular Internet services to be unavailable, with badly secured IOT devices being used as a part of the attack?

This is a very important topic.

However, I would like to argue that people often think about this in a too narrow manner. First off, we  have a tendency to focus on visible, concrete things. However, there's more to IOT than the gadgets, and I think the other parts deserve equal scrutiny.

The IOT is not in the gadget, it is in the cloud.

We have to secure the gadgets, but we also have to secure the rest of the system. And more broadly, it benefits the consumers and users to have secure, interoperable, and open solutions for both the gadgets and other parts of the IOT ecosystem. We need data that is in well-specified format, we need data that is under user control, we need systems that you and I can compose from components. But we do not need closed ecosystems.

torstai 11. toukokuuta 2017

Internet and Societies

Today we have an interesting panel discussion organised by ISOC and Chatham House on the effect of the Internet on societies. Is the Internet helping bring societies together, or creating more divisions? With the increased criticism against globalisation, fake news, and the emergence of closed social circles for like minded (and often misguided) fragments of society, it is easy to be worried about this.

But, it is also easy to focus on the most visible issues. When looking at the Internet and societies, one needs to consider the full scope of human interaction, andconsider human, technical and commercial aspects together.

What issues are affecting our ability to connect together? I want to start with five points:

Human interaction is broad, and we need to look at the whole picture. It is easy to focus on the most publicly visible forms of media, and see how the news media for instance has, to put it kindly, become more diverse.

But whole picture is broader and more nuanced, and the concepts of togetherness and divisions may not be so applicable. For instance, the Internet has made it much easier for various smaller groups to connect where they perhaps had no ability to do so before. Communities working on Wikipedia, people with special interests being able to connect, minorities connecting to their culture, and so on.

Human interaction is both about tech and human abilities. It would be a mistake to think about solely technical solutions for problems involving, say news or social media.  Our technical capabilities advance at an incredible speed, but humans are also very good at learning new skills in new environments.

But, clearly critical media reading and communication skills are needed even more in today's world. These topics need to become even more central in our schools and continuing education programs.

Division vs. unification goes beyond people discussions. The Internet continues to be embedded in the fabric of our societies. We need to consider not just the people's discussions, but how well the Internet supports all the other interactions from personal gadgets to managing cities' traffic to running businesses.

Technical and commercial considerations. TCP/IP and the web provide a platform where we have almost universal interconnectivity and lack of technical barriers.

Still, as the IAB's IOT semantic interoperability workshop pointed out last year, interoperability at the level of applications can still be a problem. Can you buy Apple lightbulbs for a house that has Microsoft light switches?

And more broadly, are commonly used Internet services such as social networks erecting borders that restrict efficient connection, for instance due to their deployment patterns as is shown in the image further down?

And, is our increasingly centralised "winner takes it all" Internet economy driving a model where it becomes difficult to switch social network/search/video/mail/application store providers?

Finding broader consensus is hard, but rewarding. As those of us who work in standards or open source realise, finding agreements in broad, diverse communities is hard and time-consuming. Yet, we find the motivation to do so because if we succeed, the benefits are much greater than with everyone running their own things. We've obviously done this not just with technical developments like the Internet, but also to a large extent with our societies, building their infrastructure and rules. And I believe we will continue to be able to do that.

And where does all this leave us? Clearly, there is a lot of work ahead of us. But that work is not merely about the public sphere of news media or social media discussions, it is also about our ability to offer communications tools for all groups, regardless of their size. Our continuing education of the human parts of the system. Our drive to improve standards so that the technology allows connections. Our drive to ensure that the business system provides the possibilities for evolution and connection.

I would also like to point to my other article for a discussion of why IOT security is a much broader topic (inline with the thoughts in this article) than people usually focus on.

What do you think? Leave a comment below! You can also follow our panel discussion online.

Jari Arkko

Acknowledgments: I would like to thank all my friends and colleagues at the Ericsson, IAB, ISOC, and Chatham House for interesting discussions in this problem space.

Picture credits: 1/ Jari Arkko 2/ Evi Nemeth for the original picture, edits by Jari Arkko 3/ World Map of Social Networks from Vincos.It. How divided is this world, even at this level? And I was surprised to find out that there are places in the world where the most popular social media application is LinkedIn :-)

lauantai 29. huhtikuuta 2017

More 10G cards

Received my 2nd 10G Ethernet card, and successfully inserted it to the router. 3rd card is on order...

I've started testing the cards, and can get 9.3 Gbits/s speed! That does feel fast. This number is from iperf. Using SSH to copy files I get a smaller number, however, around 1.2 Gbits/s to 1.8 Gbit/s depending on which crypto is being used. The smaller number is on chacha20, the faster on aes128-ctr. Still investigating what the bottlenecks here are, trying to understand what iperf measures, for instance. Preliminary results seem to indicate that a CPU core is operating at a high load when it is doing encryption for SSH, but that disks are not the source of the delay.

More research needed... but this is already a 12-18 fold increase from my earlier servers who were only able to do about 100 Mbit/s while using SSH. In this case that speed was very clearly due to the CPU being unable to do crypto at a faster speed.

With regards to getting these cards to work, my only complaint is that it is difficult to manage Linux devices when the number of type of interfaces change. The interface names change... and for some reason I don't get accurate information about link status from ethtool, and some of my interfaces seem to not work well with a /etc/network/interfaces-based definition, but rather need explicit commands to be brought up. Odd. Maybe I've misconfigured something, or maybe there's some issue with these specific cards.

Photos (c) 2017 by Jari Arkko

tiistai 18. huhtikuuta 2017

Native IPv6!

I have once again full IPv6 connectivity on ALL of my uplinks, and this time natively! Thanks to my great ADSL ISP, Nebula, who offers IPv6 as standard service for everyone, and also my great mobile network provider, DNA, who do the same. IPv6 life is good in Finland!

Nebula gives you a /56 for your own networks, and a /96 for the router-to-router interface.

Their default sales and support guys understand and know IPv6 well. All you have to do is to ask for the /56 address and it will be given to you.

DNA IPv6 comes on, completely automatically, for every user with a capable device. Which is most devices by now. Both DNA and Nebula have been providing this service for many years.

My previous setup was through a tunnel service, interrupted due to addressing changes, and now gone forever; good riddance :-) Native is the way to go! I'm now fully dual stacked natively for all my networks, be it ADSL or LTE.

A couple of observations:

  • I was so happy to find out that while on my previous setup I had to resort to hacks to do firewalling on IPv6, all functionality is now there for even dynamic connection tracking. Great!
  • Once again, the simplification of my network to offer the bare essential services only has made things like firewall configuration much easier.
  • The router advertisement daemon, RADVD, has a bug on Ubuntu 16.04. The installation scripts do not create the pid directory /var/run/radvd, and this causes the startup scripts to fail. Silently... but you can do "sudo mkdir /var/run/radvd" and everything works after that.

Copyright (c) 2017 by Jari Arkko

lauantai 15. huhtikuuta 2017

New Router

I have finally replaced my trusty old main router with a new one. The old one has been going strong, but it was running on an old Pentium II platform from 1997 that I paid 10€ sometime in the early 2000s... for that investment, it has paid off phenomenally!

Not only that, but the old machine was badly maintained and stopped accepting updates without complete reinstall which I never found time to do. I was running kernel from 2005 for twelve years! Not just bad, this was a security nightmare!

I guess I don't have a lot of attack surface even if you get past the router/firewall, but I have been at least the target of DoS attacks on the router. Here is the old router, with its proud Pentium II CPU:

But, the hardware for the new router arrived earlier this year, and now I had time to set it up properly, and disconnect the old router. While the new router is not the newest gear still either, it has a modern architecture, hopefully better settings and maintenance, and much simplified configuration. The new machine is running an ASUS CS-B motherboard and the Intel Celeron G1850 CPU in a stylish but simple Bitphoenix Phenom micro-ATX case. There's a medium-sized SSD for the machine but no other disks. The machine will not run any other services than forwarding packets, firewalling, and DHCP for the internal network. And my OS is still Ubuntu, but this time version 16, not 4.

At the same time, I've reorganised my entire network around the following principles:
  • Right things in the cloud: Keep as much of the functionality in the cloud as possible. But do not lose control of your own systems or materials. I rent my own space in the cloud and keep file storage in my own servers at home.
  • Just make it fast: Build a fast, general-purpose and simple network that supports any new service that might come up in  the future.
  • Keep it simple: No unnecessary services, no extra complications, no complex architectures.
More specifically, what I have done is this:
  • Move all external-facing web services to the cloud. With one exception, all my websites -- such as -- are now hosted by Linode, and provide TLS certificates via Letsencrypt. I have yet to move, because that is the only domain that handles e-mail, and I haven't found a reasonable, free alternative to hosting that outside our lab server at work.
  • Simplify internal network organisation. I've disabled much of the old hardware and special purpose networks. I won't be needing NAT64 any longer, and I will work with a simpler network that doesn't require the HOMENET automatic routing setup. I will still maintain two special networks, for internal and visitor networks. But I've divided the two networks to use the two redundant uplinks that I have, on ADSL and LTE Advanced. This also allows easy (but manual) switching from one uplink and router to another when something breaks.
  • Turn off dozens of services for which I had no use, or which were only partially functional.
  • Upgrade the internal network to 10G. This is still in progress, as only one of my file servers has the necessary network card. Other cards have been ordered, but I'm still searching for a reasonably priced 10G switch with at least 3 but preferably 8 10GBase-T connectors. Pointers welcome.
  • Employ IPv6 as a means to access individual services from elsewhere in the Internet.
  • Employ smaller number but larger file servers. In my case it is still beneficial to have multiple physically separate devices for safety, but they need to be appropriately dimensioned. I.e., n * 10TB rather than measly 2-4 TB each as previously.

    The primary new file server runs on a similar new computer as the router, but with the MSI A88XM-E45 motherboard and the AMD Athlon x4 760K black edition CPU. This particular CPU unit is by the way a world record holder for the Athlon x4 760Ks; it used to be overclocked up to 7.1 GHz with liquid nitrogen, but it is now enjoying retirement at a more relaxed 3.7 GHz.

  • Employ redundant disk clusters. I've turned on ZFS on my new file server, running currently 2x10TB disks in mirroring mode, so being able to provide 10TB of storage. The really excellent thing with this is that I can add more storage on the go while keeping the same logical disk structure for users, even if I run out of the 10TB. Of course, redundancy within the same case is not sufficient for problems, so in addition to having manual backups I'm also considering hosting backup servers at alternate locations, with automatic network sync.

Not everything is quite up and running yet, in particular I spent five hours last night just getting the router to work. Turned out that the mere existence of a DHCP client package affected network interfaces that had been defined as static ones.

Setting up IPv6 to work with my ADSL connection to Nebula is the next step. The LTE side of the network already has it. There's also a couple of old laptops still running something that I need to figure out what it exactly is :-) as well. One of those laptops also drives the display to the sauna and its broken display... that needs replacement.

Here's the communications closet. The new router, file server, and old computations server are sitting side by side at the far end (this whole space is under a staircase), next to the new small rack that I had built earlier.

Photos (c) 2017 by Jari Arkko

10 Gb Ethernet Cards

Visit the US, buy hardware from Fry's. Wanted to play with 10G Ethernet for my servers. Very expensive in Europe, sometimes significantly cheaper here. This one was 200$. Sadly they had only one. Where could I buy cards cheaply online/Europe?? But funny with the heatsink on a network card.

They also had a 200$ 10G switch but only with 2 10G ports, not so useful for my purpose, even with a extra 1G ports :-) Any suggestions for low-cost switches?

Also, this is 200-400x faster than my Internet connection, so obviously not so useful for external connections. But maybe I can increase the speed of my backup processes from one file server to another :-)

Photo credits (c) 2017 by Jari Arkko

19" Wine rack

A 3U wine bottle storage for your rack system! But does it come with RJ-45?

Photo credits by Canford (the supplier for this product)

AMD Ryzen

Janne is today building the first computer ever bought for him (previous ones were from Olli's scrapyard). AMD Ryzen 7 1700 CPU, Asus Prime X370-PRO motherboard, Define R5 case.

Photo credits (c) 2017 by Jari Arkko

Gaming museum in Vapriikki

Se on moro! I'm spending the winter vacation weekend in Tampere with Janne. First stop the gaming museum in Vapriikki, with ping pong, C64 in real-life settings, and the fall-the-stairs-make-insurance-claim game Stair Dismount.

Photo credits (c) 2017 by Jari Arkko


We are starting to build the new file server today. I'm still missing the intended CPU and a better cooler, but it turns out Olli the overclocker has a box of spareparts...

Photo credits (c) 2017 by Jari Arkko


I am in California for a quick 2 day visit this February, on IETF business. But I'm not so busy that I couldn't stop on the way from the airport to pick up some storage from Fry's :-)

Photo credits (c) 2017 by Jari Arkko