10 Lessons-Learned About Small/Cheap Nodes As perfSONAR Sensors
(Or how I learned to stop worrying and learned to love the SWARM)
This is a brain-dump of some elements of our project to deploy small, cheap devices on our campus network, funded by the NSF CC-NIE Program, with work performed in 2014-2015.
Since that project associated me with the idea of “deploying a bunch of Raspberries Pi as perfSONAR devices” in the research and education community, I find myself writing the same email over and over again as people ask for notes/ideas. Hopefully, this will begin to address the lack of an on-line write-up resource, and help others get started or avoid pitfalls. As of this writing, I’m sorry but you have to read the whole article to get all the goodness, there is no quick start. That’s not necessarily an omission, since one needs to follow the narrative to conceptualize the the task at hand.
Video of the eldest version of a talk, which I have updated and repeated a couple of times since, on this topic is archived by Internet2 at:
http://meetings.internet2.edu/2015-global-summit/detail/10003726/
I start at about 56:41, but you should watch everybody, if you’re thinking about small-node-pS-deployment in general.
Also note that since my initial deployments in 2014, the perfSONAR developers have made installations on any node easier and more consistent, with the release of alternative install options, like the Debian bundles.
(As far as I have seen, there aren’t really worthwhile non-Debian-derived OS distros for my ARM devices. One could cite exceptions like Pidora, but nothing can compete with the active community behind Raspbian or ARMhf Debian, and the value of going all RHEL-derived on Raspberries and Beagles is minimal at any rate. I keep encountering folks who want to know how to do CentOS on BeagleBones, perhaps because they don’t want to add Debian-ness to their repertoire (?), but I lost interest in keeping everything all-one-flavor sometime in the early 90’s, when I was called upon to deal interchangeably with SunOS, Solaris, IRIX, OSF, UItrix, UnixWare, AIX and XENIX*, so such pleas for mercy have no effect on me, really. When the game became BSD variants versus Linux variants, I didn’t bat an eye. Take solace, however, in the fact that becoming multi-distro will broaden your horizons and make you a higher-quality person in general.)
(* no, never SCO Unix, really. I may have seen it once, somewhere.)
I tend to refer to the dense perfSONAR sensor deployment at the University Of Hawaii, Manoa as “the SWARM”, which is an antenym (an as-yet unrealized retronym) that seems oddly appropriate. “The SWARM” is not mentioned in the original project proposal. The larger, central perfSONAR measurement archives that schedule tests and archive results are named “hive01” and “hive02”. The whole project as deployed during the project is now being referred to as “gen 1”, and is based on an ad-hoc architecture where sensor nodes ran owamp/bwctl daemons. The formulative “gen 2” is to-be-deployed, based on perfSONAR 3.5 testpoint bundle, and the gen 1 lessons mentioned below, and of course, a further update will probably be needed to integrate a future SWARM into the pS 4.0 vision for the future.
A follow-up post will specify how to initialize a BeagleBone Black as a sensor node, with lessons-learned incorporated.
In 2013, around the same time that the NSF solicited campus infrastructure proposals, the Raspberry Pi was approaching 1 year old, and BeagleBoard (Texas Instruments, Digi-Key, Newark element14) had released Beaglebone, a small format version of their earlier ARM-based SBC. This “ARMs race” continued, with many other ARM based or ARM-derived chipsets offered from vendors as small-format Linux platforms, for experimentation and instruction.
See also: PerfClub.Org: Little Boxes In The Internet Monitoring Biz
In R&E networking circles, the discussion which seemed to be occurring everywhere at once, was about whether/how to exploit those inexpensive, little devices to do network monitoring. In retrospect, it probably had been a discussion for a decade or more, but the real watershed that came from Raspberry Pi, Beaglebone, and the like wasn’t that they existed, it was that they were available through retail channels, and in quantity < 1000. Previously, the relevant offerings in small devices required a direct order from manufacturers in much greater quantities, or if a device was available in small lots, it was only available in very small lots, such as “sold out: click here to be on our mailing list”.
Being The 1st Lesson-Learned: The Supply Chains Do Not Necessarily Intend To Support Your Deployment Model
Of course, the scarcity mentioned above plagued the Raspberry Pi and BeagleBone releases as well. For a certain portion of their market presence each of the ARM-based, IoT-promoting devices has been available in limited quantities. In 2014, when we began deploying the first small nodes on our campus networks, we decided that Beaglebone was a better choice than Raspberry Pi for network monitoring, due to the internal connectivity of the Ethernet interface, among other details. When we went to purchase, however, BeagleBone’s release of Rev C had created an absence of stock, and even when stock was available, limited quantities per order. We bought 43 Model A.1 Raspberry Pis instead. This wasn’t as much of a setback as it might seem, since we were engaged in a process to evaluate small measurement nodes in general, and evaluating our 2nd choice was significantly more informative than simply rationalizing about it.
In deploying the first-year batch, we installed 10 BeagleBone Blacks with Ubuntu 13 and 43 Raspberries Pi with Rasbian 7. A lot of attention was given to minimizing the man-hours required to prepare a node, and in how to deploy changes to deployed nodes, en masse.
Being The 2nd Lesson-Learned: Look Into Saving Work Per Deployment, But It’s Not A Savings When The Work Saving Thing Takes As Long As Doing It The Hard Way
In order to get the initial BeagleBone Rev Bs deployed, we used a microSD card which, when used as a boot image in the BBB, wrote an image to the internal MMC flash, and then used the on-board LEDs to signal completion. This is very close to what the currently available OS images can do for themselves.
In the case of Raspberry Pi, the solution was rasbian-ua-netinst, which allowed SD cards to be prepared in 9 seconds, versus 120 seconds for a full Raspian image, and then allowing each node to install itself, once supplied with power, network, and the SD card.
See also: Reducing Raspbian Install To Minimum Therbligs With raspbian-ua-netinst
The value of raspbian-ua-netinst, which required some fiddling and stop-gapping to use effectively, was clear with 43 nodes to deploy, but less clear afterward for single node refreshes. The PiNet Team with its focus on multiple nodes for programming and STEM teaching, has described other methods of initializing/operating many Raspberries Pi at once (Their clean net-booting might be very useful in a pS sensors in a campus environment).
Now, 2.5 years after getting ua-netinstall to work for me, the process of updating the process would be, essentially, starting from scratch, so without another several-dozen-node install, I would simply write full Raspbian-Lite SD cards while multitasking something else.
Being The 3rd Lesson-Learned: The Real Issue with 100 Mbit Ethernet Isn’t About Throughput
If any criterion makes little ARM devices the Rodney Dangerfields of the perfSONAR sensor world, it’s that they have 10/100 Ethernet, which many people don’t see as enough. The first argument that comes to mind is that most of your throughput issues that show up at 1 GbE are also going to manifest with a 100 Mbps throughput test, but the truth is that throughput tests in general haven’t really played much of a role in gen 1 SWARM testing. The problem with 100 Mb Ethernet that did show up in early deployment was with putting the 100 MbE nodes on a tier 2 aggregation switch, where there were no 10/100/1000 ports, or in fact nothing less than 1 Gigabit. If you have Gigabit Ethernet on your small device, then media converters and UTP SFPs come into play, but connecting 100 MbE can be a non-starter. This is a(nother) reason why the Raspberry Pis and Beaglebones end up at the edge, where the UTP ports are.
Being The 4th Lesson-Learned: For More Than A Couple Of Nodes, Config Management Is Worth The Time To Understand And Incorporate
With deployment of the first 10 BeagleBones in late 2013, and the 43 Raspberry Pis in July 2014, the first trial-by-fire of bulk deployment came in the form of the heartbleed OpenSSL bug in April 2014, followed soon after by the shellshock bash bug in September 2014. The need to push changes out to existing nodes was spotlighted. Thankfully, PerfClub participant Josh Hoblitt had a solid grounding in config management, specifically Puppet, originally born of his need to administer nodes in Chile from his office in Tucson. Due to Josh’s input we were already doing Puppet management before the above bugs (as well as the more recent faire, like the glibc “GHOST” bug), and updates weren’t an issue.
With a Puppet infrastructure set up, the install of each node can be reduced to:
- install OS
- register it with Puppet Server
And then design Puppet tasks to make each install conform to the current environment. Since I had installed Puppet clients on the deployed nodes and registered them with the server, updating OpenSSL, Bash, later GlibC, was a matter of making sure each node was party to a rule like “require <package> >= version <N>”.
Being a non-Ruby-person, much of my Puppet configuration has been something other than formally “Puppetish”. For example, after messing about with creating Debian packages, I ended up deploying bwctl, owping, powstream, owampd, nuttcp binaries to each architecture by compiling the programs on a single node and then pushing the binary files out to deployed nodes (there is now a better way, see pS Bundles,). The point of which is, you need not worry about doing things the Puppet/Ruby-verse way, you only need do them in a way-that-works way.
The effective way to get a Puppet Server going, for my part, was to simply install The Foreman, using The Foreman Installer, which installs everything you need to have a working Puppet Server which provides The Foreman as a GUI (among other things).
It may be that our next generation adopts one of the many IoT Frameworks that exist, and that the role of Puppet is thus reduced or eliminated, but once you get a Puppet server operational, you may find that you want to register all of your servers and other miscellaneous boxes. In addition to the configuration management aspects of Puppet, the Foreman interface provides views that assist in per-node health monitoring, as well as inventory and other benefits.
Being The 5th Lesson-Learned: CDP or LLDP Can Be Extremely Useful To Help Minimal-Config Nodes Identify Themselves
Another challenge of using many identically-configured nodes is remembering which is which, and tying the test results to the relevant path, without static configuration identities. Both RPi and BBB are delivered with no serial numbers, nor MAC addresses visible on the device, nor any individual markings at all. I have done a great deal of serial numbering and MAC labeling, which helps, but the inventory control and life cycle tracking can be difficult.
The nodes in SWARM gen 1 get IP addresses from DHCP/SLAAC, so it is necessary to find a means of associating their test results with their position on the network. Obvious ways of doing this would include using DNS or IPAM, and there’s nothing wrong with those ideas, except that they each come with a list of tasks to initialize and maintain. The node-self-identification data source I finally employed was eavesdropping on the Cisco Discovery Protocol or Link Layer Discovery Protocol from the neighboring Ethernet switch. Besides providing a stable and accurate location source (even when switch locations are not filled in, our network’s switch names are informative), CDP also provides the name of the port the device is connected to, the interface description string, and the management IP of the switch. Each node runs a short Perl script once a day which captures CDP packets on the Ethernet interface and then packages the information in a Facter (subsidiary of Puppet) fact, which is then pushed up to the Puppet server on the next update. After examining such software as lldpd and lcdpd, the Perl script was chosen on the basis of simplicity. The invocation of tcpdump used to capture CDP packets was taken from:
http://www.kehlet.cx/articles/186.html
The set up of exporting CDP port information in Facter facts required an en masse update of Puppet, to get facter to version 1.6 (?), where custom facts were introduced as a feature. In order to update Puppet, it was not possible to use Puppet, so an alternative script method was used, centered around the use of sshpass and Perl scripts. The sshpass tool offers a solution to the post-install Puppet registration task as well, and has proven useful to apply a list of initialization tasks to each freshly-installed node. One interesting aspect of the minimal post-install tweaks is that prior to registering each Puppet agent with the Puppet server, it is necessary to provide good NTP configuration to the agent device, or otherwise the Puppet registration certificate installation will fail if the agent and the server don’t agree on time of day. After filtering due to the leveraged DDoS attacks involving NTP in early 2014, the NTP servers provided by the OS install may or may not be reachable.
Being The 6th Lesson Learned: Not All SD Cards Are Cut Out To Be Linux File Systems
As I said above, one of the aspects of our CC-NIE project was validating types of hardware and the capabilities of nodes, including the SD cards that serve as their on-board storage. As part of that pursuit, I ordered a variety of SD cards to use as Raspberry Pi storage volumes. One brand/model of 4 gigabyte card had a 100% failure rate within 6 weeks of deployment across 20 cards. Most others fared better, and in general cards which are:
- higher performance (class 10 or higher)
- larger capacity (>= 8 GB)
- not counterfeits (see this)
tend to survive. I have Kingston 8 GB class 10 and SanDisk 8 GB class 10 that have been in service for over 31 months, with no confirmed failures, although there have been file system corruption events not associated with card hardware failures.
No internal MMC flash storage failure (Beaglebone) has been observed.
Being the 7th Lesson Learned: ARM Devices Are Hardy Little Beasts
Dead devices are fairly rare, and usually occur as a result of power outages, or (lightning strikes). Out of 48 Raspberry Pi A, 5 Raspberry Pi 2, 2 Beaglebone “White”, 10 Beaglebone Black rev B, and 50 Beaglebone Black rev C, only one BBB rev B, and 2 Raspberry Pi 1s appear to be non functional. (At my house, during tropical storm Darby, a nearby lightning strike killed several Ethernet switches, a DOCSIS modem, a garage door opener, and one Raspberry Pi. Several other Raspberries survived unscathed.)
Being The 8th Lesson Learned: The Primary Operational -> Service Interruption -> Security Issue Is Syslog Filling Up The Storage Volume (a.k.a. There Is Nothing New Under The Sun)
The only operational issue which has resulted in non-functional nodes is the filling of the root storage volume by syslog files. If left untreated, this problem can cause a security issue by causing iptables filters to fail on reboot. Aggressive log size controls need to be asserted. If logs are kept, the contents will contain a certain amount of “darknet” information, but if the logs are not being examined, choosing not to run a syslog daemon, or log to /dev/null, is perfectly reasonable. It pays to tweak logrotate and to watchdog devices to make sure that they prioritize staying alive above keeping logs.
Being The 9th Lesson Learned: Security Compromises Happen When You Don’t Follow Protocol
A Beaglebone Black rev C which was active on the network became a spam reflector bot, and was replaced with a freshly installed device, which also became a spam reflector bot, due to hasty replacement which caused the device to reboot without the iptables filters active. You should be working from a duly-certified deployment image, and a checklist. All of your Puppet rules don’t do you any good if you don’t tell Puppet to put the device into a group and apply policies. Once a decent iptables filter is applied to every device, not much occurs that isn’t part of the test program. A general policy for SWARM gen 1 devices has been that they do not need to hear from the Internet, so filters can be very restrictive, allowing only known interactions, including package updates per node. But when the iptables filters aren’t installed and updated, each node is an instance of all its installed vulnerabilities, exposed to the Internet. A focus for SWARM gen 2 is to exploit the benefits of general IoT practice, including with the walled-garden approach of app virtualization (like Docker), and agressive OS minimization, to get rid of irrelevant packages.
Being The PENTULTIMATE LESSON-LEARNED: Small, 100 Mbit Ethernet, ARM Nodes Can Do Worthwhile Things.
More than once, after presenting at length in public, I have had people walk up to me and make declarations resembling “you’ll never tell me that these little devices make useful network monitors”, despite the fact that I had just finished telling them that. People are seemingly obsessed with getting at least Gigabit Ethernet on all the deployed nodes. Also I’ve been brushed aside by people who declare that since choices like Liva X, or GigaByte Brix provide Intel architecture and the possible installation of “real” perfSONAR for “only” $100 more than BeagleBone Black or Raspberry Pi, that the relevance of cheap ARM nodes was pretty much negligible.
I would say that cheap ARM devices are still relevant, and that the $50 price point, which enables the deployment of 50 nodes for $2500 hardware cost, versus $7500 for the Intel devices, is important for what I call dense deployment. Dense deployment is putting a node everywhere your network goes, so that the nodes have the same perspective as your network users do. The SWARM gen 1 showed us loss patterns, on our network, and we developed loss visualization that readily distinguished competitive traffic congestion from other types of loss.
This view of loss in our network led to multiple fixes, such as misconfigured CoS queues, bad transceivers, routing problems, etc. The tests were run from a central perfSONAR Toolkit node and measurement archive, with traceroute hop info filled in from NetDot data.
The loss view, with topology from traceroutes, was really the big pay-off as far as perfSONAR-like testing. Throughput tests were run at the beginning, but they don’t really point to problems like the long-term loss visualization.
The availability of bwctl services for traceroute and ping can be invaluable. There are also times when shell access is useful, as a means of visiting the network user’s perspective and checking network “health”, including vital services like DNS, dhcp, IPv6 neighbor discovery, etc.
If you’re installing <= 5 hosts at network aggregation points, then the expense and man hours of installing perfSONAR Toolkit are perhaps warranted.
A generally un-sung advantage of small ARM-based nodes is their exposure of general purpose I/O pins, with such features as I2C bus functionality, which enable the placement of environmental sensors, for temperature, humidity, for a few dollars per node (or perhaps something more exotic for the connoisseur of discerning taste).
If you’re obsessed with Gigabit Ethernet, then have a look at ODroid C2, which is an ARM platform with GigE and a $70 price point.
The utility of cheap ARM-based devices, deployed widely, if properly managed, is undeniable.
Per-Packet Provenance Part 2: Non-delivered Packets In nuttcp/iperf3 UDP Tests PerfClub Call Sept 2020 – Performance Co-Pilot
Comments are currently closed.