A perfSONAR User's Group

Per-Packet Provenance Part 2: Non-delivered Packets In nuttcp/iperf3 UDP Tests

As I mentioned in Part 0, the point in this series of posts is to observe what we normally refer to as “packet loss” but rather than simply declaring packets “lost”, we find out what happened to them. Back in November, 2016, I took advantage of some time with a test network that our group had set up to test some equipment and methodology, and I discovered that when one saves the available counters provided by Linux networking and by the ixgbe Intel Ethernet module, one can account for where most packets went.

After running 4,608 nuttcp (v 8.1.4) tests, a few things became apparent:

  • Tests run with 8948 byte payloads had less packet loss than those run with 1500 byte payloads (MTU was set to 9000 for all tests)
  • send side CPU load was higher than receive-side CPU load
  • Almost all packet loss occurred on the receive side, either as Ethernet driver queue drops (ixgbe rx_missed_err) or as UdpInErrors, as reported by the kernel.
  • Apparently:
    • increasing sender txqueuelen causes higher receive side loss
    • increasing receiver Ethernet driver RECEIVE ring buffer reduces rx_missed_err loss
    • increasing udp_mem* kernel knobs can reduce UdpInError loss

The point being that if you’re not exposing these counters and observing how they are affected by your testing, you may be looking in the wrong place when investigating the fate of non-delivered packets.

A second matrix of 4608 tests was run, this time with iperf3 and parameters intended to be identical.

iperf3’s results differed in that:

  • CPU load was generally lower on both sides than with nuttcp
  • there were no Ether driver queue drops, only UdpInErrors


Visualizations test details, and notes are at:



Comments are currently closed.