The test rig
Two identical boxes back to back, one sending and one receiving.
Line rate at every frame size
Each row is an unlimited-rate run held for 12 seconds. L1 is link utilisation, so ~98 Gbit/s L1 is a full 100G link once you account for the preamble, start-frame delimiter and interframe gap that the wire carries but the frame doesn’t.
The small-frame number is the one that matters, because small frames are the hardest thing a NIC does. 138 million packets a second of 68-byte frames is 97% of tagged line rate, from a single process.
68 bytes, not 64, because these frames are VLAN-tagged and a tagged frame can’t be smaller than 68. On an untagged link the smallest frame is 64 bytes and the packet rate is correspondingly a touch higher. See reading the numbers.
It doesn’t take many cores
Queue count is your parallelism budget, and it turns out you don’t need much of it. Same 1518-byte frames, unlimited rate, varying the queue count:
A single queue already does 83 Gbit/s. Two queues nearly saturate 100G. This is what zero-copy native XDP buys you: the NIC reads the frames straight out of the UMEM, so there’s very little per-packet work left for the CPU to do.
The receive side is real too
Sending is only half a test. Running a receiver on the far end, 1518-byte frames at 8 Mpps:The attach is quick on Mellanox
The link bounce that native XDP causes depends entirely on the driver. On thismlx5 NIC it’s barely there:
ixgbe NIC takes 8 to 11 seconds for the same thing. Either way the run clock doesn’t start until the link is back, and reruns reuse the attachment. See how it works.
For reference: 10 Gigabit
Not everyone has a 100G rig. On a 10G Intelixgbe link, Wireblast saturates the pipe at every size too, and the aggregate rate limiter holds steady regardless of queue count. Measured at --pps 1M:
You can reproduce the shape of any of this on a
veth pair with no NIC at all, at whatever rate your CPU manages. See the namespace lab.
Reproducing these numbers
Nothing here is special-cased. The frame-size sweep is just this, once per size:--queues N. The receive side is a plain receiver. Runnable versions are in the examples directory.