6

What is the relationship between a switches buffer size and the effect it has on latency? I am aware of the bufferbloat issue and understand that buffers that are too large in networks of slower BW (i.e. less then 1Gbps) can degrade performance, but how large is too large? I've also read that in networks consisting of 10Gbps and greater links usually fall victim to not enough buffering. Assuming that's true, why is that?

I understand this question may be very subjective to a given workload but any general guidelines or rules of thumb that can provide a foundation would be appreciated.

Ron Maupin
  • 98,218
  • 26
  • 115
  • 191
earnThis
  • 125
  • 2
  • 6

2 Answers2

4

First, there is a relationship between buffer size and latency only if there is congestion in the path (too many incoming packets on the input interface, or too many packets for the switching engine (ASIC/NPU/software/whatever), or too many packets in the output interface). On a "lightly loaded" switch, there is no relationship between buffer size and latency. There is a constant latency that is determined by the performance of the switching engine.

In a device that is experiencing congestion, packets sit in buffers waiting for their turn to get processed. If the waiting room gets full, newcomers cannot be accommodated, i.e. packets get dropped.

The bigger the waiting room is, more the number of packets that can be accommodated, but it gets likelier that some of them are processed really late. For this reason you could loosely say that the bigger the buffers, the more the latency.

But would you consider a dropped packet to have suffered "infinite latency" ? :-)

mere3ortal
  • 2,406
  • 1
  • 9
  • 14
  • So let's take a standard 1RU enterprise switch, is there any right answer to the question of - How large should the shared buffer space be? I can imagine this is highly dependent on the type of traffic, but what does that sizing process look like? – earnThis May 11 '16 at 13:56
  • "But would you consider a dropped packet to have suffered "infinite latency" ? :-)" the thing is that the packets will keep coming until the router drops some of them. Only then will the senders back off. – Peter Green Mar 16 '18 at 17:01
2

Mostly, modern switches don't increase latency; they switch are wire speed, since most of it is done in hardware. Buffering traffic slows it.

Switches tend to have very, very small buffers. The reasoning is that it is better to drop traffic early than to slow it down. Networks prefer that any traffic which will be dropped be dropped as soon as possible (if a switch must buffer traffic, it is likely to be dropped by subsequent network devices) to give a head start to detection and retransmission.

Ron Maupin
  • 98,218
  • 26
  • 115
  • 191
  • How would smaller buffers lead to the dropping of traffic earlier? – earnThis May 10 '16 at 13:51
  • 1
    When buffers fill, traffic gets dropped. If you have a bunch of hosts connected to a switch a 1 Gbps, and your uplink from the switch is 1 Gbps, you may end up filling the buffers on the uplink port quickly, and this would cause traffic to be dropped very shortly after it leaves the host. – Ron Maupin May 10 '16 at 13:53
  • 1
    The point of a switch is to get traffic switched and off the switch as quickly as possible. Mostly, switches do this all in hardware. This allows a switch to perform at wire-speed. – Ron Maupin May 10 '16 at 13:56
  • 2
    So what is the argument *for* larger buffers in switches, for example what Arista is known for? Is that only applicable in a larger data center environment? – earnThis May 10 '16 at 14:26
  • 1
    The Arista deep buffer is designed for environments which need to be lossless. It is a trade-off between wire-speed switching and lossless switching. For most switching applications you design so that there is a ratio between the cumulative access port bandwidth, and the uplink bandwidth. Cisco recommends a 20:1 access to distribution ratio, and a 4:1 distribution to core ratio. – Ron Maupin May 10 '16 at 14:45
  • Thanks, that's helpful. I'd upvote you if I could but I don't have the necessary rep total to do so. – earnThis May 10 '16 at 16:08
  • @RonMaupin You should have those comments in the answer too, so they don't get lost! Very helpful. – NOTjust -- user4304 Mar 03 '21 at 03:17