Does physical distance affect download speed?

Question

I just had an argument with a colleague of mine and thought I'd just reach out to the experts on this. Here's the scenario. We were using a website that measures your connection's speed. We tested using a server that is far from us (We are in Malaysia and the server was in US). It was around 2 Mbps. Then we tried with a server in Singapore and it was much faster (around 15 Mbps). My colleague believed it's because of the physical distance while I don't think it matters. My understanding is once you have done the initial handshake and the data flow has started, it doesn't matter where the server is located and the result should be almost the same. Am I missing something here? How does it really work?

Answer 1

My colleague believed it's because of the physical distance while I don't think it matters. My understanding is once you have done the initial handshake and the data flow has started, it doesn't matter where the server is located and the result should be almost the same. Am I missing something here? How does it really work?

Both of you were right at some point in history, but your understanding is mostly correct... today :). There are a few factors that changed between the older answer your friend gave, and the capabilities we have today.

• TCP Window Scaling
• Host buffer tuning

The difference in the results you saw could have been affected by:

• Packet loss
• Parallel TCP transfers

TCP Window Scaling: The bandwidth-delay effect

As your friend mentioned, older implementations of TCP suffered from the limits imposed by the original 16-bit receive window size in the TCP header (ref RFC 793: Section 3.1); RWIN controls how much unacknowledged data can be waiting in a single TCP socket. 16-bit RWIN values limited internet paths with high bandwidth-delay products (and many of today's high bandwidth internet connections would be limited by a 16-bit value).

For high RTT values, it's helpful to have a very large RWIN. For example, if your path RTT from Malaysia to the US is about 200ms, the original TCP RWIN would limit you to 2.6Mbps.

Throughput_max = Rcv_Win / RTT
Throughput_max = 655358 / 0.200*
Throughput_max = 2.6Mbps

RFC 1323 defined several very useful TCP additions which helped overcome these limitations; one of those RFC 1323 TCP Options is "window scaling". It introduces a scale factor, which multiplies the original RWIN value, in order to get the full receive window value; using window scaling options allow a maximum RWIN of 1073725440 bytes. Applying the same calculations:

Throughput_max = Rcv_Win / RTT
Throughput_max = 10737254408 / 0.200*
Throughput_max = 42.96Gbps

Keep in mind that TCP increases RWIN gradually over the duration of a transfer, as long as packet loss isn't a problem. To see really large transfer rates over a high-delay connection, you have to transfer a large file (so TCP has time to increase the window) and packet loss can't be a problem for the connection.

Packet Loss

Several years ago, internet circuits across the Pacific Ocean got pretty congested at times. I used to run video chats overseas during some of these high congestion / packet loss conditions... Even modest packet loss (such as 0.5% loss) can slow a high-bandwidth connection down.

It's also worth saying that not all TCP packet drops are bad... several TCP implementations intentionally drop packets as an endpoint-to-endpoint feedback signal. I mentioned above that TCP gradually opens the RWIN over time (as long as packet loss is not a problem). When a congested circuit drops a packet, TCP responds to the drops by slowing the connection down (and spread across enough TCP sockets, TCP's response helps mitigate circuit congestion). However, this kind of bulk congestion TCP tail drop behavior has some drawbacks... TCP sockets in this condition can significantly throttle back throughput (via CWND - congestion window) when faced with TCP tail drop (from circuit congestion).

Also related to the subject of intentional packet drops is Random Early Detection (RED). This is a great feature to enable in your TCP traffic queues when you enable QoS / HQoS on your packet queues. It works by looking at the interface queue and randomly dropping a packet. The packet drops (on TCP connections) will slow sockets down; and it's widely considered to be a much better alternative than tail-dropping TCP. It's also worth noting that this feature only works on TCP... as you may be aware, many (non-multimedia) UDP protocols do not slow down in the face of packet loss.

Parallel TCP streams

FYI, some speed test websites use parallel TCP streams to increase throughput; this may be affecting the results you see, because parallel TCP streams dramatically increase throughput in case you have some packet-loss in the path. I have seen four parallel TCP streams completely saturate a 5Mbps cable modem that suffered from 1% constant packet loss. Normally 1% loss would lower the throughput of a single TCP stream.

Bonus Material: Host Buffer tuning

Many older OS implementations had sockets with limited buffers; with older OS (like Windows 2000), it didn't matter whether TCP allowed large amounts of data to be in-flight... their socket buffers were not tuned to take advantage of the large RWIN. There was a lot of research done to enable high performance on TCP transfers. Modern operating systems (for this answer, we can call Windows Vista and later "modern") include better buffer allocation mechanisms in their socket buffer implementations.

Answer 2

Short answer: Yes, distance has an effect on single stream bandwidth.

The internet has evolved means of limiting that effect... delayed ACK, window scaling, other protocols :-) But physics still wins in the end. In this case, it's much more likely to be general network congestion over so many hops -- it only takes a single dropped packet to kill a TCP stream.

Answer 3

While there already are excellent answers to this, I'd like to add: no, speed is not necessarily affected by distance and yes, very often speed is affected by distance are both true.

Why is that?

Strongly simplified, the longer the distance, the more "hops" are involved on the way through the Internet. The maximum bandwidth is determined by the slowest hop and concurring traffic. With increasing distance and a somewhat random distribution of hop speeds, the probability for getting slower overall speeds is increasing. Additionally, physics gets in the way and increasing latency may also slow down the link.

But this is not to be taken for granted. Technology allows us to build a planet-encompassing connection of nearly any desired bandwidth. However, bandwidth and distance are enemies and both dramatically increase the cost of the connection, again making it less likely to exist just for the connection you might need right now.

Of course, this is oversimplified but in reality, this situation is what you very often find. And then again you don't when there's a surprisingly fast connection or a distribution proxy just around the corner - but when everything's instant we rarely think about the speed of the Internet...

Answer 4

Yes, here's real word data, not just theory.

Source: Average Download Speed From World Cities, Andrew Martin