How to estimate TCP throughput between servers

You have just purchased a server with a dedicated channel which is guaranteed to deliver 1Gbit/s (or more) and are disappointed to find that file transfers are not as fast as you anticipated. Before you write to tech support and look for network problems, try to evaluate the actual TCP throughput from one host to another.

There are two most important factors for the successful transfer of data via TCP:

- Size of the TCP window - is the number of bytes that one side will accept without confirmation;

- Transmission-reception circular delay (latency) - the time it takes to send a packet and confirm its delivery.

- Knowing both of these things will allow you to calculate the maximum possible bandwidth that can be shared between two hosts, regardless of bandwidth.

The formula for calculating TCP bandwidth

TCP throughput (bits/s) = TCP window size (bits) / Roundtrip latency (s)

Let's break it down with a simple example. We have a gigabit Ethernet link between servers with a circular latency of 30 ms. We need to send a large file from one server to another server via FTP. What real bandwidth can we expect?

First you need to convert TCP window size from bytes to bits. In this case, we will use the standard Windows machine TCP window size of 64 KB = 65 536 B = 65 536 * 8 = 524 288 bits.

Then you need to take the TCP window size in bits and divide it by the circular latency of the channel, expressed in seconds. For the purposes of these calculations, 30 ms turns into 0.030 s.

Maximum TCP throughput = 524,288 bits / 0.030 s = 17,476,266 bps = 17.4 Mbps

Although there is a gigabit Ethernet connection between the two servers, you cannot expect more than 17 Mbit/s when transferring files.

How can we make the network faster? The answer is obvious: increase the size of the TCP window and/or reduce the signal delay.

To negotiate a larger TCP window size requires individual manual configuration of each server. This, in turn, leads to the following question: what is the optimal TCP window size? To find this out, it is necessary to do a reverse calculation based on bandwidth.

The formula for calculating the optimal TCP window size

TCP window size (bytes) = TCP window size (bits) / 8 = Bandwidth (bits/s) * Round-trip latency (s) / 8

In the above example for a gigabit Ethernet route between servers with a circular latency of 30 ms, the following value is obtained:

1,000,000,000 bits/s * 0.030 s = 30,000,000 bits / 8 = 3,750,000 bytes.

In other words, if you configure both servers for a TCP window of 3,750 KB, the FTP connection will completely fill the bandwidth and achieve a bandwidth of 1 Gbit/s.

You should be aware that increasing the TCP window size has its disadvantages.

First, it will require more memory to buffer the servers, which is needed to store unconfirmed data in case it is present.

Second, an increased TCP window size may result in more lost packets, which in turn require an entire window to be resent. This can have a negative impact on performance. To solve this problem, the server's TCP/IP stack can activate the "selective acknowledgment" option, which is disabled by default.

In order to solve the problem, WAN Accelerators - global network gas pedals - can be placed at both ends of the line. They can do the following:

- Open an enlarged TCP window;

- provide the ability to finely optimize TCP protocol (eg, selective acknowledgement only between gas pedals);

- do not require special server configuration or additional buffer memory;

- can use special functions of the application layer of the OSI model (Layer 7 - access to network services) to reduce circular delay.

Latency

Want to reduce signal delay? Can it be done in principle? Since we cannot overcome the speed of light, we cannot influence how long it takes a signal to travel a certain distance. The best way to optimize a whole system is to install a WAN-accelerator at each end of the line, which transmits received TCP-packets to the local server, thus "cheating" it on the real data transfer rate.Instead of waiting for a stalled response from the remote server, the local server receives instantaneous acknowledgements from the gas pedal. It saves us from having to adjust a TCP window size on the servers themselves. A pair of WAAS devices uses an increased TCP window size and selective acknowledgements over the entire line segment.

In addition, WAAS effectively purges a TCP stream of redundant data, providing an extremely high level of compression (compression). Each gas pedal remembers previously viewed data. If a duplicate fragment occurs again, it is removed and replaced by a tiny 2-byte tag. This tiny mark is recognized by the remote gas pedal, which inserts the original data fragment in its place before sending traffic to the local server.

The proven result of optimization is higher throughput between servers without any special TCP tuning of the local servers.

Formula for calculating the maximum allowable delay for a given throughput

Example. On a site between two remote servers, you must guarantee 10 Gbps FTP throughput using a standard TCP window size (64KB). What is the maximum signal delay allowed?

Roundtrip latency (ms) = TCP window size (bits) / bandwidth required (bps)

524,288 bits / 10,000,000 bps = 52.4 microseconds

What to do?

Basically, you don't need to increase the TCP window or install WAN boosters. Just use multithreading and you can use the channel at 100% of its bandwidth!