TCP, UDP, and Frame Size
A key point regarding jumbo frames is:
Key Point #1: For a large frame to be transmitted intact from end to end, every component on the path must support that frame size.
This means that the switch(es), router(s), and NIC(s) from one end to the other must all support the same size of jumbo frame transmission for a successful jumbo frame communication session.
Jumbo frame communication can occur utilizing either TCP or UDP as the Layer 4 protocol, but in different manners. TCP has a setup process for each flow between sender and receiver where devices exchange their maximum MSS value. The lower of the sender and receiver maximum MSS values determines the MSS used for that flow, and the subsequent MTU and frame size. So, if PC A and PC B are both using standard Fast Ethernet NICs, they will agree on an MSS of 1460 bytes, generating an MTU of 1500 bytes, resulting in a frame size of 1518 bytes.
In the event that both ends agree to jumbo frame transmission, there still needs to be end-to-end support for jumbo frames, meaning all the switches and routers must be jumbo frame enabled. At Layer 2, not all gigabit switches support jumbo frames. Those that do will forward the jumbo frames. Those that don't will drop the frames. This is another key point:
Key Point #2: Switches that don't support jumbo frames will drop jumbo frames.
Fragmentation, which can occur in routers, as I'll discuss below, is a Layer 3 function. This leads us to another key point:
Key Point #3: For a jumbo packet to pass through a router, both the ingress and egress interfaces must support the larger packet size. Otherwise, the packets will be dropped or fragmented.
In TCP/IP packets, there is a DF or "Don't Fragment" bit. This is a TCP function, commonly set on most hosts. Routers will look at the packet, and if the DF bit is set on a packet too large for its interfaces, the router will drop the packet and send an ICMP message back to the sender saying, "fragmentation needed and DF set". The sender will then throttle down its frame size until the frames pass successfully.
Network communication enabling MSS adjustments by the sender is dependent on ICMP messages being transmitted all the way back to the sender. The problem is that many network attacks, such as DOS attacks, utilize ICMP, and for security, many firewalls block ICMP. This results in the ICMP "fragmentation needed and DF set" message not reaching the sender. The sender then gets no information to send its packets at a smaller size, nor does it get a TCP confirmation that its packets were successful. Subsequently, the sender then continuously resends the frame at the same large size, but it never reaches the destination, resulting in a condition known as a "black hole."
When the DF bit isn't set, the packets will get fragmented by the router to the largest size supported by both its ingress and egress interfaces. If any of the fragmented packets are dropped, TCP will enable a retransmit. However, the sender will resend the jumbo frame, not the smaller fragmented packet. This can result in more data being sent over the network, consuming more bandwidth and defeating the value of the jumbo frames.
The process of utilizing the DF bit and ICMP messaging is referred to as PMTUD, or Packet MTU Discovery. This process applies to TCP flows, which provides retransmission. If the jumbo frames are utilizing UDP, there is no retransmission. If the sender sends jumbo frames utilizing UDP across a network that does support jumbo frames, and the receiver doesn't accept jumbo frames, the packets will be dropped. If the sender sends jumbo frames utilizing UDP across a network that doesn't support jumbo frames, the packets will either be dropped at Layer 2 or fragmented at Layer 3.
To summarize, there are many factors working against jumbo frame transmissions. Every device from end to end must support jumbo frames of the same frame size. The maximum frame size in a transmission is determined by the smallest maximum frame size supported end to end. Security restrictions on a network may cause black hole occurrences. Dropped packets will degrade connections. Finally, fragmentation and dropped packets can result in excessive retransmissions of data, defeating throughput gains.
Hardware and Network Traffic
One of the early drivers for using jumbo frames was limited CPU speed. As I noted earlier, it takes over 80,000 standard Ethernet frames per second to fill a gigabit Ethernet pipe, which is a lot of CPU cycles. PCs with slower CPUs and bus speeds could not generate and pump enough Ethernet frames to fully utilize gigabit Ethernet's bandwidth. So network performance gains of 50% (!) were realized with jumbo frames on PCs with slower CPUs and bus speeds than the PCs in use today.
Today's CPUs and bus speeds are faster and can handle more instructions and frames per second than those used to generate the chart in Figure 2. So, ironically, today's higher CPU and bus speeds have made utilizing smaller frames over gigabit Ethernet less of a problem for PCs. NAS devices, on the other hand, have slower CPUs and bus speeds, making NAS data transfers a main beneficiary of using jumbo frames.
This brings us to the more important consideration for determining whether jumbo frames can help improve your LAN's performance: your network and the type of traffic it carries. If the bulk of your network activity is Internet and email traffic, jumbo frames are of little value, as you don't have a gigabit Ethernet pipe to the Internet. And if you have a high amount of latency sensitive traffic, such as voice, jumbo frames can be counterproductive, as they can add delay in filling the packets.
The key value of jumbo frames comes with large file transfers. As demonstrated in our NAS charts, many devices have significant throughput gains when utilizing jumbo frames. I looked at three different NAS devices we've reviewed that have gigabit interfaces and support jumbo frames, including the Synology Cube Station (CS-407), the Linksys gigabit Network Storage System (NSS4000), and the Buffalo Technology LinkStation Pro (LS-250GL). As you can see from Table 1, both Write and Read performance is improved by utilizing jumbo frames.
|1G Average Write Throughput (MB/s)||12.4||17.07||16.13|
|1G Average Write w/4k Jumbo Frame Throughput (MB/s)||13.55||19.15||19.63|
|1G Average Read Throughput (MB/s)||15.33||20.5||24.38|
|1G Average Read w/4k Jumbo Frame Throughput (MB/s)||15.5||26.25||31.88|