Why Jumbo Frames?

Every data unit on a network has to be assembled by the sender, and its headers have to be read by the network components between the sender and the receiver. The receiver then reads the frame and TCP/IP headers before processing the data. This activity, plus the headers added to frames and packets to get them from sender to receiver, consumes CPU cycles and bandwidth.

Sending data in jumbo frames means fewer frames are sent across the network. This generates improvements in CPU cycles and bandwidth. A single 9k jumbo frame replaces six 1.5k standard frames, producing a net reduction of five frames, with fewer CPU cycles consumed end to end. Further, only one TCP/IP header and Ethernet header is required instead of six, resulting in 290 (5*(40+18)) fewer bytes transmitted over the network.

It takes over 80,000 standard Ethernet frames per second to fill a gigabit Ethernet pipe, consuming a lot of CPU cycles and overhead. Sending the same data with 9k jumbo frames, only 14,000 frames need to be generated, with the reduction in header bytes freeing up 4 Mbps of bandwidth.

These savings in CPU cycles and bandwidth can produce some significant increases in network performance. In a 1999 study on jumbo frames over gigabit Ethernet, "Microsoft, Sun, Compaq, Hewlett-Packard, and IBM all recorded at least 50% increases in TCP throughput with reduced CPU utilization on single- and multi-processor systems using jumbo frames."

Figure 2 is from an often cited Alteon Networks study (Alteon has since been acquired by Nortel) showing the improvements on a gigabit network utilizing jumbo frames with throughput gains and CPU savings of nearly 50%!

Figure 2: Alteon Networks jumbo frames benefits chart

In many of our tests and reviews, we've demonstrated the value of jumbo frames. As I'll show in Table 1, Write and Read throughput to NAS devices can benefit greatly through the use of jumbo frames. Further, as showed in our recent review of unmanaged gigabit Ethernet switches, there can be significant improvements in TCP throughput by using jumbo frames.

What is a Jumbo Frame?

I already stated that a jumbo frame is a frame larger than a standard Ethernet frame, but let's get more specific. As you know, frame is the term for data units at Layer 2 of the OSI model, while packet is the term for data units at Layer 3. Figure 3 is a handy reference on the various layers of the OSI model.

Figure 3: Reference chart for the OSI model (Source: wikipedia.org)

A standard Ethernet frame is comprised of payload produced at Layer 4 and above, an IP header produced at Layer 3, and a data header produced at Layer 2. The payload at Layer 4 is the MSS, or maximum segment size, and is typically 1460 bytes. Add the TCP/IP header of 40 bytes and we have the Layer 3 MTU, or maximum transmission unit of 1500 bytes.

At Layer 2, a frame header is added to the MTU, which is comprised of the source and destination MAC addresses (6 + 6 = 12 bytes), the Ethernet type (2 bytes) and the CRC information (4 bytes), totaling 18 bytes. Many refer to an Ethernet frame as 1518 bytes, which is simply the 1500 byte MTU plus the 18 byte header. The 4 byte CRC information is sometimes not counted, leading to the 1514 byte size. If 802.1q VLAN tagging is in use, an additional 4 bytes are added, bringing the total to 1522 bytes.

So a jumbo frame is basically anything bigger than 1522 bytes, with a common size of 9000 bytes, which is exactly six times the size of a standard Ethernet frame. With Ethernet headers, a 9k byte jumbo frame would be 9014-9022 bytes. This makes it large enough to encapsulate a standard NFS (network file system) data block of 8192 bytes, yet not large enough to exceed the 12,000 byte limit of Ethernet's error checking CRC (cyclic redundancy check) algorithm.

There are other jumbo frame sizes used, but larger sizes don't always lead to better performance, as I'll explain later. We often use 4k jumbo frames on SmallNetBuilder to test NAS devices that support jumbo frames.

Tags: ethernet, gigabit, Tutorial,

Related Articles: