SmallNetBuilder

Confirming the Culprit

To explore the source of the problem, I set up three machines as shown in Figure 1. The gigabit-equipped computers had Intel PRO/1000 MT Desktop PCI NICs and the 100 Mbps machine had a built-in 10/100 Ethernet port that the device manager identified as a 3C920 (3C905C-TX compatible) Integrated adapter.

I used a Linksys SLM2008 8 port gigabit "smart" switch instead of a non-managed switch because the SLM2008 allows Flow Control to be enabled on a per-port basis. This would allow me to explore the effect of disabling flow control in the switch.

Once the three computers were plugged into the switch and correctly auto-negotiated their connections, I used IxChariot to send test traffic. I set up two High Performance Throughput scripts using TCP/IP. The scripts sent 10,000,000 Byte files repeatedly at full speed from one of the gigabit-equipped machines. One stream went to the other gigabit machine, while the other went to the 100 Mbps computer.

Figure 2 shows the result with Flow Control enabled in all NICs and their corresponding switch ports.

Figure 2: Throughput with Flow Control enabled

I delayed the start of the transfer to the 100 Mbps receiver for 5 seconds to test whether the connected, but inactive 100 Mbps receiver would affect the transfer between the gigabit transmitter and receiver. Obviously, it does not. It's only when the transfer starts to the 100 Mbps receiver that the sender throttles back its speed to near 100 Mbps speeds.

Figure 3 shows the desired behavior, i.e. the gigabit receiver operates near gigabit speeds, while the 100 Mbps receiver runs near 100 Mbps. Note that the gigabit pair's speed is limited to below 600 Mbps by the 32 bit PCI gigabit adpaters and the slower bus speeds of the two test machines (one a 2.4 GHz P4, the other an Athlon 3000+)—not due to any Flow Control mechanism.

Figure 3: Throughput with Flow Control disabled

To gain more insight as to exactly where the PAUSE command was coming from, I tried all combinations of enabling and disabling flow control in each NIC and its corresponding switch port. I had thought that disabling flow control in the 100 Mbps NIC or its switch port would prevent the throughput-throttling, since that was the slowest NIC in the group. But I was wrong!

The only way that I could prevent the throughput reduction was to disable Flow Control in either the Transmitting computer's NIC or its corresponding switch port. Disabling either one was enough to produce the desired behavior shown in Figure 3. But the disable worked only for the transmitting NIC.

This seems odd, since the PAUSE command is supposed to be issued by a receiver that can't keep up with data flow. But I repeated my experiments multiple times, tried another managed switch and also an unmanaged gigabit switch, but the results were consistent. I was able to stop the throughput throttle-back only by disabling Flow Control in the transmitting NIC.

Related Items: