How We Tested
11ax is even more complex than 802.11ac and is still undergoing rapid change as bugs are found, tracked down and fixed. Both ASUS and NETGEAR said new firmware releases were coming soon and hinted at my holding off publishing results until I had those loaded up. But there's going to be (or should be) a steady stream of releases over the next few months, so we have to start somewhere.
The goal of these tests is to assess the performance seen by a two-stream AC (aka Wi-Fi 5) device. I didn't run any tests to see if the products could reach the maximum advertised 5 GHz 4804 Mbps link rates. Those numbers are primarily for marketing bragging rights and won't be seen in normal use, even with two routers in a bridge situation. Achieving any practical benefit from such a bridge would also require multiple traffic sources and sinks. octoScope Pal-24 and Pal-5 partner devices were used as the test STAs. Both have Qualcomm QCA9984-based radios and support maximum 256 QAM modulation.
As is our practice, only released firmware was used for testing, 188.8.131.52.384_5247-g499251e for the ASUS and 184.108.40.206_1.0.20 for the NETGEAR.
HE frames were disabled on both products because they aren't used by 11ac devices and are just another potential source of problems at this point. For each product, the 2.4 GHz radio was set to Channel 6 and whichever setting produced a 40 MHz wide channel. For 5 GHz, Channel 40 was set as primary and whichever settings produced a 160 MHz wide channel.
Each router was centered on the turntable in the main octoBox 38 test chamber, as shown below (yes, I know that photo is from the AC88U review). Unlike our Version 10 router process, the router was not rotated during the test. A single iperf3 TCP connection with unlimited bandwidth was used to generate traffic.
The only benchmarks run were Rate vs. Range (RvR). This is essentially the same test as the "Profile" test found in the Router Charts; I'm finally adopting more standard terminology. This test shows how throughput (rate) changes as signal level is varied (range). Instead of using physical distance to vary the signal by walking a laptop around my home, the router is placed in an RF-tight anechoic chamber (the octoBox) and programmable RF attenuators are used to change the signal. The diagram below from the Version 10 Wireless Test Process description shows the setup.
RvR test setup
For 2.4 GHz, the attenuation was ramped from 0 to 63 dB in 3 dB steps; for 5 GHz, the range was 0 to 45 dB in 3 dB steps. Each test ran traffic for 35 seconds with the first five seconds discarded from the test average. The test sequence was:
- 5 GHz @ 80 MHz downlink
- 5 GHz @ 80 MHz uplink
- 5 GHz @ 160 MHz downlink
- 5 GHz @ 160 MHz uplink
- 2.4 GHz @ 20 MHz downlink
- 2.4 GHz @ 20 MHz uplink
ASUS RT-AC88U in test chamber
When testing 2.4 GHz, the Pal STA bandwidth was set to 20 MHz, as is our practice. This reflects a more typical view of performance in this band due to the crowded airspace most users enounter.
For 5 GHz, two sets of tests were run: one with the Pal STA limited to 80 MHz maximum; and the second with it limited to 160 MHz. So we'll get to see what sort of benefit wider channels can bring.
I ran at least two tests for each benchmark because some of the results were kind of wonky. I'll be showing the better of the two runs.
For reference, I also retested NETGEAR's R7800 with v220.127.116.11 firmware and HT 160 mode enabled. This is a four-stream AC2600 class router supporting both DFS and 160 MHz wide 5 GHz channels, using Qualcomm QCA9984 4x4 MU-MIMO-enabled 802.11ac radios on both bands. Since all you are getting right now by buying a draft 11ax router is the equivalent of a four stream AC router that supports DFS channels and 160 MHz channel bandwidth, it's a good reference (and costs about $150 less).
Test Results - 2.4 GHz
Let's get the easy stuff out of the way with 2.4 GHz. Remember these tests are with 20 MHz channel bandwidth.
In the downlink test, both draft AX devices out-perform the AC-based R7800 with around 10% higher peak throughput (145 Mbps) that doesn't start to drop off until 9 dB more attenuation is applied. Both also have a tad more usable throughput than the R7800 at 63 dB of attenuation.
RvR 2.4 GHz - downlink
2.4 GHz uplink shows a different story. This time the R7800 and ASUS have the same peak throughput of around 130 Mbps, but the NETGEAR RAX80 is stuck down at 40 Mbps. This one was a bit of a puzzler, especially because Pal logs showed it was operating with an RSSI of around -30 dBm and maximum (173 Mbps) Tx and Rx link rates.
I eventually found that if the router was rebooted and the uplink test was run without first running the downlink test, it produced throughput similar to the other routers. So my conclusion is there's something wonky in the RAX80 transmit rate adaptation that will need to be fixed in a future firmware update.
Update 1/14/19: After working with both Broadcom and NETGEAR, I have a better explanation of the low uplink performance. The bottom line is there is a known Broadcom / Qualcomm interoperability issue that is not specific to Broadcom's 11ax chipset. I happened to stumble onto it because of two differences from my normal test method.
First, the router under test was not rotated while traffic was running during the test. This is meant to average out differences in product antenna design and to try to mitigate near-field effects from the small distance between DUT and STA. Rotation will be done in the finalized new test process.
Second, the Pal STA was not reassociated between downlink and uplink tests. Since the connection could have been broken under high attenuation conditions, the STA should have been reassociated to ensure it was properly connected. This will also be done in the finalized new process.
The low uplink throughput was caused by the STA's packet aggregation becoming stuck at low aggregation when attenuation is high (low signal level) at the end of the downlink run, then not resetting to high aggregation when attenuation is reset to low (high signal level) to start the uplink test.
Reassociating the device between downlink and uplink runs causes packet aggregation to be properly reset.
New uplink plots showing both original and retest results for the RAX80 have been posted.