So much for features, let's see how fast Luma is! I was pleasantly surprised to be able to run the new Version 4 router performance tests on Luma. This was possible because Luma stays up and running indefinitely without an internet connection once it is set up. You can't change settings and the Filtering and Security features won't work, but Wi-Fi will stay up and, as I discovered, routing functions still run.
I also tried to retest eero, but could not find a way to configure it so that it had the constant internet connection it requires to function while also connected to the QACafe NTA1000 that now runs our routing performance tests.
I didn't bother trying to run the functional test suite on either product. It requires configuring router settings neither Luma nor eero provide access to.
As noted earlier, Luma doesn't report firmware version, so I don't have that information to report. Table 2 summaries the new test results. I threw in the Linksys EA9500 AC5300 class router for comparison to show how Luma is no slouch in the wired routing department. As you can see, Luma did quite well. The TCP and UDP connection tests run as part of the performance test suite and turned in expected results. So far, only NETGEAR routers pass the UDP connection test because they appear to leave UDP sessions open longer.
|Test Description||Luma||Linksys EA9500|
|WAN - LAN TCP (Mbps)||941||918|
|LAN - WAN TCP (Mbps)||941||917|
|Total Simultaneous TCP (Mbps)||1764||1524|
|WAN - LAN UDP (Mbps)||960||950|
|LAN - WAN UDP (Mbps)||950||947|
|Total Simultaneous UDP (Mbps)||1898||1721|
Table 2: Routing performance comparison
Luma is not Wi-Fi Certified and does not support Wi-Fi Protected Setup (WPS). So any client you connect to it must be able to have its Wi-Fi network password entered. It supports WPA2/AES PSK wireless encryption only.
Since we're talking about wireless features, I'll share a few other things I learned from Luma:
- Band steering is supported
- Devices are not load balanced across multiple Lumas
- 802.11v, k and r are all supported.
I didn't bother asking Luma whether they could provide a special firmware load for testing that would allow me to set channels and bandwidth. I used inSSIDer to verify channels used (11 for 2.4 GHz, 40 for 5 GHz) and link rate to determine channel bandwidth. The starting link rate for 5 GHz varied among 526.5, 650 and 702 Mbps, indicating 2x2 operation in 802.11ac 80 MHz bandwidth mode. The 2.4 GHz starting link rate was 144.4 Mbps, indicating a 2x2 802.11n connection with 20 MHz bandwidth. The test client was connected using WPA2/AES encryption.
My previous experience with eero had it drop all Wi-Fi connections within 5 minutes of being disconnected from the internet. So I kept Luma internet connected during wireless testing to be safe and had my test client connect to Luma on its LAN side, just like all other wireless routers I test.
Luma was centered on the test chamber turntable in normal operating position, as shown in the photo below. The 0° position for the router had the front facing the chamber antennas. I put Luma on a short platform to get it more in line with the chamber's antennas.
Luma in test chamber
I tested using the new V9 test process. Because I knew you'd ask, I also retested eero using the same process. The original eero test results have been replaced by the new due to the test process change, which significantly changed the results. But now both eero and Luma are more fairly compared because the new test reflect both products in 20 MHz bandwidth mode in 2.4 GHz. I have not retested any other AC1200 class routers with the V9 process, so can't fairly compare eero and Luma to anything else right now.
After the original review posted, I had a discussion with a Luma engineer to discuss the wireless test results, which were not in line with Luma's. I agreed to do some further testing and in fact found a problem with my test results. The short story is the Intel test client was moving between bands during testing, paying no attention to the band preference and roaming aggressivness settings in the driver advanced properties.
Luma was able to temporarily set up separate SSIDs for each band so that I could retest the product. A helpful reader also tipped me to the Mandatory Access Point advanced properties in the Intel PROset Wi-Fi utility that I was able to use to ensure the client would not change bands for another eero retest. I also used the 802.11n Channel Width for 2.4 GHz advanced Windows property to successfully get 20 MHz bandwidth mode measurements for eero. The bottom line is I now have wireless test results that fairly compare the two products. So let's see how they compare.
Intel STA test settings
I ran Benchmark Summaries for both routers, shown below. 2.4 GHz average downlink performance is similar, but eero outperforms Luma on 2.4 GHz average uplink and both up and down average 5 GHz throughput. I've removed wired routing performance from these summaries, to not clutter the view.
Luma & eero Benchmark Summaries
The 2.4 GHz downlink profile shows both Luma and eero with similar range, with eero holding connection 3 dB longer. eero also has higher maximum throughput (90 Mbps vs. 73).
2.4 GHz Downlink Throughput vs. Attenuation
The 2.4 GHz uplink plot shows a much wider performance gap, with eero taking the high road, starting at 73 Mbps vs. Luma's 43 and staying above Luma throughout the test run.
2.4 GHz Uplink Throughput vs. Attenuation
5 GHz downlink shows similar performance, with Luma performing slightly better. Luma's starting throughput is higher (376 vs. 333 Mbps) and it stays connected 6 dB longer, indicating better range.
5 GHz Downlink Throughput vs. Attenuation
For 5 GHz uplink, Luma's maximum throughput is about half eero's and stays below eero throughout the test run. Luma's main advantage in this benchmark is that it stays connected 6 dB longer.
5 GHz Uplink Throughput vs. Attenuation
The takeaway is that downlink performance is similar for Luma and eero, but eero is significantly better on uplink. Luma also may have a slight advantage in 5 GHz range over eero.
I didn't put Luma through the multi-client or multi-hop tests I ran for eero. I'm working on better test processes to assess both multi-hop throughput and total wireless capacity and will do a head-to-head test of both Luma and eero when I have the tests worked out.
Even if I had run the tests, I doubt the results would change my bottom line on Luma, which is to wait before you buy. Luma has a lot of work to do in its feature set, documentation of the Security and filtering features and uplink Wi-Fi performance. Even if Luma gets its Wi-Fi tuned, however, it still has a disadvange of having only two radios vs. eero's three. eero's second 5 GHz radio gives it more flexibility to balance backhaul and client traffic.
It remains to be seen whether either of these "mesh" multi-AP systems or the others in the pipeline can lure enough buyers away from the traditional BHR (big honkin' router) high-margin cash generators the home Wi-Fi industry has trained us to salivate over. The clock is ticking and both startups are under the gun to ship product and grow as big as they can before NETGEAR, Linksys and other established players come out with their multi-AP solutions, which we know they have to be working on. D-Link could be first with its "DKT-891 Unified Home Networking Kit" based on Qualcomm's Wi-Fi SON technology, even though the Q2 ship date it gave when the product was announced at CES 2016 has come and gone.
I still hope these home networking upstarts survive long enough to light a fire under their larger competitors. We need to move closer to the multi-AP systems that are needed to provide good, fast Wi-Fi all over our homes in the 5 GHz band where you need to be to take advantage of higher throughput, MU-MIMO and other fun yet to come. For now, however, it's pretty clear Luma has its work cut out for it to overtake eero.