This article originally started out to be an exploration into how to make routers fail. The goal was to develop a router "torture test" that would reveal which routers would be more (or less) reliable over time. It turns out that this is harder to do than you might think.
Given the complexity of the code in today's routers and the wide range of internet services that they have to play nice with, revealing a particular router's failure modes is a tough job. It's easy to hammer a router with a traffic stream to see how fast it will move packets from WAN to LAN and back again. But doing that alone usually doesn't do anything other than produce a huge test file that shows that the router handled all the packets just fine. (I know, I've tried.)
It's much more difficult to devise a test scenario that causes a router to lock up or drop its WAN connection intermittently, behavior that many consumers experience all too frequently.
In the course of my research, I remembered the folks at QA Cafe. This small group of self-described "packet geeks" makes a line of products built around its flagship CDRouter router test solution. CDRouter is a Linux-based router test suite that currently includes over 400 tests in its base product. The test suite can be expanded with the optional modules shown below to add even more test cases.
CDRouter Product Family
I was going to use the free demo to see if CDRouter would do the job. But after an initial discussion with the QA Cafe guys, they offered to loan me one of their CDRouter-in-a-box appliances, the NTA1000. This turned out to be a good decision on their part, because it let me bypass the hassle of configuring a Linux system with enough Ethernet ports and go straight to learning how to configure CDRouter.
QA Cafe NTA1000
It's important to point out that CDRouter is not intended as a stress test tool. Its key focus is as a regression test tool for router designers. CDRouter's test suite can verify that a wide set of router functions work in the first place and also ensure that they are still working when firmware is changed. But given the extensive range of tests and the ability to repeat tests, loop the entire sequence of tests and even shuffle the order of tests, CDRouter can also provide some level of router stress testing.
Since I had to drop back from my original plan of pushing routers to their breaking point, I decided to take a different approach. As I've noted in the past, many users swear that alternative firmware like DD-WRT or Tomato is more reliable than what comes loaded on routers. If I only had a dime for every snarky comment I've seen about how bad router factory firmware is, I'd be on a beach somewhere instead of writing this!
There is no question that alternative router distros like DD-WRT, Tomato, etc. provide access to broader feature sets than factory firmware. But do undesirable features (AKA bugs) also ride along with the new tricks that these alternative firmwares enable? I decided to find out.
The object of my attention was the ever-popular ASUS RT-N66U "Dark Knight" router. I chose it because, yes, it's popular. But it also has a unique alternative firmware available for it that takes a very different approach. Asuswrt-Merlin (AKA Merlin) has bug fixing as its top priority and only adds "a few basic features and tweaks to the original firmware". (Check Scott's Merlin review for the details.) I thought Merlin was a good firmware to test to see whether it lives up to its claim of fixing, not making bugs.
The test approach was:
- Determine the CDRouter test suite
- Flash the RT-N66U with each firmware
- Run the test suite
- Compare the results
This is admittedly a simple approach, but would at least determine if the alternative firmware tested takes a "first, do no harm" approach.
Step 1 was the most difficult. I had to choose a test suite that tested as much as it could, but that also didn't test things that the factory firmware wasn't designed to do in the first place. I also had to work with the confines of a fixed router setup. CDRouter doesn't have the ability to access the router admin GUI to change settings during testing. Since I wasn't up to the task of finding a scripting platform that could do both that and also run CDRouter, I had to use the same router setup for each firmware.
Fortunately, the guys at QA Cafe were very helpful. Given their experience with testing a lot of routers, they were able to tip me to common router weaknesses and help me determine whether the test failures I saw were real or a result of test setup misconfiguration.
The firmwares tested are listed below. To be exact, I'm showing the actual firmware files loaded.
ASUS factory: RT-N66U_18.104.22.168_260.trx
- Asuswrt-merlin: RT-N66U_22.214.171.124_266.23b.trx
- Tomato (Shibby): tomato-K26USB-1.28.RT-N5x-MIPSR2-104-AIO-64K.trx
- DD-WRT: dd-wrt.v24-19342_NEWD-2_K2.6_mega.bin
Before the each test, I made sure each router was configured as follows
Set router address to 192.168.1.1
- Dynamic DNS set for DynDNS service, test.domain.com Host name, qacafe username and qacafe123 password
- Set WAN to clone router MAC address
- Enabled LAN DHCP server, range 192.168.1.2 - 253
- WAN type set to DHCP client
After some intial runs using the factory firmware to debug my test plan, I reflashed the router with each firmware and ran the test suite once on each router. In cases where I suspected problems with test configuration, I reran the suite after correcting the router configuration.
The graphic below, taken from the CDRouter BuddyWeb console, shows all the available test modules, with the ones included in my test suite checked. Some tests are simple and take mere seconds to run, like the tests verifying VPN passthrough. Others, like the renum_dhcp group that verify that the router continues to function properly through a WAN DHCP lease renewal, take 5 minutes or longer due to the wait for DHCP lease renewal.
If you want to see more information about the tests in each group, use the online list. Although I didn't use all of the 400 available tests, I think you'll see that the tests I ran hit most of the key duties that any router must perform flawlessly.