CDRouter unfortunately doesn't have an easy way to compare test results. So I used its export feature to get the test results into Excel, sorted the Passes and Fails and summarized only the failed tests into the table below. The table is arranged to provide a separate row for each failed test. If you see the test name in a row, it failed. If you don't, it didn't.
|sip_42||sip_42||[all sip tests]|
|sip_45||sip_45||sip_45||[all sip tests]|
|sip_60||sip_60||sip_60||[all sip tests]|
|sip_61||sip_61||sip_61||[all sip tests]|
|sip_62||sip_62||sip_62||[all sip tests]|
|sip_63||sip_63||[all sip tests]|
|sip_73||[all sip tests]|
CD Router failed test summary
It's clear that UPnP failures are present in all four firmwares, as are failures in the SIP ALG (Application Level Gateway). But some of the failures common to all four firmwares can be considered "features", as we'll see in the commentary below.
To start us off, I passed some of the failures back to QA Cafe for review. Their comments, edited for clarity, are below with my additional comments preceeded by "SNB" and in italics.
dhcp_server_700: this looks like a legitimate failure if the ASUS claims that it supports the DHCPINFORM message.
SNB: I didn't confirm this with ASUS. But since the failure is common across all firmwares, the problem isn't unique to ASUS and could be considered a "feature"
firewall_2: this looks like a legitimate failure. The ASUS firewall allowed an inbound TCP connection from a host on the WAN to a client on the LAN at port 80. This test should normally pass, and the firewall appears to be enabled based on the other firewall test results.
SNB: More discussion on this below.
app_14: this test is failing because when CDRouter tries to verify that the FTP port has been closed, it finds that it is actually still open. To properly run this test you need to know what the ASUS uses for an FTP port timeout.
SNB: As noted, this is more a test configuration issue. But I left the test in to see if any of the other firmware made it go away. The test value for the timeout is 80 seconds. The actual timeout is longer than this.
ipsecpt_30: this looks like a simple configuration issue. It looks like the ASUS supports passthrough of unknown IP protocol packets.
SNB: Again, I left this test in to see if the other firmware distros exhibited the same behavior. They do since the test fails for all distros.
ipsecpt_110: this looks like a legitimate failure.
SNB: More discussion on this below.
upnp_36 and upnp_41: these look like legitimate failures, if ASUS claims support for creating port mappings using specific source IP addresses via UPnP.
SNB: More discussion below.
dyndns_10: this is a simple config mismatch. The ASUS is reporting a user agent string of ez-update-3.0.11b5 unknown  (by Angus Mackay). However, your config file does not match this - it is missing the square brackets. To fix this, just edit the testvar "dynDnsAgent" and set it to a value of "ez-update-3.0.11b5 unknown  (by Angus Mackay)".
SNB: The failure described above is accurate for the ASUS stock, Merlin and DD-WRT. But for Tomato, the test failed because the DynDNS client did not send a DynDNS update request.
dns_45 and dns_46: it looks like the ASUS does not failover on certain error codes
SNB: This error is common to all firmwares, but the failure codes varied. All firmwares had problems with error code 4. Tomato also failed on error code 8. DD-WRT was the worst, failing on codes 1, 3, 4, 6-15.
Like the cdrouter_dhcp_server_700 test, the cdrouter_ipsecpt_110 test might be also considered a "feature". Or at least, a missing feature. The notes for this test indicate that the test is only supported by routers using SPI tracking features to build NAPT mappings for return IPSEC/ESP traffic. Some routers attempt to serialize the return connections and associate incoming SPIs with outgoing connections. While this does not work if packets are reordered or lost, some routers do use this technique to attempt to allow multiple LAN clients to use the same VPN server. Since this test failed for all firmwares, this technique is obviously not supported.
The firewall_2 failure is surprising and appears on both ASUS and Merlin firmware, but not Tomato or DD-WRT. The test sends a TCP Syn packet from a host on the WAN (18.104.22.168) to a client on the LAN (192.168.1.107). The packet has a source port of 10525 and destination port of 80. This packet was actually forwarded by the router to the LAN client, implying that port 80 for that client (at a minimum) is open to hosts on the WAN.
QA Cafe noted that a port scan probably won't find this issue, because the port scan targets the WAN IP of the router under test. The CDRouter test doesn't probe the WAN for open ports; it just assumes that port 80 is open to a LAN side IP address.
The likelihood of this causing a problem in real life use is minimal, because private IP addresses are not routable on the public internet. So even though the RT-N66U does not drop private IP packets on the WAN, all of the other devices on the public internet which the ASUS is connected to should. However, most devices have drop rules on their WAN interface(s) to specifically account for this and the ASUS and Merlin firmwares don't. So this should be fixed.
The large number of SIP test failures mainly tell us that all the SIP functions that CDRouter tests are not supported by the different firmwares. But DD-WRT's failure (this is the MEGA version, which is supposed to include SIP support) to pass any of the SIP tests caused me to go back to make sure that there wasn't a software switch that I hadn't thrown. I didn't find one and reran the SIP tests and they still didn't pass. I'm by no means a DD-WRT expert, so I'll be happy to retest if someone points out any error that I have made.
The UPnP failures have a story similar to the SIP failures, i.e. they mainly reflect differences in UPnP handling implementation. Tomato differentiates itself in the least number of UPnP failures. But both Tomato and DD-WRT pass upnp_50 and upnp_202, which stock ASUS and Merlin firmware fail.
Of the two, upnp_50 is more interesting, since it tests that ports opened by UPnP are properly closed when their lease time expires. The test adds a UPnP port mapping with a lease time of 30 seconds, verifies that the mapping works, then waits 45 seconds and checks to see that the mapping is no longer there. The fact that the stock ASUS and Merlin firmware fails this test says that those firmwares don't pay proper attention to automatic UPnP port mappings.
Although it failed for all four firmwares, upnp_220 didn't fail the same way for all firmwares. This test checks that the maximum number of UPnP event subscriptions (10) can be created and then shut down. Stock ASUS and Merlin firmware were able to open ten subscriptions, but were only able to cancel subscriptions for the first three. Tomato was able to create all ten subscriptions and cancel eight of them. DD-WRT performed the best for UPnP, able to subscribe and unsubscribe all ten events. The test failed only because one UPnP NOTIFY even was not received within a 120 second window.
A big failure for DD-WRT is the renumber_2 test. This test checks that a TCP connection can be reestablished after a WAN DHCP renumbering. Failure of this test could indicate that a router could require a reboot to get going again after a WAN DHCP cycle that results in a new IP address.
The multiple scale tests failures for Tomato indicate trouble with its LAN DHCP server. I set the DHCP server range as wide as possible (2-253), so you might want to avoid this distro if you are going to have > 200 DHCP clients.
Tomato is also unique in its dyndns_1 and dyndns_2 failures. These indicate basic failures in the dynamic DNS function, i.e. the ability to send an update request when the router's WAN IP address changes and the ability to not send a request when the WAN IP doesn't change. In the first case, Tomato made the request update request, but sent the wrong WAN IP address. In the second case, Tomato tried to update DynDNS when it shouldn't have.
Finally, the failures of dns_70 and dns_120 tests in both Tomato and DD-WRT could be considered esoteric issues to some. But the fact that they failed in those firmwares while they did not in the stock ASUS and Merlin, once again indicates less complete implementations.
The dns_60 failure unique to DD-WRT is a bigger deal, however. The test logfile showed that when the WAN link was dropped and reestablished, DD-WRT properly registered the new DNS servers. But when a LAN client sent a DNS request to the router's DNS proxy, it didn't respond within the alloted 5 seconds. If that were your LAN client, your browser would have hung or returned a "can't reach the internet" message. Not a good thing.
While I have focused on failures in this story, it's important to step back and look at the big picture. All firmwares passed tests for NAT, packet forwarding and DOS protection with flying colors—all things that are essential to router function. On the other hand, I didn't test port forwarding, WAN connections other than DHCP, DMZ, URL filtering, triggered port forwarding, multicast and a bunch of other stuff. What demons lay undiscovered there will have to wait for another time.
But while the number of actual failures (vs. "features") is relatively small compared to the 200 tests in the suite, the tests that did fail tend to show that this experiment was worthwhile.
As I was grinding through the analysis of the tests that failed, I sometimes thought "Who cares?" about some of the issues. But, in the end, that's not for me, or router makers, to decide. As I said at the top of this piece, there is a wide range of internet services out there. And often what is a "who cares?" issue for one ISP or user can be a critical issue to another.
What this simple experiment has shown is that the answer to the question "Does Alternative Firmware Break Your Router?" is yes, yes it can. But whether what it breaks makes a difference to you is something only you can decide.
So go ahead and thump your chest about how great your router is now that you've gotten rid of the horrible, crappy, [insert favorite pejorative term here] factory firmware and replaced it with your wonderful favorite alternative. But just remember that along with the zippy new features you got, you just might have gained some unwanted stowaways, too.
Special thanks go to QA Cafe for the loan of their NTA1000, their help in getting me up to speed on it and interpreting its test results and especially for their patience in how long it took me to get this article done!