How We Test Hardware Routers – Version 4

Photo of author

Tim Higgins

This process applies to routers tested after 15 May 2016. The previous process can be found here.

Updated 1/29/17 – Removed UDP tests due to process error

It has been just over six years (!) since we last revised our wired router test process. Since then, many routers now routinely max out our simple throughput tests. So it’s time to work these products a lot harder.

The new process does away with the pair of computers and IxChariot, instead using QA Cafe’s NTA1000, and flagship CDRouter router test solution.

QA Cafe NTA1000

QA Cafe NTA1000

CDRouter is a Linux-based router test suite that includes over 4000 functional and performance tests. The video below provides a short overview of CDRouter.

CDRouter Overview

Setup

We test all routers by directly connecting WAN and LAN ports to the NTA1000’s Gigabit Ethernet ports. Note the same test setup is used for both wired-only and wireless routers, i.e. the wireless portion of wireless routers is not involved in this testing.

We do not test LAN – LAN, i.e. the router’s switch, performance. The switch chips used in today’s routers all support non-blocking wire-speed performance.

Router Test Setup

Router Test Setup

All routers are prepared for test as follows:

  • Upgraded to latest firmware
  • Reset to factory defaults
  • QoS features disabled (if present)
  • Remote administration disabled (if enabled)
  • WAN type set to DHCP client
  • DoS protection enabled (if present)
  • LAN DHCP server range set to 192.168.1.10 – 30
  • LAN DHCP server lease time set to 120 minutes (if available)
  • All ALG passthroughs enabled
  • Remote management (HTTP / HTTPS) disabled (if enabled, if available)
  • Local HTTPS management enabled (if available)
  • TCP port 21 forwarded to 192.168.1.200
  • UDP port 988 forwarded to 192.168.1.200
  • Triggered port forwarding (if available) set for Dialpad
    • Trigger port = 7175
    • Trigger type = UDP
    • Opened ports = 51200-51201,51210
    • Opened port type = TCP, UDP
  • Triggered port timeout set to 60 seconds (if available)

Performance

We run these tests, selected from the CDRouter perf.tcl IPV4 performance test module:

  • TCP download throughput test WAN to LAN
  • TCP upload throughput test LAN to WAN
  • TCP bidirectional throughput test
  • UDP download throughput test WAN to LAN
  • UDP upload throughput test LAN to WAN
  • UDP bidirectional throughput test
Update 1/27/17: UDP benchmarks were removed when it was found incorrect versions of the UDP benchmark tests were run for many of the products tested.

Each test is run with 10 simultaneous connections with each connection presenting a 95 Mbps load. Each test runs for 5 minutes. Note that CDRouter uses iperf to measure throughput.

For TCP throughput tests, we report throughput and retries. For UDP throughput tests, we report throughput and % packet loss. CDRouter reports up and download throughput separately for the bidirectional tests. The Router Charts bidirectional benchmarks present the sum of the separate results.

We have retired the Maximum Simultaneous Connection test. We now test whether 3,000 simultaneous TCP and UDP sessions are supported in separate tests. This new test is actually harder than the previous maximum connection test because it opens the connections over a longer period of time, then goes back and checks they are still open.

Functional Tests

CDRouter’s primary focus is router functional testing. We run 245 tests, checking DNS, DHCP, UPnP, Xbox, NAT, firewall and other functions. We also run nmap scans from LAN and WAN to check for open ports. You can download a PDF of the full test list to see exactly what we run. Note we start the test by throwing a series of DoS attacks at the router, giving it 30 seconds to recover from each one.

In the Router Charts, we report the results in the Router Functional benchmark as a % of tests passing. The list of failing tests with explanation is noted in the product review.

Related posts

How To Configure Your Ubiquiti EdgeRouter Lite

Updated - If you are struggling to get Ubiquiti's EdgeRouter Lite to function as a basic router, these tips and pre-made config files should help

How To: One Internet connection – Two Private LANs

There are times when having two separate networks - both sharing the same Internet connection - can come in handy. For example, I recently helped a community center with its network setup. They needed to provide Internet connection to tenants who were renting space, in addition to their own shared Internet. They also shared a number of folders on the network, but weren't too careful about password protecting the shares.

How We Test Hardware Routers – Revision 3

We've revised our router test process with a new Simultaneous Connection test capable of measuring as many connections as routers will provide.