Another Change To The NAS Ranker

Photo of author

Tim Higgins

It is the nature of ranking tools that use raw performance data that they can provide a false sense of performance difference. Put a different way, is there actually a real-world difference between a product with 109.7 MB/s write throughput and one with 109.6 MB/s? Of course not, but you would think so by simply using the Ranker results.

So we have made two changes to Ranker logic. For each benchmark that goes into a ranking, we have added the ability to limit the benchmark value before it is evaluated. The second change is that we can now set a ranking tolerance for each benchmark, so that a range of values is used when comparing benchmark values during ranking.

We have used the limit feature to set a 125 MB/s cap on [NASPT] File Copy To NAS, [NASPT] RAID 1 File Copy To NAS, [NASPT] RAID 5 File Copy To NAS and [NASPT] RAID 10 File Copy To NAS benchmark results. This will take write cache effects out of the ranking process, since anything above the transfer limit of a Gigabit Ethernet connection is reflecting cache effects.

We have also set the ranking tolerance for all benchmarks used for ranking at 5%. This means that benchmark results must differ more than 5% for them to be ranked differently. We chose 5% because it’s a reasonable reflection of the margin of error of our test process. And quite frankly, is a product with 100 MB/s write throughput really better than one with 95 MB/s?

If a benchmark has a limit applied for ranking, you’ll see the capped value in the Ranker Performance summary. You won’t see any difference in the Ranker Performance Summary for benchmarks with tolerances applied, since the measured values are not changed for the ranking calculation.

We hope you find that these changes enhance the NAS Ranker’s value as a product selection tool. Be sure to let us know what you think either way via the Forums or just by dropping us a line.

Related posts

New To The Charts: Cached Performance Filters

There has been long-running discussion both in the SNB Forums and directly with readers about the use of cached performance data in our NAS Charts. So we have introduced a new feature that we hope will help make the Charts more straightforward to use.

Why Cache Matters in NAS Performance

To really understand NAS performance, you have to look end to end.

How To Build a Cheap Petabyte Server: Lessons Learned

You probably don't need 1,000 Terabytes sitting on your LAN. But understanding how one company set out to build their own storage module for a tenth of what vendors charge may help you in your next NAS build.