Another Change To The NAS Ranker

Photo of author

Tim Higgins

It is the nature of ranking tools that use raw performance data that they can provide a false sense of performance difference. Put a different way, is there actually a real-world difference between a product with 109.7 MB/s write throughput and one with 109.6 MB/s? Of course not, but you would think so by simply using the Ranker results.

So we have made two changes to Ranker logic. For each benchmark that goes into a ranking, we have added the ability to limit the benchmark value before it is evaluated. The second change is that we can now set a ranking tolerance for each benchmark, so that a range of values is used when comparing benchmark values during ranking.

We have used the limit feature to set a 125 MB/s cap on [NASPT] File Copy To NAS, [NASPT] RAID 1 File Copy To NAS, [NASPT] RAID 5 File Copy To NAS and [NASPT] RAID 10 File Copy To NAS benchmark results. This will take write cache effects out of the ranking process, since anything above the transfer limit of a Gigabit Ethernet connection is reflecting cache effects.

We have also set the ranking tolerance for all benchmarks used for ranking at 5%. This means that benchmark results must differ more than 5% for them to be ranked differently. We chose 5% because it’s a reasonable reflection of the margin of error of our test process. And quite frankly, is a product with 100 MB/s write throughput really better than one with 95 MB/s?

If a benchmark has a limit applied for ranking, you’ll see the capped value in the Ranker Performance summary. You won’t see any difference in the Ranker Performance Summary for benchmarks with tolerances applied, since the measured values are not changed for the ranking calculation.

We hope you find that these changes enhance the NAS Ranker’s value as a product selection tool. Be sure to let us know what you think either way via the Forums or just by dropping us a line.

Related posts

NAS

Do Your Friends and Family Know What A NAS Is?

We're on the road this week, so things may be a bit slow on SmallNetBuilder. But thought I'd share this food for thought.

How To Build a Cheap Petabyte Server: Revisited

Updated - The last time we checked with Backblaze, their DIY "Pod" stored 67 TB for under $8,000. See how they've kept the cost about the same, but doubled the capacity and performance.

New To The Charts: NAS Backup and iSCSI Performance

Due to popular demand, we have added eight new NAS Charts. You can now easily compare attached backup and iSCSI Target performance for products that support those features.