To recap, we use iozone as our primary tool to capture NAS performance data. iozone tests overall filesystem performance, which includes both host and client-side caches. Generally, when tested file sizes are below the NAS and iozone host machine RAM capacities, throughput will be be higher than the maximum speed supported by the network connection used in the test.
With our first iozone test machine, which was purposely limited to only 512 MB of RAM, higher-than-network-speeds dominated the small file size (64 KB to 16 MB) results, but had only a minor influence on the large file size (32 MB to 1 GB) tests, which are most often used to compare NAS performance.
Figure 1 compares small file size mode write test results for the same product—NETGEAR’s high-performance ReadyNAS Pro—with the old (blue line) and new (red line) test methods. Only three data points taken with the old test method are below the 125 MB/s maximum speed of the gigabit Ethernet connection used in the test, while there are no points below 125 MB/s in the new test method results.
Figure 1: Old and New Test Method Comparison – small file size mode
Figure 2 compares write results taken in the large file size mode where there are no test points above the 125 MB/s network speed with the old method vs. three with the new. But even with just three points above network link speed, the resulting average speed is 162 MB/s, which causes confusion.
Figure 2: Old and New Test Method Comparison – large file size mode
We have always recommended using read test results as the best indication of relative NAS Performance and still do. But some NASes have large differences in read and write performance, even when cached results are removed. And other NASes show average large file size write speeds in excess of network connection speeds due to unusually high results for file sizes up to 512 MB in some cases. So clearly a change was needed.
iozone provides a -c switch that includes the time it takes to close a file in the throughput calculation. Figure 3 shows the effect of using the -c switch, which removes all write cache behavior.
Figure 3: Effect of iozone -c switch on write
By contrast, and as expected, using the -c switch has no effect on read behavior (Figure 4). (The variation shown is typical for multiple test runs.)
Figure 4: Effect of iozone -c switch on read
But changing our test method to use the -c option would add another barrier to product comparison by introducing yet another test method change. It also would remove cache effects that are within network connection speed.
So, instead, we have added a new filter to the NAS Charts, the Cached Performance Filter. When it is enabled (which is the default), it removes all points above 12.5 MB/s for the 100 Mbps benchmarks and 125 MB/s for the 1000 Mbps benchmarks from the value plotted in the Average benchmark Charts. Like the Small File Size mode, the Cached Performance Filter is cookie-based and remains in effect for the browser session as you move from chart to chart.
It has no effect on the Throughput vs. File Size and Benchmark Chart views, so you will still see all data points taken when you switch to those views, even with the filter on. It also does not effect the File Copy benchmark charts, because that data is not obtained using iozone.
The filter does effect the Performance values shown in the Price vs. Performance plots for all benchmarks except File Copy.
When it is in effect, the filter is applied to all results, obtained with both old and new test methods. It has little effect on read results, since reads generally do not show extreme cached behavior. It has the most effect on write results for the small file size mode, which generally represent mostly cached behavior. Note that in some cases, the filter will remove a product from the chart in small file size mode if all data points are above maximum network connection speed.
We think that this is a reasonable compromise to solve the data distortion caused by abnormally high cached results.
Please drop us a line or post a comment to share your thoughts about this change or if you have a question.