It's the Drive, Stupid!
It was time to contact my expert in these matters, Don Capps, creator of iozone and a recognized expert in filesystems (his LinkedIn page). Don took one look at the results in Figure 3 and said "it's the drive". It took some convincing and additional testing on my part before I convinced myself that Don was right, but he was.
Basically, drive performance specs, like too many other specs, present an optimistic picture of real-life performance. A SATA drive with a 3 Gbps interface, really has a 300 MB/s "real speed" (3 Gbps is the signaling, or clock, rate and 300 MB/s (2.4 Gbps) is the maximum data rate).
But this is really the physical bus interface speed, which applies only when the data being written can fit into the drive's internal buffer. Although buffers have been getting larger (the 7K1000 drive has a 32 MB buffer), file sizes have been increasing even more rapidly. If you are moving around a lot of ripped video files, 32 MB of drive cache isn't going to matter much. On the other hand, if your file sizes are smaller, drive cache can provide a significant write performance boost.
Drive data rate is ultimately limited by the speed that bits can be read / written to the drive platters, which comes down to bit density and rotational speed.
So I first turned to the spec for the Hitachi Deskstar HDS721010KLA330 drive used in the 509 Pro. Unfortunately, as Figure 4 shows, it wasn't much help. The Interface Transfer rate is incorrectly listed as 300 Mb/s instead of the 300 MB/s that all "3 Gbps" SATA drives have (as explained earlier).
Figure 4: Hitachi Deskstar 7K1000 spec
The Media Transfer Rate describes the physical limit that I mentioned earlier and says that the fastest that data can be read / written to a single track on the disk. Since a single track doesn't hold much data, this is going to be a very optimistic number, which it is, at 134 MB/s. This is more than twice what is shown in Figure 3.
However, in real life your data is scattered among many tracks, so the time it takes the head to move from track to track (seek time) comes into the equation. So we have to turn to real-life testing to get a more accurate measure of drive performance.
StorageReview reports the 7K1000 drive's Maximum Read Transfer Rate as 86.9 MB/s and Minimum Read Transfer Rate as 46.4 MB/s. Since our ~66 MB/s maximum throughput in Figure 3 is between these two numbers, it looks like we are getting closer to reality.
Since both the 509 Pro and system running iozone have 1 GB of RAM, it's impossible to find out which memory is providing the cache / buffering performance improvement. Fortunately, the iozone test machine's 1 GB is made up of two 512 MB DIMMS. So I removed one and re-ran the test, with the results plotted in Figure 5. The difference in iozone machine RAM doesn't appear to make a difference, with the 509 Pro's 1 GB memory taking up the slack.
Figure 5: Single drive read throughput comparison
Actually, Figure 5 contains a couple of configuration variations so that we can explore a bit more. The light blue plot line marked "1 x 80 GB" represents performance with an 80 GB Hitachi 7K160 drive (specifically a Hitachi Deskstar HDS721680PLA380) instead of the 1 TB 7K1000 drive.
I had to go to the TomsHardware 3.5" Drive charts for the 7K160's performance test data, so the benchmark program used is different. Toms' reported the Maximum Read Transfer Rate as 75.8 MB/s and the Minimum Read Transfer Rate as 36.2 MB/s. However, the iozone tests show that even with the 80 GB drive, the results are virtually the same as with the 1 TB drive.
Three of the four runs were done with a single drive. So I added a fourth plot ("3x80GB R5") to see the effect of RAID 5 on read performance. Performance is virtually the same for most of the run. But once we hit the gigabyte file size, RAID 5 performance drops down to 15.5 MB/s! It also looks like the 7K1000 drive is providing a bit of performance advantage over the 7K160 drive (1x1 TB vs. 1x80GB lines), but neither is anything to write home about.