bonnie++ is a short test, running about 30 minutes on a single client. The information it yields can be used to compare differences between different mount options, but its real value is in deriving CPU usage during the various tests it performs. A test that returns a high CPU utilization indicates that your bottleneck is the CPU itself and that a faster processor will improve performance. A low CPU utilization indicates that the problem is elsewhere.
There are two parts to the bonnie++ 'benchmark.' The first part works with large files and the second part tests with small files. As the bonnie++man page states, the first part simulates what would happen to a filesystem when being used as a database server, while the second part is good for things like a mail cache or for a cluster node hosting a large number of small output files that need processing.
|-d directory||Directory to create files in|
|-n n||Number of files (times 1,024) to create in the second part of the test. Default is 30.|
|-m name||The name of the machine, used when outputting results|
|-r size||Size of RAM (in MB). The tests you run should be at least twice the size of physical RAM so that you can be sure that the filesystem buffers don't skew your results.|
|-u user||The user ID (UID) to use when running tests. bonnie++ doesn't like running as root, so specify some non-root user that has permissions to access the directory specified earlier wth -d.|
A standard run on a 4 GB machine is something like this:
$ bonnie++ -u markk -r 4096 -d /home/markk/bonnie/ -m myhost-base
Each time you run bonnie++, change the -m option to reflect what options you've enabled or changed between each test. For example, a subsequent run that enables noatime would use -m myhost-noatime and another test with 32 K block sizes and noatime is -m myhost-noatime-32k. The naming convention is up to you.
The results of bonnie++ include a human-readable summary that lists the statistic and the results and a comma-delimited line that can easily be imported into a spreadsheet or graphing application.
Once all of your bonnie++ tests are complete, you can start analyzing the results. What you're looking for is the highest numbers regardless of CPU utilization. In the case of two numbers being the same, take the lower CPU utilization as the better of the two. Now compare this to what you'll be using the fileserver for. If you have larger files, put more importance on the results of the first half of the test. For lots of smaller files, pay attention to the second half of the tests.