Personal computing discussed

Moderators: renee, morphine, Steel

 
Scrotos
Graphmaster Gerbil
Topic Author
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Tech Report's iometer config and testing

Tue Feb 26, 2013 1:08 pm

A fresh continuation of the thread originally by emorgoch as found here: viewtopic.php?f=5&t=79993

First you'll want to snag iometer. There are two major places you can find iometer, they are here:

http://www.iometer.org/doc/downloads.html (2006.07.27, don't use this, reference only)
http://sourceforge.net/projects/iometer/files/ (1.1.0-rc1, use this)

There's a bug in the last released binaries when it comes to config files. At the top and bottom of each config file you will find a line such as this:

Version 1.1.0

Now the new version, 1.1.0, doesn't recognize this version even though it's writing it itself when it saves the config files! It's still using the old versioning scheme which has a header and a footer that looks like this:

Version 2006.07.27

If you play around with config files and save them from within iometer, bear in mind you will have to open them in notepad or vi or whatever text editor you like and change the header and footer for iometer to recognize the config file. This bug is fixed in the source on sourceforge but I guess the project is dead and no new binaries have been produced. In theory you could compile it yourself and not have to worry about this workaround. The docs are also pretty much the same on both sites and haven't been updated since 2003. The options don't appear to have changed, though, so it's still a valid reference.

For additional help, there are seemingly active mailing lists:
http://www.iometer.org/doc/mailinglists.html
http://old.nabble.com/Iometer-f3210.html

Ok, now that this is out of the way, the process for installing iometer consists of downloading it and dumping it into a folder. Dump the config file below into the same folder though you can put it anywhere you want, really. You'll want to load up iometer FIRST and THEN load the config from within the program itself. You can double-click the config file directly but I've been having bad luck with that wherein I have to reboot to get iometer working again.

What you'll need to do is select the drive or array you want to test in the Disk Targets tab. Ideally it will be unpartitioned and raw for testing purposes. If you want to test an existing system, that will be addressed in a later post. The "manager" name will typically be your local computer's name. If you see a different name, well, that's just left over from my test system.

Hit the little green flag at the top center and save the results to whatever file you prefer. Your results will look something like this:

'Target Type    Target Name     Access Specification Name
ALL             All             Database
MANAGER         YOUR-COMPUTER   Database


What I have been doing is copying the header and the line starting on "Manager" for my results, starting at IOps. So I capture data on my runs that looks like this (copied all the way to the end, omitted for readability):

IOps            Read IOps       Write IOps      MBps (Binary)   Read MBps (Binary)
2752.033412     1842.325254     909.708158      21.500261       14.393166


You will typically plot the "IOps" column by the "Queue Depth" column to get similar bar graphs as TR does.

Note that by default there are no results shown. You can choose to show them with an update every N seconds if you want, but that will affect the results and doesn't duplicate what TR does. You also shouldn't be doin' much of anything else while benchmarking, but you already knew that!

The test patterns are sometimes difficult to compare between different benchmark places. When Intel opensourced iometer, apparently they didn't release all their test patterns that people had previously been using. Either that or at a certain point they renamed them or something. So it's a little confusing. The proper test patterns are in here and already selected for the test run.

Here is the config file. Copy the contents and paste it into a text file. Name the text file something like "tr-testing-4kb-new.icf".

Version 2006.07.27 
'TEST SETUP ====================================================================
'Test Description
   TR Drive Test
'Run Time
'   hours      minutes    seconds
   0          3          0
'Ramp Up Time (s)
   0
'Default Disk Workers to Spawn
   0
'Default Network Workers to Spawn
   0
'Record Results
   ALL
'Worker Cycling
'   start      step       step type
   1          1          LINEAR
'Disk Cycling
'   start      step       step type
   1          1          LINEAR
'Queue Depth Cycling
'   start      end        step       step type
   1          32         2          EXPONENTIAL
'Test Type
   CYCLE_OUTSTANDING_IOS
'END test setup
'RESULTS DISPLAY ===============================================================
'Update Frequency,Update Type
   0,WHOLE_TEST
'Bar chart 1 statistic
   Total I/Os per Second
'Bar chart 2 statistic
   Total MBs per Second (Decimal)
'Bar chart 3 statistic
   Average I/O Response Time (ms)
'Bar chart 4 statistic
   Maximum I/O Response Time (ms)
'Bar chart 5 statistic
   % CPU Utilization (total)
'Bar chart 6 statistic
   Total Error Count
'END results display
'ACCESS SPECIFICATIONS =========================================================
'Access specification name,default assignment
   2K OLTP,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
   2048,100,67,100,0,1,0,0
'Access specification name,default assignment
   4K OLTP,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
   4096,100,67,100,0,1,0,0
'Access specification name,default assignment
   8K OLTP,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
   8192,100,67,100,0,1,0,0
'Access specification name,default assignment
   32 Byte Data Streaming Read,NET
'size,% of size,% reads,% random,delay,burst,align,reply
   32,100,100,0,0,1,0,0
'Access specification name,default assignment
   32 Byte Data Streaming Write,NET
'size,% of size,% reads,% random,delay,burst,align,reply
   32,100,0,0,0,1,0,0
'Access specification name,default assignment
   512 Byte Data Streaming Read,DISK
'size,% of size,% reads,% random,delay,burst,align,reply
   512,100,100,0,0,1,0,0
'Access specification name,default assignment
   512 Byte Data Streaming Write,DISK
'size,% of size,% reads,% random,delay,burst,align,reply
   512,100,0,0,0,1,0,0
'Access specification name,default assignment
   8K Data Streaming Read,NET
'size,% of size,% reads,% random,delay,burst,align,reply
   8192,100,100,0,0,1,0,0
'Access specification name,default assignment
   8K Data Streaming Write,NET
'size,% of size,% reads,% random,delay,burst,align,reply
   8192,100,0,0,0,1,0,0
'Access specification name,default assignment
   64K Data Streaming Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
   65536,100,100,0,0,1,0,0
'Access specification name,default assignment
   64K Data Streaming Write,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
   65536,100,0,0,0,1,0,0
'Access specification name,default assignment
   TCP/IP Proxy Transfer,NET
'size,% of size,% reads,% random,delay,burst,align,reply
   350,100,100,100,0,1,0,11264
'Access specification name,default assignment
   File Server,DISK
'size,% of size,% reads,% random,delay,burst,align,reply
   512,10,80,100,0,1,4096,0
   1024,5,80,100,0,1,0,0
   2048,5,80,100,0,1,0,0
   4096,60,80,100,0,1,0,0
   8192,2,80,100,0,1,0,0
   16384,4,80,100,0,1,0,0
   32768,4,80,100,0,1,0,0
   65536,10,80,100,0,1,0,0
'Access specification name,default assignment
   Web Server,DISK
'size,% of size,% reads,% random,delay,burst,align,reply
   512,22,100,100,0,1,4096,0
   1024,15,100,100,0,1,0,0
   2048,8,100,100,0,1,0,0
   4096,23,100,100,0,1,0,0
   8192,15,100,100,0,1,0,0
   16384,2,100,100,0,1,0,0
   32768,6,100,100,0,1,0,0
   65536,7,100,100,0,1,0,0
   131072,1,100,100,0,1,0,0
   524288,1,100,100,0,1,0,0
'Access specification name,default assignment
   Database,NONE
'size,% of size,% reads,% random,delay,burst,align,reply
   8192,100,67,100,0,1,4096,0
'Access specification name,default assignment
   Workstation,NONE
'size,% of size,% reads,% random,delay,burst,align,reply
   8192,100,80,100,0,1,4096,0
'Access specification name,default assignment
   4K Data Random Write,DISK
'size,% of size,% reads,% random,delay,burst,align,reply
   4096,100,0,100,0,1,4096,0
'END access specifications
'MANAGER LIST ==================================================================
'Manager ID, manager name
   1,NERD-ANDREW
'Manager network address
   
'Worker
   TR Worker
'Worker type
   DISK
'Default target settings for worker
'Number of outstanding IOs,test connection rate,transactions per connection,use fixed seed,fixed seed value
   1,DISABLED,1,DISABLED,0
'Disk maximum size,starting sector,Data pattern
   0,0,2
'End default target settings for worker
'Assigned access specs
   Web Server
   File Server
   Database
   Workstation
'End assigned access specs
'Target assignments
'End target assignments
'End worker
'End manager
'END manager list
Version 2006.07.27


Please give me feedback so I can update this. I want to make it as much a "click here and you're DONE" kind of thing as possible.
 
Scrotos
Graphmaster Gerbil
Topic Author
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Tech Report's iometer config and testing

Tue Feb 26, 2013 1:53 pm

Ok, so let's say you have a production system, either a workstation or a server, and you're curious as to how it performs. Well, you can't duplicate TR's testing methodology unless the drive/array to be tested is raw. What are you gonna do, wipe out your VMWare or SQL server for this? Oh hell no you ain't!

So basically you won't have a good way to compare against benchmarking sites since they aren't under the same constraints you are. You can, however, compare your own systems and possibly compare with other people using the same settings. When iometer tests a raw drive, it tests over the ENTIRE drive. When you have a drive partitioned, by default it makes a temp file and attempts to make it consume all free space before it starts to test. This file is typically c:\iobw.tst (replace c: with whatever drive you selected for testing) and if you're running a *nix I have no idea where it dumps it.

This was real fun trying to figure out when iometer just did nothing after I started the test run. Left it to run overnight and I got errors that it couldn't write to the results file. Well yeah, because there was no free space on the filesystem! DOH! Also, if you want to change sizes of the temp file or reclaim space used by it, you must delete the temp file yourself. iometer does not clean it up. If iometer detects an existing temp file, your size settings will be ignored and the existing one will be used. Finally, on your initial run with a temp file, iometer will very SLOWLY create it. Just be patient. The DOS/console window will say that that's what it's doing. I made a temp file one time and pretty much used that for all my testing. There was an app that claimed to be able to make one quickly but it didn't work for me so I'm not including a link here lest it waste your time.

So we need to limit the temp file. Obviously on a production system you can't be runnin' outta space. The guidance I have seen to get a good test is to make the temp file 2 to 3 times the size of your system's RAM so that it isn't used for caching at all. If you're testing against a SAN or something like an EMC, I dunno, they sometimes have massive caches and that's beyond my experience. Since the systems I'm testing on have 32 GB of RAM, I made the test file 100 GB. There is a direct correlation in my testing to the size of the test file and the IOps. I tested at 5 GB, 100 GB, and 200 GB. The 5 GB one was fastest and the 200 GB was slowest. For example, the IOps was around 1800 for 5 GB, then 1400 for 100 GB, then 1300 for 200 GB. So not a huge drop from 100 to 200.

In iometer, the Disk Targets tab has a Maximum Disk Size selection. Default is 0 and that means unlimited size. Now this is done in sectors. 512 KB sectors. If you want to get 100 GB exactly, you can type in some number you calculate. Me? I'm lazy. I choose 200 million (200000000) and that got me 95-ish GB or so. Close enough for me.

If you're running vmware I guess you can just run a test from inside one of your VMs. I've seen other people do the same on the 'net.

Now since I'm running comparisons for a specific purpose, I selected only the db test pattern. The server runs SQL, not web, not file, not streaming video, I don't care about none o' that. So you may want to be selective in your tests as well especially if you have to schedule downtime to do this. I also differed from TR in my testing in that I ramped up my # of outstanding IOs (Test Setup tab, lower-right) from 32 to 256. Since I expect a server is going to handle heavier loads, I wanted to see worst-case stuff. I did in fact see scaling to 128 and in some cases 256.

This was a useful tool for me to justify the price of SSDs over regular SAS in a server build and also to see how tweaking RAID controller settings affected performance under simulated db loads.
 
Scrotos
Graphmaster Gerbil
Topic Author
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Tech Report's iometer config and testing

Tue Apr 16, 2013 9:55 am

Here are the results of some of the testing I've done:

http://screenshots.rq3.com/monk/iometer ... 5-ssd.xlsx

It includes a section from testing straight up mirrors plus some comparable benchmarks I found. On the graphs I tried to have some kind of description. For instance, I did a test with SSDs with physical drive write cache enabled and with it disabled. And I had a few runs with different cache bias for read/write. I will explain any of the graphs that you find unclear.

There's some "analysis" for the mirrored results. I did have some analysis for the SSD vs SAS results but I removed most of it besides some of the costs. I was advised that were I to present that to upper-level people, it was too wordy and technical. So... sadly... I've been working on trying to reduce it to a 1-page thing with lots of color that only somehow mentions saving money and not get so cynical about the whole thing that my will to live completely leaves me.

I used this testing to see what my comparable performance would be for some servers I was building and to see what the performance was for a server being replaced. How much better would the new one be than the old one? Would it be worth the cost and risk going to consumer-level SSDs instead of mechanical SAS drives? What was the performance trade-off going from RAID 5 to 6 to 1+0? What settings could I tweak on the controller that would affect performance under a database load? Stripe size? Cache bias? I knew coming into this that my RAID controllers and backplanes and even version of PCIe would limit the performance I could get out of the SSDs, but in my world I had only 2 servers to work with and the comparable performance was what was important to me.

Storage tested:
256 GB Samsung 840 Pro SATA
146 GB 15K 6G DP SAS 512547-S21
500 GB 7.2K Western Digital

Systems tested:
HP DL380 G5 458565-001, P56 5/2/2011 firmware
2 x Xeon E5430 2.66 GHz
32 GB RAM
P400i 256 & 512 MB BBWC, 7.24 firmware
8 x 146 GB 15K 6G DP SAS 512547-S21
8 x 256 GB Samsung 840 Pro SATA

Dell PE2900ii, 2.7.0 firmware
2 x Xeon E5345 2.33 GHz
12 GB RAM
PERC 5/i 256 MB BBWC, 5.2.2-0072 firmware
2 x 146 GB 15K 6G DP SAS 512547-S21
2 x 500 GB 7.2K Western Digital

HP DL320s AH691A, W04 6/10/2008 firmware
1 x Xeon 3070 2.66 GHz
6 GB RAM
P800i 512 MB BBWC, 7.24 firmware
2 x 500 GB 7.2K Western Digital

Who is online

Users browsing this forum: No registered users and 5 guests
GZIP: On