What do you want to see in our new storage test suite?

The number of new hard drives, SSDs, and other interesting storage devices due out this year is staggering. Heck, even what’s coming in just the next few months is daunting. To meet this growing tide, I’m in the process of putting together a fresh suite of tests for storage reviews. And I’m open to suggestions—reasonable ones, anyway.

Obviously, the focus of the suite will be isolating storage subsystem performance. I’d like to have a good mix of synthetic and real-world benchmarks, and I’ll probably restrict testing to Windows 7. So, what would you like to see in our new storage test suite? Keep in mind that a good benchmark test should be repeatable and offer performance metrics that are easily measurable. Ideally, tests should also be scriptable and use freely (or cheaply) available software.  Oh, and relevant.  Duh.

You can make suggestions in the comments below or email me directly. I should also note that a few elements of our current suite will probably migrate to the new one: IOMeter and FC-Test will definitely return, and there’s a possibility that iPEAK—or something like it—could, as well. We’ll also be doing system boot time tests and our usual level load tests with an updated collection of games.

Comments closed
    • Bibby
    • 10 years ago

    Something I really want to see is the effects of an SSD while playing something like Crysis Warhead.

    Myself and other people experience trouble on certain levels in this game due to it’s demand on memory…. it starts eating into virtual memory which gives huge stutters making it almost unplayable.

    I believe I have not yet seen a website perform such a test like this… storage directly affects the frame rate of the game!

    • Bensam123
    • 10 years ago

    Surprised you haven’t caught on Geoff, but there is a giant community growing around the perc 5/i and perc 6/i off of ebay as a cheap, fast, solution to having faster storage. The15k SAS drives in general are cheaper then Raptors and if you go back a step SCSI 320s are even cheaper then that.

    §[< http://www.overclock.net/raid-controllers-software/359025-perc-5-i-raid-card-tips.html<]§ Yet SCSI and SAS drives have been absent from a lot of the roundups even though they're economic alternatives to Raptors and even comparable to Blacks price wise. I saw a 400gb 15k Cheetah go for $187 on ebay the other day. It was especially bothersome with the onslaught of SSDs that there direct competition is MIA.

      • Krogoth
      • 10 years ago

      SSDs have rendered high-RPM SAS and SCSI HDD obsolete. That is why SAS and SCSI devices are so damm plentiful on ebay and little demand on them = bargain basement prices.

        • continuum
        • 10 years ago

        Agreed, but also for an additional reason– no reason to go PERC6i when you know SATA and SAS 6Gbps are basically already here.

        Also this is primarily going to be a single-drive testbed, why even discuss RAID controllers? (as much as I enjoy seeing such reviews of RAID cards, they are a whole ‘nother degree of complexity!)

    • Dirge
    • 10 years ago

    I would like you to mention the number of platters and platter size/density where ever possible. Why because it satisfies my inner geek, but these details would also help those interested in acoustic properties or potential speed.

    • rhema83
    • 10 years ago

    I am only asking for something simple – include in the comparison the HDDs / SSDs recommended for the most recent System Guide. Heck, you can even use existing data if they are already tested with the same suite. It will be very useful to know how far we are from “the edge” when we go with a “mainstream” budget, for example the Utility Player build. I mean, the 2TB Caviar Black is nice, and so are the Sandforce SSDs. But many of us are going to read the reviews and end up buying something lower in the same series.

    Case-in-point – I just build a Utility Player type build yesterday and really wanted to use a consumer-grade SSD for my system drive, and have a 1TB Caviar Green for my storage. But I ended up with a 640GB Caviar Black instead because of budget constraints. Heck, I had to substitute the recommended P55 board with a mATX H55 one to shave off $30, reuse my NSK4480 and HD4850 512MB from two years ago.

    • wagsbags
    • 10 years ago

    Real world tests. Review sites too often make contrived tests to isolate exactly the hardware they’re trying to test but at the end of the day readers should be able to answer the question “how much faster will this make my computer.”

    • Krogoth
    • 10 years ago

    How about fall survivability?

    Granted, such a test would require sacrifices.

      • Meadows
      • 10 years ago

      …I lol’d

    • lethal
    • 10 years ago

    When doing a SSD review, including at least one mechanical drive for reference would be appreciated.

    • Bibby
    • 10 years ago

    I want to see more real-world tests:

    Windows Defender scan
    Anti-virus scan
    Windows Backup to & from a faster medium i.e. RAID
    Verify Windows file integrity via sfc.exe

    I’m sure there’s more stuff like this that is possible… and free of cost.

    • statrekgeneral
    • 10 years ago

    Windows Vista and 7 based 64 OS’s
    Worst case performance charts for all SSD’s when doing comparison
    Actual usability descriptions
    Higher End testing hardware(more realistic hardware). Core i7, 4 – 6 gb of ram, etc

    • KGA_ATT
    • 10 years ago

    I’d like to see far more drive scrutiny all around, and compare not just SATA and SSD drives but also SAS drives. Additionally I’d like to see how each drive performs off of low, mid an high end HBA & Raid Controllers. IOPS, Read, Write, Sustained and so on. Compare that to the motherboard controller performance. So many just simply accept the performance of the motherboard controllers without any thought to a ‘discrete’ (add on) controller.

    And how about attempting to dual port SATA drives with add on adapters from LSI or Promise? How do such drives perform with such alterations and the like.

    Finally how about more information about the internals? Where the components are made? Why aren’t more SATA drive’s spinning at 10k rpm theses days? The standard is mature so what is the hold up? Why not 12k or even 15k SATA drives?

      • Krogoth
      • 10 years ago

      The problem with is that discrete controllers are typically geared towards the professional crowd. It is rare to find any significant RAID array in an enthusiast rig. Those few who want such an array do their own research.

      The performance benefits of RAID and other high-end I/O solutions are mostly found in multi-user environments or moving GiBs worth of data on a daily basis. These scenarios usually do not happen on desktops. When you factor the cost. High-end RAIDs and other I/O solutions make very little sense for desktops.

      The reason why high RPM HDD doesn’t exist for desktop crowd is because they are only faster than their 7200RPM counterparts at random access speed. This only matters in multi-user environments which I had already mentioned are a rarity in desktops. Besides, SSDs already made high RPM HDD completely obsolete as SSDs are far superior at random access speed.

        • KGA_ATT
        • 10 years ago

        I realize some of your points had validity years back but many of todays enthusiasts happen to be professionals as well. And I can only conclude that there is a high percentage of RAID 0 setups on their gaming machines. There happens to be some mental block for many when it comes to the price/performance or merely performance that HBA’s & Raid Controllers offer.

        As programs have grown in size over the last decade (many have become resource gluttons)and in response we have seen faster CPU’s, GPU’s and RAM speeds in leaps and bounds. We now have 6-core CPU’s. QUAD capable GPU setups and staggering RAM speeds.

        But 7200 rpm has been around for over a decade back with EIDE drives. Much of the HD internal mechanics are not just simply a ‘matured’ technology but astoundingly ‘aged’.

        Again you may be limiting your views about enthusiasts. Many of them have networks at home, as in many households in modernized areas around the world. They transfer huge amounts of data and that will only increase. So along with the increased Ethernet throughput how about some performance solutions via add in HBA & Raid controllers? How can this not be needed? In the next years many will be saving their HD/Blu Ray movies, HQ audio/music, weekly/daily TV shows over those home networks. This is not just a ‘corporate’ or ‘enterprise’ need as you what I gather from your post

        As life gets more digitized and the GB’s of data moved around today become TB’s tomorrow, there is a clear need for the performance of Raid 0 and the security of Raid 5 in growing segments of the average household(tax, family info, pics, data, stocks, etc. Not to mention the (professional)enthusiast crowd.

        Thus the views about suggestions in what I would like to see in the new storage testing.

          • Krogoth
          • 10 years ago

          SSDs are going to be the catalyst for growth in I/O performance in the mainstream not RAIDs/HBAs.

          Nested RAIDs and HBAs are simply too complex and demanding (cost, drivers, configuration) to see any usage outside of die-hard enthusiast and professionals.

          HDDs are going to retire into archive land for their superior $$$$/GB ratio.

            • KGA_ATT
            • 10 years ago

            Very possible forecast. But the memory industry has a reputation and history of absolute greed. Therefore the MLC or SLC costs will eventually reach the lowest point they can go and will budge no further. It appears the memory industry is on the ‘greed’ march these days and that translates into higher prices for DDR, Nand and the like. Memory is ‘volatile’ when it comes to pricing.

            However magnetic technology can grow to 15TB’s or 20TB’s within 3-5 years and still maintain mainstream pricing. My hopes it to see reflected in some test done here by The Tech Report that have options/alternative configurations that make sense. Performance-wise and Cost-wise. Some of the on-board Raid controllers found on motherboards aren’t as capable as the $40 to $80 ‘add on’ HBA’s or Raid Controllers on the market. Of course those are the entry level products, but bench them in the analysis. Then the $200 HBA’s/Raid Controllers. Of course the $1k and up Raid Controllers will remain in the Enterprise segment until we have Teleporters and home MRI’s n-stuff.

            As far as ‘too complex’ for folks to use? I dunno about that, have you seen some of the latest hand held devices? It may take me hours to get one tricked out just right. However a Raid 0 setup is done in seconds..OK may minutes if you have to install the drives.

            • Krogoth
            • 10 years ago

            I was referring to nested and other more exotic flavors of RAID. 😉

            RAID 0 and 1 are pretty darn simple to setup, require very little CPU overhead for a software flavor. RAID0’s only fault is that it offers no fault tolerance and it increases your chances of data loss due to HDD failure. RAID1 on the other hand offers fault tolerance for one drive, improves read performance, but takes a slight hit at writes. Desktops typically do more reads and they do writes.

          • Flying Fox
          • 10 years ago

          q[http://faq.storagereview.com/tiki-index.php?page=SingleDriveVsRaid0<]§ SR also did a test and plotted a couple of diagrams of a "high end professional" access pattern vs a truly server-side multi user access pattern. Can't find the page now though but it shows even you think those Photoshop+3dsmax+whatever you are running together, it is still no multiuser relational database.

            • KGA_ATT
            • 10 years ago

            Somehow I’m not following your reply about Raid 0 being a performance myth? And that it’s been ‘debunked’ by whom?

            Well I’m not entirely sure how to respond to that. I view it as ‘settled’ law.

            Whatever the reason for the Raid 0,1 or 5 that many enthusiasts/professionals have it is somehow dismissive of the wide plethora of titles that range from Flight Sim’s, CRPG’s, 1st Person Shooters, RTS’s and conclude that RAID 0 configuration doesn’t offer benefits. Nonsense. It can offer enormous benefits -it depends on the Game design, game caching and the like. I myself have had titles run smooth as butter on a RAID 0 config that didn’t on a single drive install. Again, that’s surprisingly dismissive to say some analysis by some tech site, some time ago debunked RAID 0’s effectiveness. There are way to many titles out there to conclude any such thing and remain respected.

            As far as the semi-production database response, well perhaps I was not relaying what I meant enough. What I was indicating is the world is being more and more digitized. Some of that data is precious and personal to the average Joe/Jane/Smith family. Therefore a Raid 5 setup can be a mainstream config within the next years due to the Fault Tolerance it offers. Whether it is Tax data, Family records, private videos/pictures, financial portfolio’s, online records, etc. Who knows perhaps Raid 5 may find itself in standard retail configurations in the upcoming years. Imagine seeing that at Best Buy or a Walmart! After all were basic on-board Raid offering’s on the standard motherboard(or store brand) in the 90’s? Was it even a standard 8 years ago? You may have found a few models at the time but the ‘times’ have changed because basic Raid 0,1 & Jbod is nearly found on all motherboards today. Albeit they are rather anemic if performance and capability.

            • Krogoth
            • 10 years ago

            RAID != back-up

            It only offers protection against HDD failure, but the data can still can corrupted/lost by other other things. Such as memory errors, malware, PBKAC, loosing DRM/encryption keys, forgetting passwords etc.

            IMO, RAID 1 is the only solution that makes any sense for mainstream for the fact is at least prevents data loss from HDD failure and reduces downtime to some degree. You should do “real” backups with valuable data.

        • Bensam123
        • 10 years ago

        Google perc 5/i.

          • KGA_ATT
          • 10 years ago

          One of Dell’s many products.

          I personally also like the MegaRAID SAS 9261-8i, MegaRAID SAS 9240-8i, LSI SAS 9211-8i, as well as some from ATTO. The LSI’s here range in price from $600 down to $250. But all are 6Gbs per port products so they appear to have that and higher on-board cache than the Dell/Perc product. I can see myself picking up one of these adapters within a couple of months. Perhaps around early April or so.

          Also LSI makes a Dual Port interface adapter for SATA Drives so they can behave somewhat like SAS drives. I am very interested in attempting that as well. Who knows, maybe i can ‘Dual Port’ four WD Caviar Green 2TB drives in Raid 0 or 10 and get ‘nasty’ speed out of them!

            • Bensam123
            • 10 years ago

            I personally pointed those two out because they’re available in surplus and for cheap off of eBay $120~. There is a big enthusiast community that supports them too.

            • KGA_ATT
            • 10 years ago

            I noticed some of the enthusiast support from my ‘yahoo’ query. -yep I’m one of the few still giving ‘props’ / support.

            It appears that the perc controller is very configurable from firmware to bios. That must be one of the reasons why there is an enthusiast crowd giving support. A Controller card that can be adapted to the masses.

    • Shining Arcanine
    • 10 years ago

    It would be nice if the tests were less Windows-centric and showed performance across a wider variety of platforms. I suggest doing benchmarks on (Gentoo) Linux using the Phoronix Test Suite.

    • Arag0n
    • 10 years ago

    MySQL and MSSQL benchmarks please would be nice.

      • wibeasley
      • 10 years ago

      I’m not sure the results would be that much better than Geoff’s existing IOMeter Database test -especially when most readers aren’t maintaining databases that are bottlenecking performance. How would you like to see it up? There are so many hardware scenarios, even if you don’t bother with enterprise things like SANs. Would you separate (1) data files, (2) log files, (3) index files, and (4) temp dbs on to different physical drives?

      And there are so many usage scenarios of reading, writing, locking and aggregating. How big of a database and how much RAM would you test with?

    • Mikael33
    • 10 years ago

    Newer games and par2 recovery times.

      • rUmX
      • 10 years ago

      PARchive recovery/creation is generally CPU bound, not disk. But this depends on the size of the set too. Try repairing just a few broken rar files from a 25GB Bluray movie. It takes …. a long time.

    • PeterD
    • 10 years ago

    I’ld like to know:
    – speed
    – durable (broken after 3 yrs? does it work longer? – bit difficutl to test, I know)
    – and how it works with raid: is speed affected? does the pc have problems with a raid config? (they do have, you know)

      • wibeasley
      • 10 years ago

      You want to know the proportion of drives (of a model that was released a month ago) that broke in the past three years?

        • 5150
        • 10 years ago

        Hey, he said it would be difficult. 😛

          • PeterD
          • 10 years ago

          So I did.

          To a certain degree it is possible to check durability.
          E.g. by running a device continuously during a certain period, and compare that with the actual amount of time people really use it.
          You can e.g. have write operations on a hd during 10 hours, and aftherwards keep in mind how many days people have to use their pc to actually reach 10 hours of write time.

            • Meadows
            • 10 years ago

            10 hours, oh my, call me Mary Sue and give me magic powers.

            10 hours won’t do anything to any drive. And I doubt TR has time to do more, or a time machine to pull that off. You’ll just have to go by the warranty.

            • wibeasley
            • 10 years ago

            If the failure rate was uniform over time, that would be an acceptable estimation. But I bet that the failure rate in a 10 hour stretch when the drive is 3 months old is much lower than in a 10 hour stretch when the drive is 3 years old.

            I also imagine you’d need hundreds of drives to estimate that acceptably, even if TR ran them for several months straight.

    • FuturePastNow
    • 10 years ago

    I want to hear your opinion of the latency/lag issues some cheap SSDs supposedly have in day-to-day use. I realize that is a subjective thing and not a benchmark you can start and walk away from, but it is important.

    • anotherengineer
    • 10 years ago

    Hmmm

    NOISE is always a good one and I’m sure will be included.

    How about a vibration test, I know an accelerometer isnt a coner store object, but maybe a subjective feel test?

    Game loading times is nice, BUT a big BUT, is there is not really any source games, CSS or HL2 or something would be a nice one to see (I know in css, the internets probly plays some roll)

    Common application launch time maybe

    +You know what would be interesting, running a quick sustained read/write test, with the sata controller set to native ide in bios and plain sata mode, see if there is a performance hit or not between the two settings. I know switching this in the bios probly means another re-install, so maybe just a few drives to see if there is a difference.

    • ew
    • 10 years ago

    For the “play at home” crowd it would be nice if most or all tests were publicly available.

    • Thanato
    • 10 years ago

    I’d like to see low cost SSD’s RAID’ed to compete with higher cost SSD, is it worth it?. Now that there are a lot of SSD’s for under 100 bucks I’m almost ready to buy one.

    • yes
    • 10 years ago

    Samsung Spintpoint F3 series. The 500gb is single platter and the 1tb is double. Maybe they’ll be recommended over the cavaiar blacks next time. . .

      • Dissonance
      • 10 years ago

      Coverage of the latest two-platter, 1TB drives is coming soon.

    • eternalmatt
    • 10 years ago

    I’d really like to see capability of USB 3.0 flash drives if you have any available to you.

    • MadManOriginal
    • 10 years ago

    This isn’t something I’d suggest for regular reviews but every once in a while it would be interesting to see how smaller capacity drives in a family compare to their larger brethren. Virtually every HD review out there uses flagship capacity drives which are often not the best $/GB or necessary for a given use. If smaller drives don’t maintain their relative ranks then flagship capaciuty drive reviews wouldn’t be useful when you don’t get a flagship capacity drive.

    • 5150
    • 10 years ago

    More SSD reviews.

    • Krogoth
    • 10 years ago

    Don’t bother with game load times.

    They are mostly CPU-limited.

    • novacatz
    • 10 years ago

    As there is currently no way to use TRIM on SSD drives in RAID I wouldn’t mind having a test that shows raid performance for fresh and used SSD raid sets to see the difference (and compare against single non-RAID with TRIM)

    • potatochobit
    • 10 years ago

    i want you to copy your steam folder back and forth between drives

    • fpsduck
    • 10 years ago

    Benchmark for any of RAID level configuration on SSD/Hard drives
    and RAID rebuild time/CPU usage.

    • dragmor
    • 10 years ago

    Time to install a usable PC. Windows 7 install time. Office install time, etc.

    Doesn’t matter when your doing 1 PC, but when your doing 20+ its a pain.

    • Hoser
    • 10 years ago

    Power draw at different loads. I’d like to know if these new drives are as green as the manufacturers say they are at both idle & load. And also to see the actual difference between the different categories of drives.

    • Prototyped
    • 10 years ago

    IOZone workloads.

    • balzi
    • 10 years ago

    if you really want real-world you might want to recind on the “Windows 7 only” thing. I don’t know how the pre-fetch and other fancy things work, but wouldn’t it mean that the W7 results were not relevant to WIN-XP systems?

    of course, if you can quickly prove that the results are applicable to both OSs, then you wouldn’t worry about testing both.

    note: I don’t actually expect you to test on both OSs, very time consuming, BUT I saw you say real-world and well.. .. ok, pendantry over!!

      • jackaroon
      • 10 years ago

      On the other hand, storage access patterns that do not include pre-fetch are not relevant to me. I don’t have an answer that suits everyone, I just figured I’d meet pedantry with pedantry. 🙂

      • JustAnEngineer
      • 10 years ago

      If you’re still using an operating system from 8½ years ago, you’re due for more than just a hard-drive upgrade.

      • OneArmedScissor
      • 10 years ago

      Only Windows 7 has TRIM and all anyone is going to care about is SSDs, anyways.

      I use XP but I couldn’t care less if they do only 7.

      • Flying Fox
      • 10 years ago

      Fragmentation may throw off the results though.

        • cygnus1
        • 10 years ago

        Doesn’t have to:

        -defrag the system just before starting the build (just to be sure)
        -image the system
        -run the build and time it a few times
        -restore the image and do it a few more times (you can reset the SSDs too if you want)

        include the avg, min and max times… and the point of the image is that it’s portable across the different drives

    • adisor19
    • 10 years ago

    I think launching multiple applications at the same time is a must, especially on boot up. Example :

    On boot up, start automatically :

    – Browser (Firefox)
    – IM program 1 (MSN)
    – Voip Program (Skype)
    – Mail Program (Thunderbird)
    – Calendar program (…)
    – Music player (iTunes)
    – Twitter program

    This is just an idea based on what i autostart on every boot, so feel free to choose difference programs or add/remove some.

    Adi

Pin It on Pinterest

Share This