RAID0 not keeping up with FRAPS

All things storage here: hard drives, DVD RW drives, little wicker baskets.

Moderators: morphine, Steel

RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 5:41 pm

i do quite a bit of FRAPS recordings. the drive that i'm recording to is a RAID0 array of 1TB seagate barracuda 7200.12 drives, which has a sustained write of over 210MB/s. a 60 FPS recording at 1920x1080 calls for somewhere around 120 MB/s, so my RAID0 shouldn't have any problems. however, i am getting pretty severe stuttering after a couple minutes recording, which indicates that the drive isn't keeping up.

does anyone know exactly what's wrong? or is FRAPS recording performance not determined by sequential write?
moriz
Gerbil XP
 
Posts: 445
Joined: Tue Aug 19, 2008 7:35 am

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 5:54 pm

Did you enable write caching? Maybe that would help. Defrag? Just spit balling here.
5150
Minister of Gerbil Affairs
 
Posts: 2387
Joined: Wed Jun 04, 2003 6:22 pm
Location: Sales Tax Is For Commies

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 5:56 pm

Was gonna suggest making sure the drive is well defragmented too :)
i5-2500k @ 4.5ghz | ASUS P8Z68-V | nVidia GTX780 | 32gb DDR3| 2x SSDs 2x HDD | Antec Solo II | Dell U2713HM
Mentawl
Gerbil XP
Silver subscriber
 
 
Posts: 446
Joined: Sun Dec 26, 2004 5:21 pm
Location: UK

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 6:06 pm

If FRAPS is recording raw 24-bit color pixels (never used it, so I don't know what the format of the data written to disk is...), your calculation is off by a factor of 3.

1920x1080 pixels x 3 bytes per pixel x 60 frames/sec = 373248000 bytes/sec, i.e. nearly 400 MB/sec.

Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37497
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 8:02 pm

just brew it! wrote:Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)


If Windows Vista or 7:

From an admin command prompt:

winsat disk -n X -seq -read -v

X = disk # on your system. Disk Management will tell you the disk number.

Image
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3506
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 9:20 pm

just brew it! wrote:If FRAPS is recording raw 24-bit color pixels (never used it, so I don't know what the format of the data written to disk is...), your calculation is off by a factor of 3.

1920x1080 pixels x 3 bytes per pixel x 60 frames/sec = 373248000 bytes/sec, i.e. nearly 400 MB/sec.

Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)


just checked a 1920x1200@60 FRAPS avi file. the total bitrate is 1658880 kbit/s, or 202.5 MB/s (assuming my math is right, of course). since 1920x1080 is 90% that of 1920x1200, it works out to be 182.25 MB/s. definitely greater than the 120 MB/s that everybody's been telling me. judging by the looks of it, i'll need faster drives :(

Ryu Connor wrote:
just brew it! wrote:Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)


If Windows Vista or 7:

From an admin command prompt:

winsat disk -n X -seq -read -v

X = disk # on your system. Disk Management will tell you the disk number.

Image


here's the result:

Image

so in light of my recent calculations, looks like my array isn't powerful enough, especially given how inconsistent it is. bummer. i wonder if it is worthwhile to get another drive for a triple RAID0, or replace them with something faster. i can get around this issue by using fancycache and using 8GB of RAM as a deferred write buffer (10 seconds defer), but it is not a particularly elegant solution. not to mention, it costs 8 GB of RAM. maybe if i chop it down to 4GB with 5 seconds defer...
moriz
Gerbil XP
 
Posts: 445
Joined: Tue Aug 19, 2008 7:35 am

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 10:45 pm

Cache will only help you until the cache fills up, then you are limited by the sequential write speed again.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37497
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 11:22 pm

the cache is also constantly clearing itself as well. the trick is to find a cache and defer amount that gives the drive enough time to catch up before the cache runs out. all i know is that with the 8GB cache and 10 second defer, the system can record for an hour without experiencing a single stutter. of course, i don't know what will happen if it needs to go longer than that, since i've never recorded that much.

let's do some math:

in 10 seconds, the cache accumulates 180.25 MB/s * 10 s = 1802.5 MB of data. at that point, the data start being written to the array at an absolute low of around 120 MB/s. this means that it takes at most 15 seconds for the array to write the same amount of data, which if sustained, requires the cache to retain an additional 5 seconds worth of data every 10 seconds, which works out to 901.25 MB. if this keeps up, the cache gets filled in... 7+1=8 cycles, or about 80 seconds. so obviously, the cache is insufficient when the array is nearly full.

if we use the average of 175 MB/s, the same setup results in an offset of 0.3 seconds, resulting in an accumulation of 54.075 MB per cycle, which means that the cache gets filled in 118+1=119 cycles, or about 19.83 minutes. again, not entirely sustainable.

theoretically, we can half the cache and half the deferred time. it all boils down if the array likes writing data in 901.25 MB chunks as much as it likes writing 1802.5 MB. we can also keep the same cache size but decrease the deferred time, which theoretically lowers the accumulation amount per cycle. again though, it depends on how well the array handles smaller data chunks. technically after the first large chunk, everything becomes pure sequential writes anyway, so it shouldn't make any difference in that regard.

obviously the easiest fix would be to add another drive, which technically increases its bare minimum performance up to 180 MB/s, assuming the Intel onboard RAID can keep up. i have an overclocked 2600k, so it SHOULD be alright.

am i doing my math right? or am i completely off base?
moriz
Gerbil XP
 
Posts: 445
Joined: Tue Aug 19, 2008 7:35 am

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 11:34 pm

moriz wrote:theoretically, we can half the cache and half the deferred time. it all boils down if the array likes writing data in 901.25 MB chunks as much as it likes writing 1802.5 MB. we can also keep the same cache size but decrease the deferred time, which theoretically lowers the accumulation amount per cycle.

It also cuts the time it takes to *fill* the cache in half, so you gain nothing.

901 MB vs 1802 MB isn't going to make a difference, since (as you've already surmised) it is all sequential once it starts streaming anyway.

You'd probably be better off setting the defer time much lower, so that the data can start streaming out to the drives as soon as there's a few tens of MB in the cache.

moriz wrote:obviously the easiest fix would be to add another drive, which technically increases its bare minimum performance up to 180 MB/s, assuming the Intel onboard RAID can keep up. i have an overclocked 2600k, so it SHOULD be alright.

Yeah, I *think* it should be able to keep up.

Also, don't discount the potential effects of fragmentation. If the RAID array is at all fragmented, your sequential transfer rates will go down the toilet.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37497
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: RAID0 not keeping up with FRAPS

Postposted on Tue Oct 11, 2011 11:54 pm

Apologies is this has already been mentioned, but why not save directly encoded, even if just to a lossless format? That should help ease the disk I/O.
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
morphine
Gerbil Khan
Silver subscriber
 
 
Posts: 9932
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Oct 12, 2011 12:02 am

i DO gain something though: 4 GB of RAM. granted, i have 16 GB and have no realistic use for it, so i suppose i'll try something like 8GB cache and 1 second deferred. that should results in an average accumulation of only 5.25MB per cycle, which means that at the average write speed of 175 MB/s, we will have 1526.05 seconds, or 25.43 minutes of uninterrupted recording. which is kinda strange, since i've recorded continuously for well over twice that amount and there was no stutter.

or i can just stop being cheap and get myself another drive. :S

as for fragmentation, the array spends most of its time EMPTY. i have split into into two partition, with the larger one occupying the outside tracks and the much smaller one on the inside tracks (1.5TB and 0.5TB respectively). the smaller partition is filled with mostly static data, and the partitioning itself should keep fragmentation to a minimum. then there's defragging every week and all.

morphine wrote:Apologies is this has already been mentioned, but why not save directly encoded, even if just to a lossless format? That should help ease the disk I/O.


technically, Fraps does encode the files. if we're dealing with true RAW files, as our head brewmaster had already mentioned, requires something around 400 MB/s incompressible sequential write. that's well into RAID0 vertex 3 pro performance, and i don't have the budget for it. if there's a way to compress the files AGAIN while simultaneously recording with Fraps, then i'm all ears.
moriz
Gerbil XP
 
Posts: 445
Joined: Tue Aug 19, 2008 7:35 am

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Oct 12, 2011 12:08 am

moriz wrote:i DO gain something though: 4 GB of RAM. granted, i have 16 GB and have no realistic use for it, so i suppose i'll try something like 8GB cache and 1 second deferred. that should results in an average accumulation of only 5.25MB per cycle, which means that at the average write speed of 175 MB/s, we will have 1526.05 seconds, or 25.43 minutes of uninterrupted recording. which is kinda strange, since i've recorded continuously for well over twice that amount and there was no stutter.

The disconnect between theory and practice could well be the difference between decimal and binary megabytes. Unless all of the numbers are using the same units, the data rate calculations could be off by several percent in either direction, and when you're close to the edge like this that could make a significant difference in how long it takes to exhaust your available cache.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37497
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Oct 12, 2011 12:13 am

morphine wrote:Apologies is this has already been mentioned, but why not save directly encoded, even if just to a lossless format? That should help ease the disk I/O.


Fraps does, exactly this. A low-compression lossless codec, specifically designed to compress screen grabs at a high rate of speed with low memory and CPU usage. They use the FourCC code FPS1.

It's possible, or it used to be, to trick Fraps into encoding to a buffer, have another codec reprocess that buffer, and write the re-compressed file, but it's prone to error, not easy to setup, and the gains are questionable.
I've been here long enough that I think I can forgo a signature.
Forge
Darth Gerbil
 
Posts: 7959
Joined: Wed Dec 26, 2001 7:00 pm
Location: SouthEast PA

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Oct 12, 2011 1:46 am

moriz wrote:if there's a way to compress the files AGAIN while simultaneously recording with Fraps, then i'm all ears.


In premise NTFS Compression might fit that bill. There's a chance that it won't bother trying to compress the already compressed video. It's meant to be a highspeed compression and it's heuristics are pretty smart. You'd have to test and see the behavior.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3506
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Oct 12, 2011 2:13 am

actually, i need something that can compress the files AGAIN before it even gets written to disk. otherwise, there's no benefit, since i'm not looking to save disk space; i have plenty of THAT fortunately.
moriz
Gerbil XP
 
Posts: 445
Joined: Tue Aug 19, 2008 7:35 am

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Oct 12, 2011 5:26 pm

You may want to look into getting an SSD if it's that big of an issue. My 120GB Corsair Force 3 is considered a pretty slow drive for being SATA III. I'm still getting 451.07 MB/s with the command Ryu gave. Two of these (with MIR) would cost about $300 US, but would be a huge speed increase in RAID0 over your Seagates.
Lenovo W520
IBM dx340
Nokia Lumia 928
Sony a7 with far too many lenses to list or even count
lonleyppl
Gerbil XP
 
Posts: 343
Joined: Wed Jan 26, 2011 2:59 pm

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Oct 12, 2011 6:28 pm

SSDs are unfortunately not a choice, since even two 120GB in RAID0 is too little space. not to mention, the easy fix in this situation is to drop in another 1TB seagate for $50. there's not much point in spending $350ish for SSDs.
moriz
Gerbil XP
 
Posts: 445
Joined: Tue Aug 19, 2008 7:35 am

Re: RAID0 not keeping up with FRAPS

Postposted on Wed Nov 16, 2011 9:11 am

what hard drive raid controller are you using? most on board controllers can't keep up with fast sustained writes. i'd say invest in an LSI 9260 raid card and get 4 WD RE4 drives. the enterprise level drives from seagate and western digital will do 80 to 140mbs in a single drive set up. Since you're doing a lot of HD recording i would use a dedicated true raid card, i know a decent one starts at $200 but the on board options suck for raid.
MCP MCDST MCSA MCTS MCITP
A+ Net+
Intel Core i7-950 Intel DX58SO Mobo 6GB Corsair XMS3 Tri-Channel BFG Geforce 260 GTX
2x 160GB Seagate HDs RAID 0 2x 500GB WD RE3 HDs RAID 0
Built 40K+ systems and still counting
EV42TMAN
Gerbil
 
Posts: 38
Joined: Fri Jun 10, 2011 11:50 am


Return to Storage

Who is online

Users browsing this forum: No registered users and 1 guest