W7-Any way to increase "file cache size" for real?

Monopoly money comes in many flavors: 7, Vista, XP, 2K, ME, 98, etc.

Moderators: Flying Fox, Ryu Connor

W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 1:02 pm

I found this "fsutil.exe behavior set memoryusage 2" but one person that was suppose to know said
that it was there ONLY for compatibility and does nothing.

In my testing it does nothing in caching file transfers....

I do a lot of video files 6-7 Gigs in a ramdrive and love how 80% of the file
"appears" to copy over in 2 seconds or so but then it slows down to 13 seconds total.

So I am guessing the cache is around 4 Gigs.
It would be really wonderful if I could get the WHOLE file into the cache
and out of my way in 2-3 seconds!
While I'm getting the other files ready to go.

Been doing a lot of googles and can not seem to find anything but the above useless command line.

(W7Pro,32G-ram,Intel 6 core)
Thoughts very welcome!
jmc2
jmc2
Gerbil
 
Posts: 82
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 1:27 pm

Did you reboot after running the fsutil command?
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37505
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 1:41 pm

Yep, rebooted and still 13 sec copy time. :(

jmc2
jmc2
Gerbil
 
Posts: 82
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 1:51 pm

How much RAM is available to to the OS? (I.e. how much of that 32GB is allocated to the ramdisk?)
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37505
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 2:02 pm

You are seeing disk cache burst and not windows cache burst. that 1st few seconds is the local disk drive cache sending a *instantaneous* sata-3 or sata-6 burst of 32-64megs of data ... this "throws off" the windows timer which only displays a "average" based on total transfer at the time of display.

windows it lying to you by lazy coding is all...

Even if you loaded the whole file into a ram drive you cannot exceed the write speed of the destination drive.
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 643
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 2:14 pm

maxxcool wrote:You are seeing disk cache burst and not windows cache burst.

I'm not aware of any consumer hard drives that have a 4GB cache...
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37505
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 2:26 pm

I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

Regardless, I don't think anyone here really has a solid handle on what the file transfer dialog is really saying. I know I've seen the same results on my big ram machines (16+ GB), and I've done some pondering, with no clear conclusions.
I've been here long enough that I think I can forgo a signature.
Forge
Darth Gerbil
 
Posts: 7959
Joined: Wed Dec 26, 2001 7:00 pm
Location: SouthEast PA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 2:31 pm

Forge wrote:I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

OP wrote:80% of the file "appears" to copy over in 2 seconds or so

So it is more than just projected completion time. *Something* is gobbling up the first few GB of the file rather quickly, and the OS's file cache is the only logical place it could be going.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37505
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 2:36 pm

just brew it! wrote:So it is more than just projected completion time. *Something* is gobbling up the first few GB of the file rather quickly, and the OS's file cache is the only logical place it could be going.

Sounds like time to grep through some really big Process Monitor logs. VMMap might come in handy as well.
It is one of the blessings of old friends that you can afford to be stupid with them. Ralph Waldo Emerson.
Captain Ned
Global Moderator
Gold subscriber
 
 
Posts: 20112
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 3:58 pm

Explorer on Win 7 and later will use a large amount of ram as a read buffer when doing a file copy. If you watch explorer's memory usage in taskman you will see it skyrocket around the same time it's reporting fast copy rates.
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1545
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 4:28 pm

just brew it! wrote:How much RAM is available to to the OS? (I.e. how much of that 32GB is allocated to the ramdisk?)


8Gigs to W7 and 24 to the Ram drive.

Oh, did find that checking the "Turn off Windows write-cache buffer flushing"
in disk management-properties-policies dropped the transfer time from 16sec to 13sec.
jmc2
Gerbil
 
Posts: 82
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 4:53 pm

jmc2 wrote:8Gigs to W7 and 24 to the Ram drive.

Well I'll bet that's your issue right there. Windows isn't going to use *all* of the RAM available to it for write-behind cache. The OS itself and the working sets of any applications you're running all take up RAM, and the OS also keeps some percentage of the RAM in reserve to meet short-term demand.

My suggestion is to try dropping the size of the RAM disk to 20GB and see if that has an effect.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37505
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 4:59 pm

just brew it! wrote:
Forge wrote:I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

OP wrote:80% of the file "appears" to copy over in 2 seconds or so

So it is more than just projected completion time. *Something* is gobbling up the first few GB of the file rather quickly, and the OS's file cache is the only logical place it could be going.


(Everything is done in the ramdrive and then copied to physical drive)

Yes, W7 and the file cache will be quietly transfering the file to the harddrive in the back ground for for a fair number
of seconds after it appears to be finished. I've watched the drive light continue to flicker for a while.
If I copy a 1 gig file over it appears "instantly" on the harddrive. The 6 gig file in 13 seconds must be going to a
ram cache for most of it.

The important point is that my video program "thinks" that it is finished and has saved the file and is ready to
accept the next file for processing long before the file is actually saved by W7!

My dream would be to get the 6-7Gig movies to appear to "instantly" be saved off like the 1 Gig files do.
If there was someway to double the W7 file cache I think that would take care of it.

But I guess 13 seconds for a 6+ Gig file is pretty good.

Thanks,
jmc2
jmc2
Gerbil
 
Posts: 82
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 5:14 pm

just brew it! wrote:
jmc2 wrote:8Gigs to W7 and 24 to the Ram drive.

Well I'll bet that's your issue right there. Windows isn't going to use *all* of the RAM available to it for write-behind cache. The OS itself and the working sets of any applications you're running all take up RAM, and the OS also keeps some percentage of the RAM in reserve to meet short-term demand.

My suggestion is to try dropping the size of the RAM disk to 20GB and see if that has an effect.


Hmm, I had not thought of that... fingers crossed!

Boy, if that works I going to have to buy more ram. Working out of a Ramdrive has really spoiled me.
The times I have had to work with and wait on a physical Hd are painful. :)

thanks,
jmc2
jmc2
Gerbil
 
Posts: 82
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 5:27 pm

I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

Regardless, I don't think anyone here really has a solid handle on what the file transfer dialog is really saying. I know I've seen the same results on my big ram machines (16+ GB), and I've done some pondering, with no clear conclusions.


Much more eloquently put :)
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 643
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 6:24 pm

If you have enough RAM, SuperFetch will cache entire large file transfers and then finish writing them to the disk in the background.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3506
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 6:48 pm

Ryu Connor wrote:If you have enough RAM, SuperFetch will cache entire large file transfers and then finish writing them to the disk in the background.

Keep your UPS up to snuff.

Sorry, but delayed writes just scare me.
It is one of the blessings of old friends that you can afford to be stupid with them. Ralph Waldo Emerson.
Captain Ned
Global Moderator
Gold subscriber
 
 
Posts: 20112
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 6:54 pm

A move wouldn't - typically - require such backfilling unless you're transitioning between partitions or disks. As it can just modify the master file table. Moving - in certain conditions - creates various ACL, EFS, and NTFS compression issues too, so there are more reasons to not use move.

A copy would leverage backfilling and if things go wrong, you still have the original file.

So I guess in short, don't move files?

If you fear data loss an UPS is pretty much a prerequisite regardless of your file transfer methods.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3506
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 6:56 pm

Ryu Connor wrote:If you fear data loss an UPS is pretty much a prerequisite regardless of your file transfer methods.

Most of that is the day job talking.

And yes, I have a UPS. If I wore suit pants I'd probably go belt & suspenders.
It is one of the blessings of old friends that you can afford to be stupid with them. Ralph Waldo Emerson.
Captain Ned
Global Moderator
Gold subscriber
 
 
Posts: 20112
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 7:31 pm

Ryu Connor wrote:A move wouldn't - typically - require such backfilling unless you're transitioning between partitions or disks.

True; however, to get back to the OP's situation, he is taking a large file that has been staged to a ramdisk for processing, and copying it back to permanent storage. So by definition it is going to a different "disk".
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37505
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Postposted on Thu Jan 31, 2013 8:04 pm

just brew it! wrote:True; however, to get back to the OP's situation, he is taking a large file that has been staged to a ramdisk for processing, and copying it back to permanent storage. So by definition it is going to a different "disk".
And since we're talking about a RAMDisk, concerns about delayed writes are the least of your reasons to be considering a UPS.
Captain Ned wrote:If I wore suit pants I'd probably go belt & suspenders.
Honestly, the first time I read that I missed the word "suit" entirely.
UberGerbil
Gerbil Khan
 
Posts: 9971
Joined: Thu Jun 19, 2003 3:11 pm

Re: W7-Any way to increase "file cache size" for real?

Postposted on Fri Feb 01, 2013 11:11 am

Ryu Connor wrote:If you have enough RAM, SuperFetch will cache entire large file transfers and then finish writing them to the disk in the background.


Hmm, something else to try. I have SuperFetch turned off. Will have to experiment with it.

The other experiment...
fsutil.exe behavior set memoryusage 2

I split my ram evenly 16G to W7 and 16 to Ram drive...Alas no change.
I watched the "available" memory in task manager and it goes from
around 15 Gig to 10.9 Gig when I start the 6 Gig file copy.

The "file copy bar" is gone in 13 seconds but the available memory
takes 30 something seconds to recover.

After I wrote this...

"Oh, did find that checking the "Turn off Windows write-cache buffer flushing"
in disk management-properties-policies dropped the transfer time from 16sec to 13sec."

I realized that I could find myself thinking that the file has been saved
and turn the computer off while it REALLY needs 30 something more seconds
to finish saving.

Am I correct in thinking that if I UNCHECK "Turn off Windows write-cache buffer flushing"
I will not lose anything by closing windows down. That it will continue and save the file
and actually close when it is safe to do so?

I'll accept the extra 3 seconds time penalty because I KNOW that at some point
that it will come up and bite me. :)

thanks,
jmc2
jmc2
Gerbil
 
Posts: 82
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Postposted on Fri Feb 01, 2013 11:56 am

That setting is very dangerous without an UPS. It lets the hard drive manage when data is actually written to the disk from the disk buffer. A power outage could result in substantial data loss (64MB on modern disks).

A shutdown does flush the buffer to disk.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3506
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Fri Feb 01, 2013 12:55 pm

Ryu Connor wrote:That setting is very dangerous without an UPS. It lets the hard drive manage when data is actually written to the disk from the disk buffer. A power outage could result in substantial data loss (64MB on modern disks).

A shutdown does flush the buffer to disk.


Ok, I have a good UPS.
So I can check the second option in policies for faster file copy time and
W7 will still safely flush everything if I try to shut it down too soon.
Great!

thanks,
jmc2
jmc2
Gerbil
 
Posts: 82
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Postposted on Fri Feb 01, 2013 5:01 pm

Shutting Windows down cleanly would not be a problem for any of those settings. Part of shutting down is flushing all caches.

Sysinternals makes a nice little util called "sync" that will manually flush all buffers. It's a good tool to have around. It's one of the few things that Windows is flat out missing that almost every other OS has.
I've been here long enough that I think I can forgo a signature.
Forge
Darth Gerbil
 
Posts: 7959
Joined: Wed Dec 26, 2001 7:00 pm
Location: SouthEast PA

Re: W7-Any way to increase "file cache size" for real?

Postposted on Fri Feb 01, 2013 5:13 pm

jmc2 wrote:The "file copy bar" is gone in 13 seconds but the available memory
takes 30 something seconds to recover.
You shouldn't draw any conclusions from this. The OS won't reclaim the memory immediately (unless there is severe memory pressure, in which case you wouldn't see it return as available at all since something else would be gobbling it up). The memory manager shuffles pages between the various lists on a background thread, so you can't infer anything from the speed it goes back to being available. You really need to look at IO rather than memory if you're trying to figure when the transfer is "done"
UberGerbil
Gerbil Khan
 
Posts: 9971
Joined: Thu Jun 19, 2003 3:11 pm


Return to Windows

Who is online

Users browsing this forum: No registered users and 2 guests