Personal computing discussed

Moderators: renee, Flying Fox, Ryu Connor

 
jmc2
Gerbil XP
Topic Author
Posts: 364
Joined: Mon Aug 22, 2011 8:30 am

W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 1:02 pm

I found this "fsutil.exe behavior set memoryusage 2" but one person that was suppose to know said
that it was there ONLY for compatibility and does nothing.

In my testing it does nothing in caching file transfers....

I do a lot of video files 6-7 Gigs in a ramdrive and love how 80% of the file
"appears" to copy over in 2 seconds or so but then it slows down to 13 seconds total.

So I am guessing the cache is around 4 Gigs.
It would be really wonderful if I could get the WHOLE file into the cache
and out of my way in 2-3 seconds!
While I'm getting the other files ready to go.

Been doing a lot of googles and can not seem to find anything but the above useless command line.

(W7Pro,32G-ram,Intel 6 core)
Thoughts very welcome!
jmc2
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 1:27 pm

Did you reboot after running the fsutil command?
Nostalgia isn't what it used to be.
 
jmc2
Gerbil XP
Topic Author
Posts: 364
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 1:41 pm

Yep, rebooted and still 13 sec copy time. :(

jmc2
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 1:51 pm

How much RAM is available to to the OS? (I.e. how much of that 32GB is allocated to the ramdisk?)
Nostalgia isn't what it used to be.
 
maxxcool
Gerbil Elite
Posts: 855
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$
Contact:

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 2:02 pm

You are seeing disk cache burst and not windows cache burst. that 1st few seconds is the local disk drive cache sending a *instantaneous* sata-3 or sata-6 burst of 32-64megs of data ... this "throws off" the windows timer which only displays a "average" based on total transfer at the time of display.

windows it lying to you by lazy coding is all...

Even if you loaded the whole file into a ram drive you cannot exceed the write speed of the destination drive.
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 2:14 pm

maxxcool wrote:
You are seeing disk cache burst and not windows cache burst.

I'm not aware of any consumer hard drives that have a 4GB cache...
Nostalgia isn't what it used to be.
 
Forge
Lord High Gerbil
Posts: 8253
Joined: Wed Dec 26, 2001 7:00 pm
Location: Gone

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 2:26 pm

I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

Regardless, I don't think anyone here really has a solid handle on what the file transfer dialog is really saying. I know I've seen the same results on my big ram machines (16+ GB), and I've done some pondering, with no clear conclusions.
Please don't edit my signature for me. Thanks.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 2:31 pm

Forge wrote:
I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

OP wrote:
80% of the file "appears" to copy over in 2 seconds or so

So it is more than just projected completion time. *Something* is gobbling up the first few GB of the file rather quickly, and the OS's file cache is the only logical place it could be going.
Nostalgia isn't what it used to be.
 
Captain Ned
Global Moderator
Posts: 28704
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 2:36 pm

just brew it! wrote:
So it is more than just projected completion time. *Something* is gobbling up the first few GB of the file rather quickly, and the OS's file cache is the only logical place it could be going.

Sounds like time to grep through some really big Process Monitor logs. VMMap might come in handy as well.
What we have today is way too much pluribus and not enough unum.
 
SuperSpy
Minister of Gerbil Affairs
Posts: 2403
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 3:58 pm

Explorer on Win 7 and later will use a large amount of ram as a read buffer when doing a file copy. If you watch explorer's memory usage in taskman you will see it skyrocket around the same time it's reporting fast copy rates.
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: MacBook Pro 2017 2.9GHz | 16 GB | Radeon Pro 560
 
jmc2
Gerbil XP
Topic Author
Posts: 364
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 4:28 pm

just brew it! wrote:
How much RAM is available to to the OS? (I.e. how much of that 32GB is allocated to the ramdisk?)


8Gigs to W7 and 24 to the Ram drive.

Oh, did find that checking the "Turn off Windows write-cache buffer flushing"
in disk management-properties-policies dropped the transfer time from 16sec to 13sec.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 4:53 pm

jmc2 wrote:
8Gigs to W7 and 24 to the Ram drive.

Well I'll bet that's your issue right there. Windows isn't going to use *all* of the RAM available to it for write-behind cache. The OS itself and the working sets of any applications you're running all take up RAM, and the OS also keeps some percentage of the RAM in reserve to meet short-term demand.

My suggestion is to try dropping the size of the RAM disk to 20GB and see if that has an effect.
Nostalgia isn't what it used to be.
 
jmc2
Gerbil XP
Topic Author
Posts: 364
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 4:59 pm

just brew it! wrote:
Forge wrote:
I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

OP wrote:
80% of the file "appears" to copy over in 2 seconds or so

So it is more than just projected completion time. *Something* is gobbling up the first few GB of the file rather quickly, and the OS's file cache is the only logical place it could be going.


(Everything is done in the ramdrive and then copied to physical drive)

Yes, W7 and the file cache will be quietly transfering the file to the harddrive in the back ground for for a fair number
of seconds after it appears to be finished. I've watched the drive light continue to flicker for a while.
If I copy a 1 gig file over it appears "instantly" on the harddrive. The 6 gig file in 13 seconds must be going to a
ram cache for most of it.

The important point is that my video program "thinks" that it is finished and has saved the file and is ready to
accept the next file for processing long before the file is actually saved by W7!

My dream would be to get the 6-7Gig movies to appear to "instantly" be saved off like the 1 Gig files do.
If there was someway to double the W7 file cache I think that would take care of it.

But I guess 13 seconds for a 6+ Gig file is pretty good.

Thanks,
jmc2
 
jmc2
Gerbil XP
Topic Author
Posts: 364
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 5:14 pm

just brew it! wrote:
jmc2 wrote:
8Gigs to W7 and 24 to the Ram drive.

Well I'll bet that's your issue right there. Windows isn't going to use *all* of the RAM available to it for write-behind cache. The OS itself and the working sets of any applications you're running all take up RAM, and the OS also keeps some percentage of the RAM in reserve to meet short-term demand.

My suggestion is to try dropping the size of the RAM disk to 20GB and see if that has an effect.


Hmm, I had not thought of that... fingers crossed!

Boy, if that works I going to have to buy more ram. Working out of a Ramdrive has really spoiled me.
The times I have had to work with and wait on a physical Hd are painful. :)

thanks,
jmc2
 
maxxcool
Gerbil Elite
Posts: 855
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$
Contact:

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 5:27 pm

I think Maxxcool is indicating that the 4GB of super-speed transfer simply isn't happening. The disk cache gobbles the first tiny piece of the transfer so quickly that Windows projects the entire transfer at that speed, and then requires multiple uncached transactions before it revises the estimate downward.

Regardless, I don't think anyone here really has a solid handle on what the file transfer dialog is really saying. I know I've seen the same results on my big ram machines (16+ GB), and I've done some pondering, with no clear conclusions.


Much more eloquently put :)
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
 
Ryu Connor
Global Moderator
Posts: 4369
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 6:24 pm

If you have enough RAM, SuperFetch will cache entire large file transfers and then finish writing them to the disk in the background.
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
Captain Ned
Global Moderator
Posts: 28704
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 6:48 pm

Ryu Connor wrote:
If you have enough RAM, SuperFetch will cache entire large file transfers and then finish writing them to the disk in the background.

Keep your UPS up to snuff.

Sorry, but delayed writes just scare me.
What we have today is way too much pluribus and not enough unum.
 
Ryu Connor
Global Moderator
Posts: 4369
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 6:54 pm

A move wouldn't - typically - require such backfilling unless you're transitioning between partitions or disks. As it can just modify the master file table. Moving - in certain conditions - creates various ACL, EFS, and NTFS compression issues too, so there are more reasons to not use move.

A copy would leverage backfilling and if things go wrong, you still have the original file.

So I guess in short, don't move files?

If you fear data loss an UPS is pretty much a prerequisite regardless of your file transfer methods.
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
Captain Ned
Global Moderator
Posts: 28704
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 6:56 pm

Ryu Connor wrote:
If you fear data loss an UPS is pretty much a prerequisite regardless of your file transfer methods.

Most of that is the day job talking.

And yes, I have a UPS. If I wore suit pants I'd probably go belt & suspenders.
What we have today is way too much pluribus and not enough unum.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 7:31 pm

Ryu Connor wrote:
A move wouldn't - typically - require such backfilling unless you're transitioning between partitions or disks.

True; however, to get back to the OP's situation, he is taking a large file that has been staged to a ramdisk for processing, and copying it back to permanent storage. So by definition it is going to a different "disk".
Nostalgia isn't what it used to be.
 
UberGerbil
Grand Admiral Gerbil
Posts: 10368
Joined: Thu Jun 19, 2003 3:11 pm

Re: W7-Any way to increase "file cache size" for real?

Thu Jan 31, 2013 8:04 pm

just brew it! wrote:
True; however, to get back to the OP's situation, he is taking a large file that has been staged to a ramdisk for processing, and copying it back to permanent storage. So by definition it is going to a different "disk".
And since we're talking about a RAMDisk, concerns about delayed writes are the least of your reasons to be considering a UPS.
Captain Ned wrote:
If I wore suit pants I'd probably go belt & suspenders.
Honestly, the first time I read that I missed the word "suit" entirely.
 
jmc2
Gerbil XP
Topic Author
Posts: 364
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Fri Feb 01, 2013 11:11 am

Ryu Connor wrote:
If you have enough RAM, SuperFetch will cache entire large file transfers and then finish writing them to the disk in the background.


Hmm, something else to try. I have SuperFetch turned off. Will have to experiment with it.

The other experiment...
fsutil.exe behavior set memoryusage 2

I split my ram evenly 16G to W7 and 16 to Ram drive...Alas no change.
I watched the "available" memory in task manager and it goes from
around 15 Gig to 10.9 Gig when I start the 6 Gig file copy.

The "file copy bar" is gone in 13 seconds but the available memory
takes 30 something seconds to recover.

After I wrote this...

"Oh, did find that checking the "Turn off Windows write-cache buffer flushing"
in disk management-properties-policies dropped the transfer time from 16sec to 13sec."

I realized that I could find myself thinking that the file has been saved
and turn the computer off while it REALLY needs 30 something more seconds
to finish saving.

Am I correct in thinking that if I UNCHECK "Turn off Windows write-cache buffer flushing"
I will not lose anything by closing windows down. That it will continue and save the file
and actually close when it is safe to do so?

I'll accept the extra 3 seconds time penalty because I KNOW that at some point
that it will come up and bite me. :)

thanks,
jmc2
 
Ryu Connor
Global Moderator
Posts: 4369
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: W7-Any way to increase "file cache size" for real?

Fri Feb 01, 2013 11:56 am

That setting is very dangerous without an UPS. It lets the hard drive manage when data is actually written to the disk from the disk buffer. A power outage could result in substantial data loss (64MB on modern disks).

A shutdown does flush the buffer to disk.
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
jmc2
Gerbil XP
Topic Author
Posts: 364
Joined: Mon Aug 22, 2011 8:30 am

Re: W7-Any way to increase "file cache size" for real?

Fri Feb 01, 2013 12:55 pm

Ryu Connor wrote:
That setting is very dangerous without an UPS. It lets the hard drive manage when data is actually written to the disk from the disk buffer. A power outage could result in substantial data loss (64MB on modern disks).

A shutdown does flush the buffer to disk.


Ok, I have a good UPS.
So I can check the second option in policies for faster file copy time and
W7 will still safely flush everything if I try to shut it down too soon.
Great!

thanks,
jmc2
 
Forge
Lord High Gerbil
Posts: 8253
Joined: Wed Dec 26, 2001 7:00 pm
Location: Gone

Re: W7-Any way to increase "file cache size" for real?

Fri Feb 01, 2013 5:01 pm

Shutting Windows down cleanly would not be a problem for any of those settings. Part of shutting down is flushing all caches.

Sysinternals makes a nice little util called "sync" that will manually flush all buffers. It's a good tool to have around. It's one of the few things that Windows is flat out missing that almost every other OS has.
Please don't edit my signature for me. Thanks.
 
UberGerbil
Grand Admiral Gerbil
Posts: 10368
Joined: Thu Jun 19, 2003 3:11 pm

Re: W7-Any way to increase "file cache size" for real?

Fri Feb 01, 2013 5:13 pm

jmc2 wrote:
The "file copy bar" is gone in 13 seconds but the available memory
takes 30 something seconds to recover.
You shouldn't draw any conclusions from this. The OS won't reclaim the memory immediately (unless there is severe memory pressure, in which case you wouldn't see it return as available at all since something else would be gobbling it up). The memory manager shuffles pages between the various lists on a background thread, so you can't infer anything from the speed it goes back to being available. You really need to look at IO rather than memory if you're trying to figure when the transfer is "done"

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On