TR Forums

Hi all, just registered on the forum.

I had a new Samsung EVO 850 SSD for less than a year, and dedicated one of its partitions to virtual machine. I assigned 365GB to that partition. No other system files or features (e.g. System Restore, Windows Search etc) were turned on on that partition. 

Yesterday I ran into a really weird problem. I was running the VM and copying files to it, and VMWare warned me that the disk ran out of space. Windows showed zero bytes available, but the size of VM on the disk was only 301GB. I ran tools like WinDirStat and they all said I got a total of ~300GB of files on that disk. Can anyone please help me with this? Where could that ~60GB of space go?

Windows counts Gibibytes (GiB).  Hard-drive manufacturers count Gigibytes (GB).  1 GiB = 1.073741824 GB.

JustAnEngineer wrote:

Windows counts Gibibytes (GiB).  Hard-drive manufacturers count Gigibytes (GB).  1 GiB = 1.073741824 GB.

This is why your space ship unexpected crashes or comes out of warp in the middle of nowhere.

Tech: Captain, Windows shows zero bytes but we should still have 60GB!

*space ship rematerializes in the middle of a sun*

Computer: BSOD alert, BSOD alert!

Captain: REBOOT!!!

Decimal vs binary GB isn't enough to explain this though. There must be a hidden file or folder taking up space.

bloomfielderic wrote:

dedicated one of its partitions to virtual machine. I assigned 365GB to that partition. No other system files or features (e.g. System Restore, Windows Search etc) were turned on on that partition.

bloomfielderic wrote:

the size of VM on the disk was only 301GB

I am a bit confused by the above. Are you mounting the host partition/disk directly from inside the VM, or did you use a .vmdk file with a max allocation of 365GB? If you are mounting the partition directly and can see the files from the host, then "size of VM on the disk" does not make sense here...

JustAnEngineer wrote:

Windows counts Gibibytes (GiB).  Hard-drive manufacturers count Gigibytes (GB).  1 GiB = 1.073741824 GB.

The GBs mentioned above are all in Windows terms.

Flying Fox wrote:

bloomfielderic wrote:

dedicated one of its partitions to virtual machine. I assigned 365GB to that partition. No other system files or features (e.g. System Restore, Windows Search etc) were turned on on that partition.

bloomfielderic wrote:

the size of VM on the disk was only 301GB

I am a bit confused by the above.  Are you mounting the host partition/disk directly from inside the VM, or did you use a .vmdk file with a max allocation of 365GB?  If you are mounting the partition directly and can see the files from the host, then "size of VM on the disk" does not make sense here...  :o  :oops:

Just meant all is resided on that partition is a VM(vmdk file, didn't preallocate space though I set the max. capacity to 320GB <- still less than 365GB). The error message appeared in the middle of a VM session. Now the VM is force shutdown, and cannot finish bootup coz of no available space.
Sorry for the confusion.

Samsung magician takes 10 gb per 120 to a thing called overprovisioning, probably the 30 gb you need are the ones that samsung is taking to increase the lifespan of your ssd, as my 120 gb drive takes 11 gb by default, yours probably is taking 30 gb.

Did you set Windows Explorer to show system and hidden files and see if anything else shows up? It really sounds like something (maybe a swap file?) got created on the drive and is chewing up the additional space.

joselillo_25 wrote:

Samsung magician takes 10 gb per 120 to a thing called overprovisioning, probably the 30 gb you need are the ones that samsung is taking to increase the lifespan of your ssd, as my 120 gb drive takes 11 gb by default, yours probably is taking 30 gb.

I don't think I have that enabled.

just brew it! wrote:

Did you set Windows Explorer to show system and hidden files and see if anything else shows up? It really sounds like something (maybe a swap file?) got created on the drive and is chewing up the additional space.

Yes. I didn't put any system file on that partition. It is solely for VM.

Kind of off-topic, but why did you even make a separate partition?

On-topic: Maybe some screenshots of all these dirs and Samsung Magician and such would help.

bloomfielderic wrote:

just brew it! wrote:

Did you set Windows Explorer to show system and hidden files and see if anything else shows up? It really sounds like something (maybe a swap file?) got created on the drive and is chewing up the additional space.

Yes. I didn't put any system file on that partition. It is solely for VM.

Are you sure the system didn't put something there behind your back?

just brew it! wrote:

bloomfielderic wrote:

just brew it! wrote:

Did you set Windows Explorer to show system and hidden files and see if anything else shows up? It really sounds like something (maybe a swap file?) got created on the drive and is chewing up the additional space.

Yes. I didn't put any system file on that partition. It is solely for VM.

Are you sure the system didn't put something there behind your back?

Damn hippies... always blaming the system...
As someone else mentioned, it's very likely that your VM software is creating an additional space for the VMDK to support growth.  Look for a hidden file on the partition.

I'm failing to parse this right, can someone help me out?

OP created a partition specifically for use with this VM (presumably NTFS). The only thing that OP put in this partition is a dynamically allocated VMDK image file, theoretically capped at 300GB. It sounds like there is Windows VM in that image, and nothing said so far about the partition scheme within.

Is the problem with the host partition? Or a partition inside the VM?

TwistedKestrel wrote:

I'm failing to parse this right, can someone help me out?

OP created a partition specifically for use with this VM (presumably NTFS). The only thing that OP put in this partition is a dynamically allocated VMDK image file, theoretically capped at 300GB. It sounds like there is Windows VM in that image, and nothing said so far about the partition scheme within.

Is the problem with the host partition? Or a partition inside the VM?

It's not clear to me what the OS of the actual VM is, I think Windows is the host OS. I don't think any partitioning was done WITHIN the VM...

Maybe the dynamically sized VM wrote to some blocks and the SSD didn't have time to TRIM/GC so Windows saw the space as used. That's my best guess at this point.

Unless you have some specific reason for creating additional partitions within physical disks, you might try avoiding that next time as to have the full space at your disposal.

Maybe VMware bungled pre-allocating space, or didn't finish a write transaction or something?

OK, so it sounds like the NTFS file system's internal bookkeeping data was in an inconsistent state. I'd say the two most likely culprits are:

- Host system crashed or lost power while the VM was running, or shortly after VM was running and shut down. This could cause the file system on which the VM is stored to be left in an inconsistent state. Normally, the NTFS journal should allow the system to recover automatically the next time the system starts up; but depending on how the NTFS journaling interacts with the drive's internal cache, it's possible that this recovery could fail. This is especially true in the case of a power loss, since anything still "in flight" that the drive hasn't yet committed to flash gets lost.

- Hardware problems (bad RAM, flaky SATA controller, failing SSD) causing corruption of the contents of the SSD. If you're not noticing any other random stability issues with the host or guest systems, I'd say this scenario is somewhat unlikely.

Now, to address a few of the questions/issues this raises:

- The reason your other tools like diskpart, Samsung Magician, and WinDirStat didn't detect the problem is that none of these tools are designed to detect or correct inconsistencies in the NTFS file system; that's chkdsk's job! In the case of diskpart, it doesn't know (or care), since it is mainly concerned with raw disk partitions. Samsung Magician is mainly concerned with the health of the SSD itself, not the file system created on it by your OS. And WinDirStat likely didn't spot the issue because it is simply trusting the very file system meta-data that got corrupted by this glitch.

- This is not a TRIM issue. TRIM tells the *drive* that the contents of a disk block are no longer valid. This is related to (driven by), but separate from NTFS's notion of which blocks are allocated or free. Failure of the drive to properly TRIM/GC would result in a performance penalty, but would not cause NTFS to lose track of the freed blocks since that is done separately.

Edit: This sort of file system corruption was quite common back in the pre-NTFS (FAT/FAT32) days. Since NTFS was introduced it is fairly rare, as NTFS is *usually* quite good at preventing the file system meta-data from becoming inconsistent. But no file system is 100% bullet-proof...

just brew it! wrote:

OK, so it sounds like the NTFS file system's internal bookkeeping data was in an inconsistent state. 

Exactly. And it is possible apps can cause it. When I was teaching myself how to use Robocopy, after one syntax/command error I belatedly noticed the target SSD was "empty" according to windows, but up to 10GB was taken by an invisible file. Coincidentally the same size as the movie file I was using as a test file. Nothing would reveal it so I just quick reformatted the SSD to get the space back. 

Kougar wrote:

just brew it! wrote:

OK, so it sounds like the NTFS file system's internal bookkeeping data was in an inconsistent state. 

Exactly. And it is possible apps can cause it. When I was teaching myself how to use Robocopy, after one syntax/command error I belatedly noticed the target SSD was "empty" according to windows, but up to 10GB was taken by an invisible file. Coincidentally the same size as the movie file I was using as a test file. Nothing would reveal it so I just quick reformatted the SSD to get the space back. 

Sounds like maybe you got "had" by this issue.

It should not be possible for an application to corrupt the internal file system meta-data.

just brew it! wrote:

Sounds like maybe you got "had" bythis issue.

It should not be possible for an application to corrupt the internal file system meta-data.

Ohhh, that must've been it. I had show hidden files/folders enabled but I do also have it set to hide protected system files to keep the clutter down. The latter didn't even occur to me. Thanks for the link!

bloomfielderic wrote:

Thanks to everyone who replied... Yes TwistedKestrel you are right. The problem is in the host OS.

I kind of gave up and just ran chkdsk to see what happens and it magically identified the "hidden" free sectors and bam all the free space was back. Now I just got my 60GB back out of the blue...

Can anyone explain to me what exactly happened? I am overwhelmed right now...

Recap: host OS Win10 Pro, dedicated one of the SSD partitions to a VM, the VM's max capacity was capped at 320GB (dynamic file size), the partition itself was 365GB, hadnt enabled anything on that disk except NTFS compression, which meant no pagefile, no hibernation, no Windows Search etc. (I put them in another SSD).
-> Was running VM and in the middle the host OS said the partition ran out of space. Went to Explorer and it showed 0 bytes free. VM shut down because of this and could not boot up ever since. Explorer showed the total size of all files on the partition being ~300GB.
-> Ran WinDirStat and all sorts of other software. No hidden files spotted. Nothing but the files for VM.
-> Didn't have any over provisioning enabled in Samsung Magician.
-> After assigning and formatting the partition it was used solely for VM. Didn't put any other files onto it.
-> Ran chkdsk, and problem solved. Free sectors were identified.

How come the host OS screwed up with identifying free sectors? Before running chkdsk all other tools (diskpart, etc.) were all giving false information...

As a performance optimization, VMware can store the page file of the guest OS outside of the VMDK file. I believe this is the default behavior in current versions. Thus, you could easily consume additional storage up to the guest's maximum page file size. This affects all VMs to some extent, but it is limited to maximum page file size specified in the guest OS. Thus, it is not likely to explain the 60+ GB entirely.

Snapshots also eat up a considerable amount of disk space if you use them. Best practice is to use them sparingly and delete/consolidate frequently.

Someone implicated trim as a potential cause; this cannot be the case. Trim happens below the file system, and the OS will see that space as available regardless of whether the drive trimmed those sectors.

If chkdsk freed a significant amount of that space, then your host file system was corrupt. If the problem persists, you may need to look at your underlying disks/controller---both for proper configuration and signs of defect. Write caching and write coalescing are good for performance, but a sudden halt is more likely to cause issues with those features enabled.