Share in Windows "Size vs Size on Disk" massive difference question.


egtrev

28 posts in this topic Last Reply

Recommended Posts

Hi,

 

I have a folder within a share in UNRAID that is mirrored with a local folder in Windows.

When I check the properties in Windows of both folders, the folder on UNRAID shows a massive difference between Size (150GB) and Size on Disk (200GB) the same mirrored folder locally on the Windows machine does not.

 

Is this normal? Something to worry about? Why is the Size on Disk so high for the folder on UNRAID? Thanks  :)

 

Pics of properties attached below.

 

 

 

Windows.png.9a935220486f1c90ba088a5395bca8c6.png

UNRAID.png.529e79917beaa68fe19d177c859f0fe7.png

Link to post
  • 2 weeks later...

I just realized my system is showing the same thing ... a few examples:

 

Size:  147GB  => Size on Disk:  204GB

 

Size:  90KB  =>  Size on Disk:  18.0MB

 

Size:  327GB  =>  Size on Disk:  517GB

 

I could, of course, delete one of the entire backup folders (e.g. the 327GB) ... look at the UnRAID used/free stats;  then re-copy it and look again to see the REAL amount of space it's using => but is there a Linux command I can simply run on the server that will show this?

 

I'm really surprised I'd never noticed this before -- I guess I'd not done a "Properties" of the folders before ... at least none that had this big discrepancy.    Not all folders have this problem -- any idea why??  [

 

 

Link to post

I just realized my system is showing the same thing ... a few examples:

 

Size:  147GB  => Size on Disk:  204GB

 

Size:  90KB  =>  Size on Disk:  18.0MB

 

Size:  327GB  =>  Size on Disk:  517GB

 

I could, of course, delete one of the entire backup folders (e.g. the 327GB) ... look at the UnRAID used/free stats;  then re-copy it and look again to see the REAL amount of space it's using => but is there a Linux command I can simply run on the server that will show this?

 

I'm really surprised I'd never noticed this before -- I guess I'd not done a "Properties" of the folders before ... at least none that had this big discrepancy.    Not all folders have this problem -- any idea why??  [

 

Pretty sure all folders have the "problem", just that it is not as visible depending on the file sizes and amount of files.

 

I assume there are 18 files in the 90KB folder? :)

Link to post

Yes, there are 18 files in the 90kb folder -- clearly Wndows "thinks" there's a 1MB min allocation size.

 

HOWEVER ... I was "fiddling" a bit and discovered that this is NOT a problem with Windows 7 -- only with Windows 10.

 

Not only does Windows 7 show the correct results ... but it completes a "Properties" window [i.e. right-click on a folder and select Properties]  DRAMATICALLY faster than Windows 10 does.  For example, the two results below show the same folder -- one done in Windows 10; on in Windows 7.  Windows 7 completed the Properties window in ~ 1 minute;  Windows 10 took nearly 10 minutes.  And as you can see, the Windows 7 result is correct.

 

 

Full_Bakup_Properties_-_W10.jpg.f36bb86f56b0dcf798b1dd12f2925a67.jpg

Full_Backup_Properties_-_W7.jpg.4d46609ba0fc2bc9313460c114428c6f.jpg

Link to post

HOWEVER ... I was "fiddling" a bit and discovered that this is NOT a problem with Windows 7 -- only with Windows 10.

Not only does Windows 7 show the correct results ... but it completes a "Properties" window [i.e. right-click on a folder and select Properties]  DRAMATICALLY faster than Windows 10 does. 

 

Yes, I've just tried Windows 7 and it's shows correct and completes way quicker as you have mentioned compared to Windows 10.

Strange!

Link to post

The difference between size and size on disk becomes far, far worse if there are hardlinks involved in the folder in question, because Windows calculates the size of each link thinking that its a separate file.  In the case of my Appdata share, Windows believes that the size on disk is ~1.8TB, which isn't too bad considering that my entire cache drive is a whopping 240GB

Link to post

This value is being reported by Samba wrong and because of it, if you check the properties of directory it can get wildly invalid, eg, 40MB of small files shows up as 1.5GB on disk.  The problem stems from this Samba parameter:

 

      allocation roundup size (S)

 

          This parameter allows an administrator to tune the allocation size reported to Windows

          clients. The default size of 1Mb generally results in improved Windows client performance.

          However, rounding the allocation size may cause difficulties for some applications, e.g. MS

          Visual Studio. If the MS Visual Studio compiler starts to crash with an internal error, set

          this parameter to zero for this share.

 

          The integer parameter specifies the roundup size in bytes.

 

          Default: allocation roundup size = 1048576

 

          Example: allocation roundup size = 0 # (to disable roundups)

 

Often there are doc errors in Samba so if we set this thing to 4K it may or may not result less "improved Windows client performance."

 

Some of you please add this line to your smb-extra.conf file (Settins/SMB Settings/SMB Extras):

 

allocation roundup size = 4096

 

or even

 

allocation roundup size = 0

 

and let me know if you notice any performance differences.

 

Link to post

Tom -- thanks for looking at this.

 

I tried changing the SMB allocation parameter -- set to both 4096 and then 0 with very similar results.

 

There's still a HUGE performance difference between Windows 7 and Windows 10, but at least the massive difference in "size on disk" is gone ... although interestingly it's still not shown as the same number in the two OS's.

 

The following results are from both Windows 7 and Windows 10 after changing the allocation parameter to 0.  Note that the huge difference is gone, but the results are still not the same in the two OS's.  Also, the time to generate the Properties window was about 15 seconds in Win7, and nearly 3 minutes in Win10 !!  Definitely STRANGE !!

 

 

 

BDocs_-_W10_-_Allloc_0.jpg.ab1f0ebf5bc49a3d84fb4144f09b8ce0.jpg

BDocs_-_W7_-_Alloc_0.jpg.2398d3d0abdb1095918a278b07f47b88.jpg

Link to post

... and yet another thought:  Changing the allocation roundup seems to effectively "fix" the "size on disk" issue.  But I have to wonder why there's such a huge difference in performance between '7 and '10.    Any ideas?  [Different SMB version perhaps?]  Any thoughts on a network configuration change in '10 that might help?

 

This is not, of course, a real problem => and changing the allocation roundup parameter effectively "fixes" the massive glitch in "size on disk" (although it's still strange that '7 and '10 give different results) ... but the performance difference between '7 and '10 is perplexing.  I have no idea why it takes over 10 times as long for '10 to compute the Properties window.

 

 

Link to post

...  In the case of my Appdata share, Windows believes that the size on disk is ~1.8TB, which isn't too bad considering that my entire cache drive is a whopping 240GB

 

Just for grins, does changing the allocation roundup to 0 resolve this for you?

 

Definitely interesting to see a 1.8TB "size on disk" on a 240GB disk  :)

Link to post

The "correct" setting for this should be:

 

allocation roundup size = 4096

 

because all our file systems used a 4K block size.  When set to 4096 do you see a big disparity between win7/win10?

 

I'm not so much interested in how long it takes to populate the Properties for a directory, instead I want to know what impact this setting might have on transfer rates.

 

Also, the way storage space is calculated is just how we do it on the Shares page (and how 'du' command does it) - by using 'stat' to fetch the size of each and every file in the directory.  This is going to produce a great deal of network traffic and can be affected by whether the inode for those files exists already in RAM on the server or not.

Link to post

I changed the setting to 4096 with the following results ...

 

"Size on Disk" is exactly the same as with a 0 setting -- i.e. it's fine.

 

But there's still a difference in the reported numbers for '7 and '10 ... definitely STRANGE.

 

The actual Size of the folder I used to test this is 158,161,341,405 bytes.  The "Size on Disk" is shown as 158,199,072,768 bytes in Windows 7, and 158,320,988,160 bytes in Windows 10.  Not significant, of course, but still strange that they'd be different.

 

Performance wise, there's still a BIG difference.  Windows 7 took 16 seconds to show the finished Properties page.  Windows 10 took 2:46  (166 seconds) ... or just over ten times as long.

 

This folder is on a v6.2.4 dual parity server.  I have one server still on v5.0.6 that has the same issue -- does changing the allocation rounding just require adding the "allocation roundup size = 4096"  line to the network.cfg file on the flash drive?    I presume making this change there will also resolve the issue.

 

Link to post

... I'm not so much interested in how long it takes to populate the Properties for a directory, instead I want to know what impact this setting might have on transfer rates.

 

I copied a 16GB file from the server to see if changing this setting had any impact ... and it doesn't seem to have changed anything.  Copied at ~ 110MB/s, which is what I typically see (basically network limited).

 

I agree how long it takes to populate the Properties window doesn't really matter -- it's a bit frustrating KNOWING that it's so much longer than it needs to be ... but not something I really EVER do, so it doesn't matter.  It IS nice to have the size disparity resolved (but also strange that the results are different in '7 and '10).

 

 

 

Link to post

Just for grins (with the allocation rounding set to 4096) ...

 

Vista populates the Properties window in the same time as Win7, and has exactly the same result.

 

XP actually populates it a bit quicker (about 12 seconds) ... and has exactly the same result as '7 and Vista.

 

Windows 98 takes 40 seconds to populate the Properties window, and has yet-another result for "on disk" !!  (158,179,085,312)

 

Windows ME takes 45 seconds to populate the window, and has the same result as '98

 

Windows 2000 Pro takes 12 seconds, and has the same result as XP, Vista, and '7

 

Windows 95 doesn't work correctly with a folder that large -- displays the # of files and folders okay, but totally hoses the size of the folder (and doesn't show a "size on disk").

 

 

So ... excluding the ancient Win 9x OS's, all Windows OS's from 2000 - Windows 7 work very nicely and have identical results;  Windows 10 gives slightly different results and takes ten times as long to produce them.

 

 

 

 

 

 

Link to post

...  I have one server still on v5.0.6 that has the same issue -- does changing the allocation rounding just require adding the "allocation roundup size = 4096"  line to the network.cfg file on the flash drive?

 

Answered my own question by just trying it -- this does NOT change the results, so I assume there's some other way to change the parameter on v5.  So ... what do I have to change ??

 

Link to post

...  I have one server still on v5.0.6 that has the same issue -- does changing the allocation rounding just require adding the "allocation roundup size = 4096"  line to the network.cfg file on the flash drive?

 

Answered my own question by just trying it -- this does NOT change the results, so I assume there's some other way to change the parameter on v5.  So ... what do I have to change ??

 

Wow thanks for the detailed analysis!  Probably there's a simple setting somewhere to speed up win10.

 

For unRaid-5 you can type 'smbd -v' to get the samba version number and then try to find the smb.conf man page for that version; could be they didn't support that setting then...

Link to post
  • 6 months later...
On 1/2/2017 at 1:32 PM, limetech said:

The "correct" setting for this should be:

 

allocation roundup size = 4096

 

 

Confirmed, this makes Windows 10 report as expected now 4K size (it was previously using 1MB). UnRaid is version 6.4rc7a, smbd v4.6.6

Link to post
  • 4 months later...

I can confirm this change works! This should be a setting that is made in unRAID when it is installed. Because default: allocation roundup size = 1048576 creates problems in shares when trying to copy files and reports wrong size used in Windows. 

 

I tried to copy 800GB of data to my unRAID server but got the message that the data would not fit even though i had 2TB free.. This made me find this thread... The correct setting should be: 

allocation roundup size = 4096

 

Setting this was a little unclear cause another guy talked about changing in a file on the USB... 

 

Here is the correct way to change it (It's often not explained in this forum i have noticed):

In the unRAID web GUI (v6.3.5) click Settings -> SMB (under Network Services) then you add "allocation roundup size = 4096"  (without the " ") in the text box "Samba extra configuration".. 

 

After that im not 100% sure if the settings will take effect or if you have to stop and then start the array.. I ended up with rebooting unraid after that it is working correctly! :) Finally!

 

Maybe someone could report this as a bug (or change request)? Or I will when i get the time in a few weeks or so... 

 

 

unRAID.JPG

 

Below you can see the that the allocation size has changed.. Size on disk: now shows 4KiB instead of 1MiB! :)

 

unRAID2.JPG

 

Edited by Meph
Link to post
  • 1 month later...

This is fantastic!  I noticed this when I first setup unRAID a few years ago but searches at that time didn't provide a solution.  I can confirm that the solution provided using Meph's method of applying it works like a champ.  The only other thing I had to do was stop the array and then start it again.  I did not have to reboot.  Thanks for this!  Been driving me nuts :).

Link to post
  • 1 month later...
On 1/3/2017 at 1:19 AM, limetech said:

 

Wow thanks for the detailed analysis!  Probably there's a simple setting somewhere to speed up win10.

 

For unRaid-5 you can type 'smbd -v' to get the samba version number and then try to find the smb.conf man page for that version; could be they didn't support that setting then...

 

I'm having a similar issue here

 

What has the solution at the end? it was fixed in unraid?

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.