400GB of actual files takes up 729GB of space on disk... What's going on here!

[webrob123]

I've installed unRAID rc3 and I'm having problems. Size on disk is way to big.

 

I copied some files off an NTFS disk over the network from a Windows 8.1 PC.

 

I had a problem where it seemed like my directory tree was too deep for the target drive and so I had to cancel and start again. I told Windows to skip the files that had already been copied.

 

Apart from the fact that I'm concerned that the directory depth issue caused the copy to stall (can someone explain if another format would avoid this?), why am I seeing the following on the disk I've copied to in unRAID:

 

http://imgur.com/N4d1PIH

 

I went with the XFS defaults when formatting 3TB WD Red disk that is in the drive. See screenshot for details:

 

http://imgur.com/zpumInX

 

Keen to get things running with unRAID, but with this and my GPU passthorugh problems I'm starting to doubt if unRAID is a good choice at the moment.

 

P.S. I should also mention that I'm running without a parity drive or a cache drive at the moment (i.e. just UnRaid OS USB and the 3TB as disk1). Will add SSD cache drive and larger 4TB Hitachi parity drive if I can get things running smoothly.

[Squid]

http://www.howtogeek.com/180369/why-is-there-a-big-difference-between-size-and-size-on-disk/

[webrob123]

I'm aware that block sizes on disk can cause a discrepancy. But over 300GB extra is being used here, I've never encountered anything near this sort of  difference on NTFS, HFS+, EXT2 or EXT4. Have I just been lucky so far or is it something to do with XFS. Me thinks it is the latter, but I'm happy to be proved wrong.

 

Also any ideas why XFS seems to have triggered a maximum directory depth issue that I've never had on any of the above formats? Good be unlucky here as it was an extremely long path - sadly one created by a UE4 project and so I can't really change it.

 

 

[sparklyballs]

I've never encountered anything near this sort of  difference on NTFS, HFS+, EXT2 or EXT4.

 

 

only relevant if it were the exact same data everytime

 

 

[webrob123]

Sparkyballs: of course you're factually correct here. I am going off instinct to some extent and feel that a not far off doubling of size on disk compared to the original data is something I've never come across before. This is the first time I've used XFS for any volume and so I wondered if that had anything to do with it. I must have copied at least 1000TB from disk to disk over the course of the last 20 years a so my instinct should play a part here when something is a bit different. Wish I had another spare 3TB drive as I'd format it to EXT4 and copy the data over and compare.

 

Is the block size smaller by default on these other file systems? There are a lot of small files, source code, within the data that I've copied over. I'm starting to wonder if the default 4k block size of XFS is responsible for this unexpected increase and that I should have gone with 512 bytes instead.

 

Sounds like I need to do some more reading and testing. Thanks so far for those that have tried to help.

 

P.S. Just realised that I posted this in the v5 group rather than the v6 group. Mod feel free to move to correct part of the forum.