File copy error after Cache Drive gets full

[pbcal]

Hi,

I am new to unRAID, using it for about a week now. I have a server with 4 8TB drive (3 data, 1 parity) and 2 500GB SSD cache pool.

 

I wanted to transfer few terabytes of data from my external hard drive to the unRAID server. I created a share named "Share" to use all disks and Use Cache = "Yes". Then I mounted the external hard drive using Unassigned Devices. Opened a terminal window from the web gui and entered following commands: (I wanted to copy the file with user permission, not root, so I can access them later through SMB share.)

 

# sudo -s -u username

$ cd /mnt/user/Share/Old_Files

$ cp -R /mnt/disks/ExtHD/* .

 

Once the copy starts I can see my cache drive in getting filled (in web gui) and once the cache drive is full, the data drive used space starts to increase. But during the transition from cache to data drive the last file that goes to cache gets an error (No space left in the drive) and the file does not get copied completely.

 

So what I think is happening here is unRAID does not check the space left in cache drive before starting to copy a file, once that file gets the disk full error the next file goes to the data drive.

 

Is this a bug or I am doing something some thing wrong? Any advice will be greatly appreciated.

 

[trurl]

In the more general case of writing files, how is it supposed to know how large a file will become when it begins to write it?

 

You need to go to Global Share Settings and set Minimum Free for cache to be larger than the largest file you expect to write. If there is less than Minimum Free available on a disk when it starts to write a file, it will choose another disk.

 

Each User share also has a Minimum Free setting which works the same way.

[pbcal]

Thanks for your quick response. I will give it a go tonight.

[pwm]

4 hours ago, trurl said:

In the more general case of writing files, how is it supposed to know how large a file will become when it begins to write it?

 

There are function calls that can be used to pre-allocate disk space when copying data (fallocate() or posix_fallocate()).

 

Great to reserve the required space up-front so a file copy doesn't fail because some other disk writes consumes part of the free space making the file copy fail.

Also great to reduce disk fragmentation since two concurrent file transfers will not interleave the two data streams on the disk.

 

It's just that most programs do not make use of pre-allocation in which case neither the copying program nor the file system layer will know if the file will fit until the disk suddenly runs out of space or the program finally closes the file handle.

 

The disadvantage with pre-allocation is that the user will see a file of full size even if only a small fraction of the file has actually been copied.