Writing to network share halts to a crawl when cache fill up

[ptr727]

I am busy moving several TB of data from my existing server to Unraid.

I have a 4 x 1TB SSD cache setup, plus several HDD's.

I tested performance with the share set to not using cache, and it is running at about 400Mbps, I can see disk IO is split between a storage drive and the parity drive.

If I enable cache preferred, I get about 1Gbps, expected saturation of the network, and all IO is to the SSD cache.

As the cache is filling up, I get tons of warnings, and around 95% speed becomes drops between 0 and few Kbps, and the client starts timing out.

I expect full speed until the cache is near full, then back to HDD speeds around 400Mbps.

I do not expect the system to fall over or crawl to a halt.

 

I had to turn the cache function off as it is not working as I expect.

 

Is there anything I can do to prevent the system become unresponsive when the cache gets full?

 

 

[itimpi]

What have you set for the Minimum Free Space setting for the cache drive?    Is it set to be larger than any file you expect to be copying?

[ptr727]

Looks like cache min free space is at default 2000000.

Per the help that looks like "2000000 => 2,048,000,000 bytes" = 2GB.

 

Some files may be larger, but if the cache is not large enough for the file, should it not go direct to HDD?

I understand that based on how the write happens the filesystem may not know the end size, but still, run out of space write to HDD, or get near out of space start moving files.

Or is it run out of space and fail?

 

 

Edited May 3, 2019 by ptr727

[itimpi]

11 minutes ago, ptr727 said:

Looks like cache min free space is at default 2000000.

Per the help that looks like "2000000 => 2,048,000,000 bytes" = 2GB.

 

Some files may be larger, but if the cache is not large enough for the file, should it not go direct to HDD?

I understand that based on how the write happens the filesystem may not know the end size, but still, run out of space write to HDD, or get near out of space start moving files.

Or is it run out of space and fail?

 

 

UnRAID will never move a file it is in the process of writing once it has selected the disk - it will fail that file when the space runs out.

[ptr727]

Ok, that is fine, but that means it better never run out of space.

Will it start moving files when it is near out of space?

[itimpi]

3 minutes ago, ptr727 said:

Ok, that is fine, but that means it better never run out of space.

Will it start moving files when it is near out of space?

No.     That is the purpose of the Minimum Free Space setting on the cache.    When the free space drops to that level then Unraid will stop even trying to use the cache disk for new files.

[ptr727]

Ok, so the only time a file copy should fail is if it is larger than the min free space, and the copy operation did not try to reserve space before copying?

My experience shows that performance starts to severely degrade as soon as space near runs out.

 

I stopped copying, started the mover, will take the array offline and increase min free space and try again.

It does seem like a high and low watermark for the cache with auto move is missing.