Hello there, mover is running and I started getting messages about the cache being full. I checked few topics and checked the cache floor and shares minimum requirements. They all seem ok. The disk is also mostly empty so not sure why the message. I noticed this since the logs memory was almost full.
Any idea of why this may be happening?
Thanks in advance!
I've attached the diagnostics.
diagnostics-20220211-1047.zip
~~~~~~~~~~~~~~~~~~~~~~
Edit: After experimenting around, this may actually be a bug. Is important to note Docker was disabled during the whole process to avoid files moving around/being used in appdata.
TL;DR:
When having 2 pools, one cache's floor setting causes mover get stuck while moving the other pool.
Steps to repro:
Create two cache pools. Let's call them Pool A and Pool B.
Create a two sets shares, one to be used in Pool A the other for Pool B. In my case:
Pool A:
Share A. Floor set to 5GB, Set to No
Share A2. Floor set to 5GB, Set to No
Share A3. Floor set to 0, Set to No
Share A4. Floor set to 5GB, Set to No
Pool B:
Share B. Floor set to 100GB. Set to Prefer.
Make it so that one pool reaches the floor limit. Chose Share B for Pool B here.
In my case I set Share B to Prefer and apply changes.
Transfered files until pool B reached the limit (100GB of free space).
Put some items in [Share] A, A2, A3 and A4. I had a total of 16.4GB.
Set [Share] A, A2, A3 and A4 to Prefer, Apply changes.
Kick off the mover.
Monitor system logs, it may move some items, but eventually it will print that Pool A is full, when in fact is not. Mover will be in a loop and will never stop.
I don't think floor size or the number of shares used per pool matters too much.
The setup:
Server is running version 6.9.2.
2 Pools used as cache:
Pool 1:
Composed of one NVMe SSD
RAID1
Used for system, domains, appdata and isos. Set to Prefer and floor of 5GB, except for isos which was set to 0.
Pool 2:
Composed of two NVMe
RAID1
Used as file transfer cache for a specific share. Set to Prefer and floor set to 100GB
I did notice the setting for Pool 2 was set to Prefer, which as not intended. I intended for Yes to in place.
Prior to running mover where the problem originated:
I had a replaced the drive in Pool 1 for another, same size however. for this I followed these steps:
Set all Pool 1 shares to Yes.
Ran mover.
Checked Pool 1 to be empty.
Set Pool 1 shares to No.
Powered off machine.
Replaced NVMe drive.
Formatted and reconfigured pool, shares and options as before.
Changed [New] Pool 1 to Prefer.
Ran the move. <-- This is where the problem started. Didn't noticed until next morning since I left it running overnight. But was transfering fine before I went to sleep.
Problem:
Mover was running and got stuck printing cache from Pool 1 was full when in fact it was almost empty.
Status of the system at the time:
Pool 1 was almost empty.
Pool 2 was at 99GB, so full based on floor settings.
Docker was disabled throughout the whole process, prior, during and after moving.
Changes I made:
Stopped mover with mover stop command.
Set all Pool 1 shares to Yes.
Ran mover. Wait for it to complete.
Double check all cache was moved to disk and nothing was left in Pool 1. Set all Pool 1 shares to No.
Set Pool 2 share, just have one, to Yes.
Ran mover again, waited for completion.
Changed all Pool 1 shares to Prefer.
Ran mover again and waited for completion.
The whole thing just worked as expected after. But somehow the previous settings had made the mover get stuck. Almost seemed like it used Pool 2 check but blocked Pool 1 items from being moved.
Let me know if you need more info. Hopefully this is not too confusing to understand.