(SOLVED) Constant cache disk full errors, mover stuck

[Fulgurator]

Hello,

 

Yesterday I had a couple of notifications from my Unraid server that warned that the disk space on the cache was low. I was not around when this happened so the cache continued to fill up completely. The cache is a 500GB SSD drive, with 10GB as minfreespace. I had recently started downloading a very large torrent, presumably one of the files inside was bigger than 10GB and was created on the cache...

 

As an attempt to fix this I have set most of my shares to "Yes: Cache" (some of them were set to "Prefer: Cache" previously). Now it seems I am stuck, after 24hrs the cache is still full and the log is chock full of "cache disk full" errors.

 

How can I get out of this jam? If I change my shared folders to not use the cache, will the mover clear them out, or is that up to me to sort out?

 

I cannot stop the array since the mover is running, but presumably the mover will never finish now due to the cache disk being full...

sauron-diagnostics-20210205-0742.zip

Edited February 6, 2021 by Fulgurator
Solved

[JorgeB]

Don't see any attempts to run the mover in the log, try again, also make sure mover logging is enable.

[Fulgurator]

I can assure you, the mover is running

 

I just enabled Mover Logging but I guess it is too late, I would have had to enable it before it started. I thought the mover bypassed the cache but perhaps I am misunderstanding how this works.

 

I don't see an obvious way of stopping the mover, I don't suppose rebooting the server (through the web interface) is a good idea.

 

sauron-diagnostics-20210205-0954.zip

[ChatNoir]

I see that some on your shares are Cache YES and their content should be moved.

 

However, mover cannot move open files.

Are you sure that the files you want to move are not used by a docker container or a VM ? (torrent share for example)

 

If they are in use, you would have to go to Settings and disable Docker and VM.

Then run mover.

When it is done, go back to Settings to reactivate Docker and VM.

[Fulgurator]

Thanks @ChatNoir, I should have thought about that, it makes a lot of sense. I noticed that the mover script implicitly paused my VM while it was running but my Docker containers (including the one running Deluge) were still up.

 

It is too soon to know if the mover is working, I don't see any writes to the array so far. It has not stopped running since it started many hours ago. Hopefully this will clear itself up after a few hours.

 

For next time, is there a way to automatically pause or stop a specific container when mover is started?

[ChatNoir]

Not that I know of, but I did not really look for that as I do not download from my server.

[Fulgurator]

Ok, turning off Docker made my server completely idle now, all disks have spun down. It still says the mover is running but this is clearly a lie. 😮

 

What can I do now? Reboot? (through the UI, not a hard reset)

[JorgeB]

2 hours ago, Fulgurator said:

I can assure you, the mover is running

Don't doubt that, but there's nothing in the log about it, not even the mover starting, so can't see the problem, you can try rebooting and grab new diags.

[Fulgurator]

I found a way to stop the mover from the commandline ("mover stop" via ssh) without rebooting Unraid. I then restarted the mover manually via Settings | Scheduler and I finally see the files being moved, the cache is slowly being cleared out as it was supposed to in the first place.

 

It is interesting that the mover got stuck like that, I will keep mover logging enabled so I can have a better idea of how it happened next time. I suppose there was just not enough space in the cache for the download to fit.

[itimpi]

2 hours ago, Fulgurator said:

at, I will keep mover logging enabled

Just be careful you do not end up getting problems due to your log filling up with mover messages.

[Fulgurator]

It is unfortunate that I would have had to have the mover log enabled in the first place to know what happened. I can keep it disabled but then next time I will be blind once more.

 

Maybe the real fix (for me anyway) would be to put the cache minfreespace absurdly high, like 250GB. So on a 500GB cache that leaves me with a 250GB buffer for overruns like that. I don't often download a single ~280GB torrent but I see now how this was a really bad idea, since the intermediate download folder was "cache: yes" it meant everything would be open and stay in the cache until I stopped the torrent.

 

Another easy workaround, just add more cache... 😁