(Solved) Cache disk full, cannot move files out of the cache disk, io errors

[Celsiuss]

Reading from the syslog the cache disk got full this morning without me being aware of it.

I notices that my docker containers were acting up, and it appears that it was because the cache disk was full.

I stopped all my VMs and docker containers and invoked the mover, however the mover has failed to move any files.

Trying to move my files manually, I get errors like this:

mv: cannot stat 'path/to/file': Input/output error

 

I can't read anything on the appdata share, I am unable to move or delete any files on the cache.

Attached is the full system diagnostics.

whale-diagnostics-20210220-1825.zip

Edited February 20, 2021 by Celsiuss

[trurl]

You have several shares set as cache-prefer. Prefer means prefer to keep files on cache permanently and if any wind up on the array try to move them back to cache when there is space.

 

Typically you only want appdata, domains, system shares as cache-prefer. What is the purpose of these other shares that you have configured to keep on cache?

 

You are having errors on cache and it isn't producing SMART report. Maybe it is also corrupt due to filling it. Check connections and post new diagnostics.

[Celsiuss]

8 minutes ago, trurl said:

You have several shares set as cache-prefer. Prefer means prefer to keep files on cache permanently and if any wind up on the array try to move them back to cache when there is space.

 

Typically you only want appdata, domains, system shares as cache-prefer. What is the purpose of these other shares that you have configured to keep on cache?

 

You are having errors on cache and it isn't producing SMART report. Maybe it is also corrupt due to filling it. Check connections and post new diagnostics.

Yes I have two other shares with cache-prefer. Those are just small shares with little data and is not used by any VM or docker container.

 

I am unable to produce SMART results of the cache, tried through the web interface.

Edited February 20, 2021 by Celsiuss

[trurl]

Your cache is probably too small to use for caching any user share writes and you should only use it for appdata, domains, system shares. Also, why have you given 40G to docker.img? Have you had problems filling it?

 

 

[Celsiuss]

1 minute ago, trurl said:

Your cache is probably too small to use for caching any user share writes and you should only use it for appdata, domains, system shares. Also, why have you given 40G to docker.img? Have you had problems filling it?

 

 

I know that my cache is too small to be used for a cache for array writes. My media share was set to yes mistakenly and unknowingly, which is most likely what caused it to fill up. My concern right now is to free up space on the cache, but it seems impossible for me to move or remove files from the cache.

As for the docker.img, I can't give you a clear answer on that as I didn't know that myself until now. I probably did that about a year ago.

[Celsiuss]

I did a reboot, and the cache SSD showed up under unassignewd devices, so I assigned it to the cache pool again and everything worked. I invoked mover and it ran without problems.