Hi!

 

I have been dealing with a cache disk that is misbehaving. I have already taken the following diagnostic steps:

- Extended SMART report: passed

- Memtest86+: passed

- Investigated log files myself with a lot of googeling: no definitive result

 

I guess I have a similar issue to this post:

 

I know I've had issues with the cache filling up completely, due to the Arr stack trying to download faster than the mover could keep up with. However, in those scenarios the docker containers would simply freeze, I'd kick the mover and it would all come back to life after a few minutes. In this scenario, the cache disk is not full at all.

 

Somehow after running memtest the cache was writable, and I was able to recreate the docker image slightly larger than before (30GB instead of the old 25GB, it was time again). However, right after this operation the cache locked itself again to read only. I do have a backup of the old docker image on the array.

 

I have attached diagnostics of both today, right after doing a reboot (so everything is fresh, apologies for the cleared syslog), and from a while ago when I was last investigating the issue. If any additional diagnostics are needed, please let me know.

 

I would also love to know how I could've found the exact issue myself, would be great if some pointers could be given to what I missed in my searches 🙂

 

Thanks for ya'lls time!

tower-diagnostics-20240116-1909.zip tower-diagnostics-20240103-1224.zip

Edited January 16, 20242 yr by MagicLegend

The filesystem is corrupt, with btrfs I usually recommend backing up and re-formatting the pool, since it still mounts you can use you favorite tool to copy everything you can elsewhere, then restore after formatting.

Hi Jorge,

 

Thanks for taking a look. What would make it that the entire array would be corrupt, and not just the cache disk here? Formatting just the cache disk would be a lot simpler than having to find a temporary location for those terrabytes

 

Thanks again!

23 minutes ago, MagicLegend said:

I know I've had issues with the cache filling up completely, due to the Arr stack trying to download faster than the mover could keep up with.

You should set the Minimum Free Space on the cache to be larger than any file you expect to download so that when it drops below this new files overflow to the main array rather than filling the cache pool to the point of causing problems.

1 minute ago, itimpi said:

You should set the Minimum Free Space on the cache to be larger than any file you expect to download so that when it drops below this new files overflow to the main array rather than filling the cache pool to the point of causing problems.

 

Ah yeah, you are totally right. Didn't know that was an option in Unraid Thanks for the pointer, will definitely configure that once I get it back up and running!

14 minutes ago, MagicLegend said:

What would make it that the entire array would be corrupt, and not just the cache disk here?

I only mentioned the pool, not the array.

30 minutes ago, JorgeB said:

I only mentioned the pool, not the array.

Ah yeah, of course. Forgot it was renamed.

 

I've gotten it all back up and running again. Seems like the radarr database was the only thing that ended up corrupted in my appdata backup during all of this. Could be worse!

 

For others; I followed the docs on reformatting the cache drive: https://docs.unraid.net/unraid-os/manual/storage-management/#reformatting-a-cache-drive

 

For me the mover was not moving anything that was left on the cache, so I did it manually over SSH. I had a backup of the appdata already using CABackup. Keep in mind that by default it only saves 7 backups, and if you're short on time to fix this (like me) then keep in mind that it might overwrite your previous backups from when everything was working.

 

Thank you both!

 

You should post new diagnostics so we can see what you have now. Looks like you had your Docker and VM shares set to get moved to the array, which is not the ideal setup.

2 minutes ago, trurl said:

You should post new diagnostics so we can see what you have now. Looks like you had your Docker and VM shares set to get moved to the array, which is not the ideal setup.

 

Hmm, I don't think so? I don't use VM's to be clear (I only have a quad code in there thanks to Intel, so I'm bound to local VM's for now), so only Docker is of concern for me. New diagnostics are attached!

 

Thanks!

tower-diagnostics-20240119-2239.zip

6 minutes ago, MagicLegend said:

I don't use VM's to be clear

You're earlier Diagnostics showed you had VM Manager enabled. I guess I was looking at some other diagnostics about your shares.

 

But, appdata does have some files on disk2.

 

 

7 minutes ago, trurl said:

You're earlier Diagnostics showed you had VM Manager enabled. I guess I was looking at some other diagnostics about your shares.

 

But, appdata does have some files on disk2.

 

 

 

Ah yeah, the manager is enabled but no VM's are actively configured (only one in cold storage that the VM manager doesn't know about).

 

Hmm, you are correct (obviously). I see some Jellyfin data is on disk2, but it's the only appdata that is on disk2. Weird, since the appdata share is configured to move from the array to pool. Could this simply be because I have the array configured as secondary storage? I'm assuming these settings are in the diagnostics somewhere as well, but if not:

 

The jellyfin data:

 

<user>@<machine>:~# ls -lhat /mnt/disk2/appdata
total 0
drwxrwxrwx 14 nobody users 268 Jan 19 21:12 ../
drwxr-xr-x  3 nobody users  30 Aug 31 21:34 jellyfin/
drwxrwxrwx  3 nobody users  30 Aug 31 20:29 ./
<user>@<machine>:~# ls -lhat /mnt/disk2/appdata/jellyfin/jellyfin/
total 0
drwxr-xr-x 3 nobody users  30 Aug 31 21:34 ../
drwxr-xr-x 6 nobody users  74 Aug 31 20:32 ./
drwxr-xr-x 7 nobody users 103 Aug 31 20:29 data/
drwxr-xr-x 4 nobody users  44 Aug 31 20:29 dlna/
drwxr-xr-x 2 nobody users  10 Aug 31 20:29 log/
drwxr-xr-x 2 nobody users  10 Aug 31 20:29 cache/

 

Edited January 19, 20242 yr by MagicLegend
Added jellyfin directory listing

Probably that container had to write some appdata and cache wasn't available for writing.

 

Mover ignores shares that have nothing set as Secondary.

 

Nothing can move open files, so you have to stop that docker before its appdata can be moved. I usually just say Disable Docker and VM Manager in Settings.

 

Mover won't replace files so if a file exists in both places, you will have to decide which to keep.

On 1/20/2024 at 7:03 AM, MagicLegend said:

Ah yeah, of course. Forgot it was renamed.

 

I've gotten it all back up and running again. Seems like the radarr database was the only thing that ended up corrupted in my appdata backup during all of this. Could be worse!

 

For others; I followed the docs on reformatting the cache drive: https://docs.unraid.net/unraid-os/manual/storage-management/#reformatting-a-cache-drive

 

For me the mover was not moving anything that was left on the cache, so I did it manually over SSH. I had a backup of the appdata already using CABackup. Keep in mind that by default it only saves 7 backups, and if you're short on time to fix this (like me) then keep in mind that it might overwrite your previous backups from when everything was working.

 

Thank you both!

 

I think I have the same problem now. How did you do the mover manually over SSH?

Edited January 29Jan 29 by bobalot

3 hours ago, bobalot said:

I think I have the same problem now. How did you do the mover manually over SSH?

Hi! I believe that I manually just used mv to move files around, I didn't actually invoke the mover (since it wasn't moving data).