My server has started getting the above error on the console screen 6 times (3 are for the same logical number, the other 3 are 3 different ones. )

 

Everything seems to be running fine, it ran a parity check on the 1st and it didn't find any errors. I had a drive with SMART errors, I swapped that out, but it didn't make any difference.  I am attaching my diag.

 

Any ideas? 

plexnas-diagnostics-20240508-1010.zip

Corrupt docker.img. Seems unlikely that you would have filled it though. Have you done memtest? If not, you might do that before anything else just in case. You never want to run any computer unless memory is working perfectly.

 

You will have to delete, recreate, reinstall docker, but we will get to that later.

 

Your appdata and system shares have files on the array.

 

Ideally, appdata, domains, system shares would have all files on cache or other fast pool, with nothing on the array, so Dockers/VMs will perform better, and so array disks can spin down since these files are always open.

 

Nothing can move or delete open files, so you will have to disable Docker and VM Manager in Settings before you can work with these.

 

Then set these shares to Primary:cache; Secondary:array; Mover action:array->cache.

 

Then run Mover, wait for it to finish, and post new diagnostics.

Thank you for the quick reply and great instructions!

 

I ran the memtest and found that one stick of my memory was bad and I removed it leaving me with 8GB instead of 16. I then adjusted the settings for the shares and ran mover. It moved everything to cache as expected, however it left 22.57 GB on the array. I'm not sure why that is, I double and triple checked all the settings and didn't see anythign wrong, and there is 725 GB free on the cache drive. I also double checked that docker was shut down.  After that was completed I attempted to gather the diagnostics but it kept crashing my browser. It would scroll through for 20 to 30 minutes and then the browser would give me an error page. I tried in both Chrome and Firefox.  I rebooted the server and ran diagnostics again which I am attaching. 

 

There is good news. It seems to be the munin server that is corrupted and not my Plex. I don't even use munin so I can just delete it and leave it stopped.

 

Thanks again, let me know how the new diagnostics look. 

plexnas-diagnostics-20240508-2013.zip

There will always be some overhead with XFS, but if you are not sure everything was moved, enable the mover logging, run the mover, post new diags.

I disabled docker again and ran mover (logging was already on). I think I have found the problem:

May  9 07:01:56 PlexNAS move: file: /mnt/disk1/system/libvirt/libvirt.img
May  9 07:01:56 PlexNAS move: move_object: /mnt/disk1/system/libvirt/libvirt.img File exists
May  9 07:01:56 PlexNAS move: file: /mnt/disk3/system/docker/docker.img
May  9 07:01:56 PlexNAS move: move_object: /mnt/disk3/system/docker/docker.img File exists
May  9 07:01:56 PlexNAS move: mover: finished

 

The docker.img on disk 3 is dated 2023-10-17, the one on cache is dated today. I *assume* I can just delete the one on disk3, but I wanted to ask someone that knows what they are doing before I do anything silly.  The libvirt.img is the same situation, the one on disk one is dated 2023-11-17 (the day I was clearly hacking at things instead of asking first and messed something up). The one in cache has yesterday's day, which makes sense, I never turned the VMs back on since I haven't used a VM so far.

 

Is it safe to manually delete the two old copies that are still on the array?

 

 

As long as your dockers and VMs are functioning as you expect then it should be safe to delete the ones on the array.