HELP PLEASE! server keeps crashing, docker stops

[Greygoose]

This is ongoing for the past week, once i reboot it works for a short period then locks up or docker crashes,

 

Apr 23 05:02:48 mediaserver kernel: CIFS: VFS: \\192.168.30.201\media Close cancelled mid failed rc:-9
Apr 23 05:02:51 mediaserver kernel: I/O error, dev loop2, sector 11430744 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Apr 23 05:02:51 mediaserver kernel: BTRFS error (device loop2: state EA): bdev /dev/loop2 errs: wr 1, rd 34070, flush 0, corrupt 6, gen 0
Apr 23 05:03:01 mediaserver kernel: I/O error, dev loop2, sector 11430744 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Apr 23 05:03:01 mediaserver kernel: BTRFS error (device loop2: state EA): bdev /dev/loop2 errs: wr 1, rd 34071, flush 0, corrupt 6, gen 0
Apr 23 05:03:11 mediaserver kernel: I/O error, dev loop2, sector 11430744 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Apr 23 05:03:11 mediaserver kernel: BTRFS error (device loop2: state EA): bdev /dev/loop2 errs: wr 1, rd 34072, flush 0, corrupt 6, gen 0

 

mediaserver-diagnostics-20230422-0821.zip

Edited April 23, 2023 by Greygoose

[Greygoose]

This looks like it being my cache drive.

 

Is this a drive issue or could it be cable.

[JorgeB]

Apr 21 19:55:49 mediaserver kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Apr 21 19:55:49 mediaserver kernel: ? _raw_spin_unlock+0x14/0x29
Apr 21 19:55:49 mediaserver kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

Apr 22 01:31:41 mediaserver kernel: XFS (sdc1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x6301a433 dinode
Apr 22 01:31:41 mediaserver kernel: XFS (sdc1): Unmount and run xfs_repair

Check filesystem on cache

[Greygoose]

Thanks jorge

 

Where did you see these warnings in the logs?  just for future refrence.

[JorgeB]

In the syslog.

[Greygoose]

Im back again Jorge

 

I tried to stop the array and now its constantly retrying to stop the array.

 

I have no idea whats causing it.    I checked the syslog but unable to find a solution.

 

 

mediaserver-syslog-20230424-1638.zip

[JorgeB]

Apr 24 17:36:50 mediaserver root: rmdir: failed to remove '/mnt/user': Device or resource busy

Something is preventing /mnt/user from being unmounted, an open SSH session there for example will cause this, if you don't know what it is force a reboot by typing 'reboot' on the console.

[Greygoose]

Hi Jorge,

 

I had to restart the server.

 

I put it in maintance mode as i wanted to move everyhting off the cache drive to the array and now im getting errors and it wont move the appdata folder and whatever else is on that drive.

 

Another issue

 

Apr 25 17:00:57 mediaserver kernel: CIFS: VFS: \\192.168.30.201\media Close cancelled mid failed rc:-9
Apr 25 17:03:34 mediaserver kernel: CIFS: VFS: \\192.168.30.201\media Close unmatched open for MID:61222186
Apr 25 17:04:36 mediaserver kernel: CIFS: VFS: \\192.168.30.201\media Close unmatched open for MID:61271094
Apr 25 17:05:07 mediaserver kernel: CIFS: VFS: \\192.168.30.201\media Close unmatched open for MID:61296616
Apr 25 17:05:48 mediaserver emhttpd: shcmd (8373): /usr/local/sbin/mover &> /dev/null &

 

mediaserver-syslog-20230425-1607.zip

[JorgeB]

56 minutes ago, Greygoose said:

Apr 25 17:00:57 mediaserver kernel: CIFS: VFS: \\192.168.30.201\media Close cancelled mid failed rc:-9
Apr 25 17:03:34 mediaserver kernel: CIFS: VFS: \\192.168.30.201\media Close unmatched open for MID:61222186

This is from an SMB mounted share with UD, you might post about it in the UD support thread.

[Greygoose]

Hi Jorge,

 

Thanks again.

 

I have narrowed down my issue...   hopefully the last one.

 

I have files from my dockers showing on cache and array where the move process stopped, either from me restsrting from a paused system or the docker still running during the move of the appdata from cache to array.

 

How do i sort this, how do i get the rest of the data off the cache to array, move doesnt do anything.

 

 

Edited April 26, 2023 by Greygoose

[JorgeB]

Enable mover logging and check the syslog to see why the files are not being moved, they won't be moved if for example they already exist on destination, or if they are in use.

[trurl]

Another thing you need to take a look at is the fact that you have no Minimum Free set for your shares or pools, and all of your disks are very full.

[Greygoose]

Hi everyone,

 

I've checked the files not copied to array.

 

I see the issue being the new files created and now mover is unable to overwrite the files.

 

I moved everything back to the cache. So mostly everything is back on the cache drive. But some of the dockers have lost the configs, such as Plex not showing my server to claim. Immich not allowing login of existing accounts and wanting a new account created.

 

Is there a way to resolve this issue.

[trurl]

Post new diagnostics.

 

Do you have appdata backup?

[Greygoose]

I just checked my reliable backup methods...  Or at least i thought so.

 

I dont have a recent appdata backup, only 6mths prior to this issue.

 

I am moving back the data from the cache to the array as I have a replacement drive to install.  I will post the diagnostics once this has completed, as hopefully then we can see what files wont copy onto the array.