Hi, since updating to 6.12.6, Plex and Deluge will just stop overnight.  Plex will not be accessible from the web or apps and deluge stops and throws a execution error when trying to restart.  A fresh reboot will fix both and then overnight will revert again.  The error log shows something to do with my cache disk which is a NVMe M.2 1 tb drive.  I've done a parity check and everything is fine.  A fresh reboot log shows everything looks ok but when I start deluge the errors start.  I'm not really sure what is happening with plex....Can anyone help.

 

Error log:

 

Jan 23 19:28:38 Tower avahi-daemon[12559]: Service "Tower" (/services/ssh.service) successfully established.
Jan 23 19:28:38 Tower avahi-daemon[12559]: Service "Tower" (/services/smb.service) successfully established.
Jan 23 19:28:38 Tower avahi-daemon[12559]: Service "Tower" (/services/sftp-ssh.service) successfully established.
Jan 23 19:28:39 Tower kernel: Bridge firewalling registered
Jan 23 19:28:39 Tower kernel: Initializing XFRM netlink socket
Jan 23 19:28:40 Tower emhttpd: shcmd (182): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1
Jan 23 19:28:40 Tower kernel: loop3: detected capacity change from 0 to 2097152
Jan 23 19:28:40 Tower kernel: BTRFS: device fsid 67a68631-261b-441f-96df-ef4e191eb5c2 devid 1 transid 95 /dev/loop3 scanned by mount (13503)
Jan 23 19:28:40 Tower kernel: BTRFS info (device loop3): using crc32c (crc32c-intel) checksum algorithm
Jan 23 19:28:40 Tower kernel: BTRFS info (device loop3): using free space tree
Jan 23 19:28:40 Tower kernel: BTRFS info (device loop3): enabling ssd optimizations
Jan 23 19:28:40 Tower root: Resize device id 1 (/dev/loop3) from 1.00GiB to max
Jan 23 19:28:40 Tower emhttpd: shcmd (184): /etc/rc.d/rc.libvirt start
Jan 23 19:28:40 Tower root: Starting virtlockd...
Jan 23 19:28:40 Tower root: Starting virtlogd...
Jan 23 19:28:40 Tower root: grep: /etc/libvirt/qemu/*.xml: No such file or directory
Jan 23 19:28:40 Tower root: grep: /etc/libvirt/qemu/*.xml: No such file or directory
Jan 23 19:28:40 Tower root: Starting libvirtd...
Jan 23 19:28:40 Tower kernel: tun: Universal TUN/TAP device driver, 1.6
Jan 23 19:28:40 Tower emhttpd: nothing to sync
Jan 23 19:28:40 Tower unassigned.devices: Mounting 'Auto Mount' Remote Shares...
Jan 23 19:28:40 Tower unassigned.devices: Using Gateway '10.0.0.1' for Remote Shares.
Jan 23 19:28:40 Tower unassigned.devices: Waiting 5 secs before mounting Remote Shares...
Jan 23 19:28:40 Tower dnsmasq[13789]: started, version 2.89 cachesize 150
Jan 23 19:28:40 Tower dnsmasq[13789]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile
Jan 23 19:28:40 Tower dnsmasq-dhcp[13789]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Jan 23 19:28:40 Tower dnsmasq-dhcp[13789]: DHCP, sockets bound exclusively to interface virbr0
Jan 23 19:28:40 Tower dnsmasq[13789]: reading /etc/resolv.conf
Jan 23 19:28:40 Tower dnsmasq[13789]: using nameserver 10.0.0.1#53
Jan 23 19:28:40 Tower dnsmasq[13789]: read /etc/hosts - 3 names
Jan 23 19:28:40 Tower dnsmasq[13789]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names
Jan 23 19:28:40 Tower dnsmasq-dhcp[13789]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jan 23 19:28:41 Tower kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Jan 23 19:28:41 Tower kernel: nvidia-uvm: Loaded the UVM driver, major device number 238.
Jan 23 19:28:43 Tower rc.docker: binhex-plexpass: started succesfully!
Jan 23 19:28:56 Tower emhttpd: shcmd (186): /usr/local/sbin/mover &> /dev/null &
Jan 23 19:31:00 Tower kernel: docker0: port 1(veth8469c0d) entered blocking state
Jan 23 19:31:00 Tower kernel: docker0: port 1(veth8469c0d) entered disabled state
Jan 23 19:31:00 Tower kernel: device veth8469c0d entered promiscuous mode
Jan 23 19:31:00 Tower kernel: docker0: port 1(veth8469c0d) entered blocking state
Jan 23 19:31:00 Tower kernel: docker0: port 1(veth8469c0d) entered forwarding state
Jan 23 19:31:00 Tower kernel: docker0: port 1(veth8469c0d) entered disabled state
Jan 23 19:31:00 Tower kernel: eth0: renamed from veth3b31da9
Jan 23 19:31:00 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth8469c0d: link becomes ready
Jan 23 19:31:00 Tower kernel: docker0: port 1(veth8469c0d) entered blocking state
Jan 23 19:31:00 Tower kernel: docker0: port 1(veth8469c0d) entered forwarding state
Jan 23 19:31:00 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:00 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:01 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:01 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:01 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:01 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:01 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:01 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:05 Tower kernel: nvme_log_error: 35 callbacks suppressed
Jan 23 19:31:05 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:05 Tower kernel: blk_print_req_error: 35 callbacks suppressed
Jan 23 19:31:05 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:05 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:05 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:05 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:05 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:05 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:05 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:06 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:06 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:06 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:06 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:06 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:06 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:08 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:08 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:08 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:08 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:31:08 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 23 19:31:08 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 23 19:36:11 Tower monitor: Stop running nchan processes
Jan 23 19:46:55 Tower webGUI: Successful login user root from 10.0.0.65
 

You should post your diagnostics.

tower-diagnostics-20240124-1930.zipThanks for your response, attached is the diagnostics....tower-diagnostics-20240124-1930.zip

Diags are after rebooting, grab new ones after the problem and before rebooting.

tower-diagnostics-20240125-1302.zipThank you for responding, current diagnositic log attached and the problems are currently happening.   Any help would be very appreciated thank you!  Currently, the dashboard shows plex as running but when I click on it and choose webui I get 404 not found.  If I try to start deluge a pop up comes up with an execution error.

Edited January 25, 20242 yr by Hobby31

52 minutes ago, Hobby31 said:

If I try to start deluge a pop up comes up with an execution error

 

Your drive looks like it's failing

 

Jan 25 04:27:11 Tower kernel: BTRFS info (device nvme0n1p1: state EA): forced readonly

 

 

It does look like the NVMe device has issues:

 

Jan 24 19:37:01 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 24 19:37:01 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 24 19:37:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 24 19:37:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 24 19:37:02 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 260912176, 8 blocks, I/O Error (sct 0x2 / sc 0x81) 
Jan 24 19:37:02 Tower kernel: critical medium error, dev nvme0n1, sector 260912176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

 

Cool, I'll try replacing the cache drive and see what happens.  Thanks for the replies.  I'll post back in a few days after I replace that drive....hopefully that fixes plex as well...

Ok, I had to replace the cache drive.  I couldn't get 89gb of stuff off of it even though I changed all my shares to the array.  The mover would not do anything.  In the end, I had to reinstall deluge and plex and rebuild the plex data.  So far it's been 24 hours and no issues.  Thank you all for your responses ....

9 minutes ago, Hobby31 said:

mover would not do anything

You should have asked for help with that.

17 minutes ago, Hobby31 said:

mover would not do anything

Nothing can be moved from read-only, whether by mover or some other means. Move has to copy from source to destination, then delete from source, but source is readonly, so can't be modified. You would have to copy the data instead of attempting to move. 

Thank's for replying again,  I tried using Krusader and and it would hang when trying to read from it.  I also tried the old NVME cache drive in an external enclosure  and tried to copy from it to the new drive and same result, krusader would stall then say it couldn't read source...

Edited January 28, 20242 yr by Hobby31