thatja

root@Plexified:~# cat /etc/mtab rootfs / rootfs rw,size=49452720k,nr_inodes=12363180,inode64 0 0 proc /proc proc rw,relatime 0 0 sysfs /sys sysfs rw,relatime 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=32768k,mode=755,inode64 0 0 /dev/sda1 /boot vfat rw,noatime,nodiratime,fmask=0177,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,flush,errors=remount-ro 0 0 /dev/loop0 /lib squashfs ro,relatime,errors=continue 0 0 overlay /lib overlay rw,relatime,lowerdir=/lib,upperdir=/var/local/overlay/lib,workdir=/var/local/overlay-work/lib 0 0 /dev/loop1 /usr squashfs ro,relatime,errors=continue 0 0 overlay /usr overlay rw,relatime,lowerdir=/usr,upperdir=/var/local/overlay/usr,workdir=/var/local/overlay-work/usr 0 0 devtmpfs /dev devtmpfs rw,relatime,size=8192k,nr_inodes=12363180,mode=755,inode64 0 0 devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /dev/shm tmpfs rw,relatime,inode64 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 hugetlbfs /hugetlbfs hugetlbfs rw,relatime,pagesize=2M 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 tmpfs /var/log tmpfs rw,relatime,size=131072k,mode=755,inode64 0 0 rootfs /mnt rootfs rw,size=49452720k,nr_inodes=12363180,inode64 0 0 tmpfs /mnt/disks tmpfs rw,relatime,size=1024k,inode64 0 0 tmpfs /mnt/remotes tmpfs rw,relatime,size=1024k,inode64 0 0 tmpfs /mnt/addons tmpfs rw,relatime,size=1024k,inode64 0 0 tmpfs /mnt/rootshare tmpfs rw,relatime,size=1024k,inode64 0 0 /dev/md1p1 /mnt/disk1 xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/md2p1 /mnt/disk2 xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/md3p1 /mnt/disk3 xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/md4p1 /mnt/disk4 xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/md5p1 /mnt/disk5 xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/md6p1 /mnt/disk6 xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/md7p1 /mnt/disk7 xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/nvme0n1p1 /mnt/nvme xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/nvme1n1p1 /mnt/nvmedl btrfs rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/ 0 0 shfs /mnt/user0 fuse.shfs rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other 0 0 shfs /mnt/user fuse.shfs rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other 0 0 /dev/loop2 /var/lib/docker btrfs rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/ 0 0 /dev/loop2 /var/lib/docker/btrfs btrfs rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/ 0 0 nsfs /run/docker/netns/fdcd16e64b60 nsfs rw 0 0 nsfs /run/docker/netns/default nsfs rw 0 0 /dev/loop3 /etc/libvirt btrfs rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/ 0 0 googledecrypted: /mnt/nvmedl/plexified/mounts/google fuse.rclone rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 mergerFS /mnt/nvmedl/plexified/mounts/moviesrc/Movies fuse.mergerfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0 mergerFS /mnt/nvmedl/plexified/mounts/secret fuse.mergerfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0 nsfs /run/docker/netns/93f74b6ab06f nsfs rw 0 0 nsfs /run/docker/netns/5b147fc09a9a nsfs rw 0 0 nsfs /run/docker/netns/21aa0e5b657b nsfs rw 0 0 nsfs /run/docker/netns/efbd08065b39 nsfs rw 0 0 nsfs /run/docker/netns/d628991141dd nsfs rw 0 0 nsfs /run/docker/netns/021970ea8a50 nsfs rw 0 0 nsfs /run/docker/netns/cf6e4881fffc nsfs rw 0 0 nsfs /run/docker/netns/ef2fa253537f nsfs rw 0 0 nsfs /run/docker/netns/3a977d309e1d nsfs rw 0 0 nsfs /run/docker/netns/2382e5f02b25 nsfs rw 0 0 nsfs /run/docker/netns/a3dd8518c453 nsfs rw 0 0 nsfs /run/docker/netns/29bd7cce0d5e nsfs rw 0 0 tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=9893912k,nr_inodes=2473478,mode=700,inode64 0 0 nsfs /run/docker/netns/b5ada1918a90 nsfs rw 0 0 nsfs /run/docker/netns/3ca37854a52b nsfs rw 0 0 nsfs /run/docker/netns/db6327bbe313 nsfs rw 0 0 That's what I get when I run cat /etc/mtab The other two just hang without an output.

Okay, so I've just tried running the first one df -h And my ssh window is hanging atm. Has been for around 3minutes now

It has just happened again after almost 12 days of uptime.

Okay, I will do that. As for mergerfs, when I boot up my server, I have a bash script that I created that mounts my rclone, mergerfs and autoscan. I run this file around a minute after I start my array. Here's the script #!/bin/bash # Start a screen session named "files" screen -dmS files # Attach to the "files" screen session and execute the first command screen -S google -X stuff $'rclone mount --config=/mnt/nvme/plexified/mounts/rclone/rclone.conf --allow-other --no-traverse --vfs-cache-mode full --cache-dir /mnt/nvmedl/plexified/mounts/googlecache/ --vfs-cache-max-size 250G --dir-cache-time 96h --vfs-fast-fingerprint --vfs-refresh --drive-impersonate [email protected] googledecrypted: /mnt/nvmedl/plexified/mounts/google/\n' # Wait for the command to start sleep 2 # Execute the mergerfs commands mergerfs -o defaults,allow_other,use_ino,fsname=mergerFS /mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0000/:/mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0001:/mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0002:/mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0003:/mnt/nvmedl/plexified/mounts/google/MoviesSrc/0004/ /mnt/nvmedl/plexified/mounts/moviesrc/Movies/ sleep 2 # Wait for 2 seconds before running the next mergerfs command mergerfs -o defaults,allow_other,use_ino,category.create=ff,fsname=mergerFS /mnt/user/plexdata/:/mnt/nvmedl/plexified/mounts/moviesrc=NC:/mnt/nvmedl/plexified/mounts/google/Data=NC /mnt/nvmedl/plexified/mounts/secret/ # Wait for 30 seconds before starting autoscan sleep 30 # Start a screen session named "autoscan", change to the correct directory, and then run the autoscan command screen -dmS autoscan screen -S autoscan -X stuff $'cd /mnt/nvme/plexified/services/autoscan\n' screen -S autoscan -X stuff $'./autoscan_v1.4.0_linux_amd64\n' Then I start my docker containers.

How would I find out about the rootfs-ramdisk being full? or likewise if a plugin is writing to it? I haver 96GB of RAM in the server, I restarted the system via reboot on SSH using my phone on an app called Termius, only the web UI ssh isn't responsive.

The unclean shutdown was because power was pulled from the server, this wasn't a crash related to UNRAID but a power outage on my end, sorry for the confusion regarding that. The crashes today caused by UNRAID/Something else occurred at 10:30AMish and 10:50AMish. Those are what the syslogs above cover before/after events of. Also nothing at all has changed between when things were working good, and the first ever crash relating to this, all I've done is update plugins/docker containers where they have updates available, I've had 6months without issue until the first crash happened at the time of this thread creation.

This is where I am stuck, the clean shutdown was because I could not get into ssh OR the UI, that was the first crash at 1AM. Secondly, the last 2 syslog provided above was before I restarted the server. (after /mnt became inaccessible) - this was from /boot/logs as I did enable syslog server. I have had the server with nothing running at all, no docker containers but Plex, no mergerfs and it was fine. As soon as I mounted my mounts, I got another crash. Could you please elaborate on this?

I have been using the system all morning without mergerfs and it was working fine, 10mins after I mounted mergefs I got a crash. I am highly suspecting either failing USB or mergerfs to be the cause. Which is worrying as my setup depends on mergerfs, and I'm not sure what's changed because for 6months it has been solid.

plexified-diagnostics-20240517-1052.zip /mnt inaccessible again

Crashed 10mins after a reboot this time. syslog

Managed to get this off the flash drive in /boot/logs syslog

Okay so server restarted again. I can't access UI but can SSH, and when running "diagnostics" it just sticks on; root@Plexified:/mnt# diagnostics Starting diagnostics collection... root@Plexified:/mnt# diagnostics Starting diagnostics collection... Could this be a case of a failing flash drive? or bad files on flash drive? /mnt is once again inaccessible.

I was unable to get into unraid at all, even to get diagnostics. I can't really start my plex server without mergerfs so I'm not sure what to do.

And it's down again, I've just woken up and my server is down in the same state as before. Not sure what to try next.

Funny you should mention GPU driver, it did update a couple hours before this first outage occurred. I will try going back one driver!

Hi. I didn't reboot the servewr at 2.45AM. As for my mergerfs, its a pretty simple command that has been working since I started using UNRAID in December. mergerfs -o defaults,allow_other,use_ino,fsname=mergerFS /mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0000/:/mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0001:/mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0002:/mnt/nvmedl/plexified/mounts/google/Data/MoviesSrc/0003:/mnt/nvmedl/plexified/mounts/google/MoviesSrc/0004/ /mnt/nvmedl/plexified/mounts/moviesrc/Movies/ AND mergerfs -o defaults,allow_other,use_ino,category.create=ff,fsname=mergerFS /mnt/user/plexdata/:/mnt/nvmedl/plexified/mounts/moviesrc=NC:/mnt/nvmedl/plexified/mounts/google/Data=NC /mnt/nvmedl/plexified/mounts/secret/ Worth noting, that /mnt/user/plexdata is my array, the rest are all rclone mounts merged to make /mnt/nvmedl/plexified/mounts/secret/ nvmedl is the name of my cache drive and it is an nvme as the name suggests. Previously, over the past 5 days it had been happening after around 18hours, however yesterday it managed 1 day and 7 hours uptime, before the same happened around 5 hours ago. Again, the only fix was to hard reboot the server. I did try unmounting mergerfs folders but that didn't help.

Hello, my system seems to be crashing and I think mergerfs is the cause, this has only been happening recently, is there a way to downgrade to the previous version so I can test if it is indeed the new version of mergerfs? If so, how can I achieve this?

Worth noting, I got the system back up at 9:50AM this morning, its now 15:56PM and I haven't had a crash, this is with mergerfs too. Not sure if that rules mergerfs out, or not.

Well I'm not sure what that means per se, but I've used mergerfs since I first started using UNRAID in around December 2023, and mergefs is crucial to my setup, as I am merging my google drive and storing new files on my unraid array, so without mergerfs my system doesn't really work. Mergerfs has been updated quite a lot over the past couple of months, it could be a bad update I guess, but I'm not sure if there is a way to downgrade mergerfs?

The server had issues between 2:50AM-5:20AM and that's the logs that I grabbed when I woken. I've since rebooted the machine and all is working again. However, this has been the pattern for the past 5 days, it'll stay up for 16-18hrs before the same problem occurs, its done it every day for the past 4 days. The only real difference has been updating the nvidia GPU driver, its been stable for 39 days previous to upgrading the GPU driver. But could a GPU driver really cause the issue that's occuring?

Here you go. syslog.TXT

syslog

/mnt is inaccessible via FTP or SSH.

When running diagnostics download it gets stuck on sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/plexified-diagnostics-20240513-0907/shares/a-----a.cfg' 2>/dev/null

Between 4-18hrs after server start, my entire folder structure becomes unresponsive and I cannot start or stop Docker services and get the following error My Plex server becomes unresponsive, but all other apps like Sonarr UI loads, but can't import/delete/access any files as the storage system is locked up. The only thing thats fixes this is a hard reboot of the server, which is not practical to do multiple times per day. It can go anywhere from 4-18hours without doing it, before being stuck in this state again. I have attached diagnostics to the post, but not sure if its a full one as I'm struggling to gain access due to the lockups. Can anyone advise on what to do to troubleshoot this? plexified-diagnostics-20240513-0855.zip