Entire folder directories become unresponsive

[thatja]

Just now, Rysz said:

 

Ok, that rules out the theory of a full rootfs ramdisk. Did you test your RAM sticks with memtest in the meantime?

 

Yes and there was no error for almost 24 hours

[thatja]

Diagnostics is stuck on this stage

 

[thatja]

I don't think this is complete, but its all I could get

plexified-diagnostics-20240531-2110.zip

[Rysz]

Can you do: cat /var/log/syslog

And see what the last log lines are?

[thatja]

 

Server crashed around 21:00-21:15ish

 

Edited May 31 by thatja

[Rysz]

1 minute ago, thatja said:

 

Server crashed around 21:00ish

 

 

Nothing stands out, I honestly have no idea. 😞 

[thatja]

I am 99.9% sure its to do with mergerfs though, it only happened when scanning in a very large Music library which is mounted with mergerfs

[thatja]

But then if it was mergerfs, i would expect unmounting them would bring the server back to "life" but it doesn't. I have to do a full reboot.

 

[Rysz]

7 minutes ago, thatja said:

I am 99.9% sure its to do with mergerfs though, it only happened when scanning in a very large Music library which is mounted with mergerfs

 

Well you can try: killall -9 mergerfs

And see if that unfreezes your server.

 

I really don't see how it can be mergerFS though, when there's zero indication in the logs (apart from garbage collection, which actually says that mergerFS is still operating normally at that point) and tons of users are using it without any problems... unless something in your configuration causes this (which I don't think either, because it works 90% of the time).

 

Edited May 31 by Rysz

[thatja]

5 minutes ago, Rysz said:

 

Well you can try: killall -9 mergerfs

And see if that unfreezes your server.

 

I really don't see how it can be mergerFS though, when there's zero indication in the logs (apart from garbage collection, which actually says that mergerFS is still operating normally at that point) and tons of users are using it without any problems... unless something in your configuration causes this (which I don't think either, because it works 90% of the time).

 

 

Yeah, and why would the entire /mnt become unaccessible? I wonder if @JorgeB can shed any light on this.

[JorgeB]

Not without something in the logs, I've never seen this before, I suspect something that you are using is causing this, and still think it would be worth re-testing without mergefs, I don't see anything pointing directly to it, but it's still something that should be ruled out.

[thatja]

Just now, JorgeB said:

Not without something in the logs, I've never seen this before, I suspect something that you are using is causing this, and still think it would be worth re-testing without mergefs, I don't see anything pointing directly to it, but it's still something that should be ruled out.

The problem with this is my setup doesn't work without mergerfs so it wouldn't be a good test

[JorgeB]

If it's not easy to remove mergefs, you can try the other way around, remove all other plugins and leave VM and docker services disabled, then restest.

[timc6896]

Have you tried replacing the rclone mounts with local mounts?  Like add a drive or two or three.  Mount the drives and run mergerfs off of the local drives?  This eliminates rclone as a possible issue but allow you to successfully run mergerfs, except you won't have any content/files since it's local with empty hard drives.  Hope this makes sense.

[thatja]

On 5/31/2024 at 9:29 PM, Rysz said:

 

Well you can try: killall -9 mergerfs

And see if that unfreezes your server.

 

I really don't see how it can be mergerFS though, when there's zero indication in the logs (apart from garbage collection, which actually says that mergerFS is still operating normally at that point) and tons of users are using it without any problems... unless something in your configuration causes this (which I don't think either, because it works 90% of the time).

 

 

I understand you are the maintainer for mergerfs for UNRAID? Apologies if not.

 

I am still getting this issue, though nearly not as much as I was before, however I am wondering if there is a way I can improve the performance of my setup and, I acknowledge ahead of time that my setup is rather niche and unique in a sense that its not something you would come across often. Because of this I'll give you a small rundown of what's what.

 

/mnt/user/plexdata/ - This is my share for my Array on UNRAID (local)

 

/mnt/nvmedl/plexified/mounts/moviesrc - This is my movies source that is a combination of folders on Google Drive mounted via Rclone

 

/mnt/nvmedl/plexified/mounts/google/Data - This is my main Google Drive folder that is mounted via Rclone and where most of my files are located.

 

/mnt/nvmedl/plexified/mounts/secret/ - This is the end product with all 3 of the above merged together using Mergefs. The command I am using for this is;

 

mergerfs -o defaults,allow_other,statfs_ignore=nc,use_ino,category.create=ff,fsname=mergerFS /mnt/user/plexdata/:/mnt/nvmedl/plexified/mounts/moviesrc=NC:/mnt/nvmedl/plexified/mounts/google/Data=NC /mnt/nvmedl/plexified/mounts/secret/

 

and my Rclone mount is:

 

rclone mount --config=/mnt/nvme/plexified/mounts/rclone/rclone.conf --allow-other --no-traverse --vfs-cache-mode full --cache-dir /mnt/nvmedl/plexified/mounts/googlecache/ --vfs-cache-max-size 250G --dir-cache-time 96h --vfs-fast-fingerprint --vfs-refresh --drive-impersonate [email protected] googledecrypted: /mnt/nvmedl/plexified/mounts/google/

 

I then use Sonarr which via instructions from Mergerfs to import all new files to the Array folder, and my config in Sonarr docker to forward the folder in a way that my Plex server can read from an old bare metal install as below: (Plex also has this same container/host path setup)

 

 

Now, Sonarr and Radarr also use this location, and I use this to manage my files however I have noticed that importing is rather slow to do, I am wondering if there is anything in my setup that could be improved that would make my import performance faster, and I also wonder if this is what contributes to the way my server hangs and then ultimately crashes.

 

Any input is very greatly appreciated and I look forward to reading your reply.

Edited September 5 by thatja