I am having a problem where every night my array goes offline and requires a reboot: 

/bin/ls: cannot access '/mnt/user': Transport endpoint is not connected

I found this post and have tried ran through the procedure of checking the disk filesystems both via the terminal as well as via the web ui for both my xfs, and btrfs cache drives (which have a different procedure). I took a look in the syslog and I can't find any clues. This box has been running relatively trouble-free for years so I am asking the community for some help. I have attached my anonymized diagnostics. 

black-diagnostics-20220730-1159.zip

Edited July 30, 20223 yr by old man
added detail

Are you running unifi-controller (or any other app) on its own IP address?  Go to Settings Docker, disable the service and then in advanced view set the network driver to be ipvlan instead of macvlan

I made this change and rebooted. I will report back if there are no further crashes in 2 days. :fingers_crossed: Out of curiosity and for posterity what lines led you to believe that could be the problem? 

Edited July 31, 20223 yr by old man

Yes I am running apps on their own ip addresses and made the above change but the problem persists. Attached is an updated diagnostics zip. Any help would be greatly appreciated. 

black-diagnostics-20220803-1210.zip

Aug  3 02:15:23 black shfs: shfs: ../lib/fuse.c:1451: unlink_node: Assertion `node->nlookup > 1' failed.

 

It's the issue below:

 

 

Some workarounds discussed there, mostly disable NFS if not needed or you can change everything to SMB, can also be caused by Tdarr if you use that.

Interesting. I disabled NFS and will report back in a few days with an update. 

I disabled nfs but the problem persists this is driving me nuts. I see the related log entry now at least so thx for pointing that out. I don't use tdarr and I can methodically try the other work arounds. 

logs/syslog.txt:Aug  4 09:17:04 black shfs: shfs: ../lib/fuse.c:1451: unlink_node: Assertion `node->nlookup > 1' failed.

This usually happens when some program tries do delete/move files that are already not there.

Got it. I stopped most the many docker containers I am running including prometheus, grafana, and glances, and I have had 4 days of uptime where I was once having nightly crashes. I also set the mover to invoke every night vs every hour. I will methodically enable one at a time to determine the root cause application and post back here. Thank you VERY much for your help @JorgeB and @Squid. 

On 7/30/2022 at 10:31 AM, Squid said:

Are you running unifi-controller (or any other app) on its own IP address?  Go to Settings Docker, disable the service and then in advanced view set the network driver to be ipvlan instead of macvlan

 

I wonder if we will ever have rock solid docker implementation on unraid. I just ran into this issues and had to change from ipvlan to macvlan, then reboot to resolve.