All shares vanished

[nnhoang]

Hi, I'm running unRAID v 6.8.3.

 

Every 2-3 days for the past week or two, my shares completely vanish until I reboot the server.

 

For Plugins, I  have

• Community Applications
• User Scripts
• Command Line Tool
• Dynamix Schedules
• Dyanmix SSD TRIM
• Dyanmix System Information
• Dyanmix System Statistics
• Dyanmix System Temperature
• Recycle Bin
• ssh Plugin

 

The only thing I can remember before it happening was updating a couple of plugins. I don't exactly remember which as unRAID has been very stable so I didn't think much about it. Now it's become a major annoyance. I have to reboot the server every time I start to watch a movie and Plex tells me the file can't be found.

 

Where can I find any logs, or diagnostics to narrow down what's happening?

 

Thank you!

[trurl]

If possible before rebooting and preferably with the array started
Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

[nnhoang]

Hi, I already rebooted the server (to watch movies), so I'll try getting the Diagnostic file next time it happens. For now, here's the file after rebooting if it helps...

nnvault-diagnostics-20201215-1905.zip

[nnhoang]

On 12/15/2020 at 8:27 PM, trurl said:

If possible before rebooting and preferably with the array started
Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

Hi, it happened again tonight while trying to access files from Plex.

 

Here's the diagnostic files!

nnvault-diagnostics-20201218-2211.zip

[JorgeB]

Same issue as this:

 

I believe most usually caused by a docker moving/renaming files.

[nnhoang]

The only docker I use is a telegraph instance to track statistics with Graphana. Do you think that might be the cause? Just turn it off to test?

[JorgeB]

19 minutes ago, nnhoang said:

Just turn it off to test?

If it's the only one do it.

[nnhoang]

On 12/19/2020 at 8:41 AM, JorgeB said:

If it's the only one do it.

 

Well it happened again, all shares are gone from the shares tab and if I ssh into my Plex VM then try to cd into the unraid share, it says permission denied.

 

When I ssh into the unraid box, and try to cd into /mnt/user it gives me an error

Transport endpoint is not connected

 

I have also attached another diagnostics file. Any help appreciated... it's getting really annoying now.

nnvault-diagnostics-20201220-1007.zip

[JorgeB]

Probably caused by this unmount:

Dec 18 03:10:27 NNVault rpc.mountd[4373]: authenticated unmount request from 192.168.1.10:708 for /mnt/user/NNVault (/mnt/user/NNVault)
Dec 18 03:13:31 NNVault shfs: shfs: ../lib/fuse.c:1451: unlink_node: Assertion `node->nlookup > 1' failed.

Dec 20 03:10:25 NNVault rpc.mountd[4465]: authenticated unmount request from 192.168.1.10:732 for /mnt/user/NNVault (/mnt/user/NNVault)
Dec 20 03:14:05 NNVault shfs: shfs: ../lib/fuse.c:1451: unlink_node: Assertion `node->nlookup > 1' failed.

 

See if it happens always a few minutes after that.

 

 

[nnhoang]

Looks like that unmount with the IP address is my esxi backup script running nightly. Odd that it's causing issues as it's been running fine since I first spun up my unRaid box. I tried to manually run my daily backup scripts that mount/unmount mounts but the shares are still there.

 

I do have a few backup scripts running simultaneously on different mounts so that could be the cause? I'll let it run again tonight to see if it causes any errors, I did also run a parity check last night with no errors.

 

Any other suggestions where to look?

[nnhoang]

Hi, so I have an update. I found many posts with similar errors, but they were largely left without a solution so hopefully this helps someone else!

 

I tried moving my script's cronjobs around and noticed the 'unlink_node: Assertion `node->nlookup > 1' failed' error appeared roughly around the same time, but when I moved it a few hours ahead of schedule and had nothing around the 3am mark, my shares stopped vanishing.

 

I seeemed to have narrowed it down to the 'Recycle Bin' plugin. The Recycle Bin plugin was set to run it's cleanup and dump it's folder around 3am too.I believe since I'm moving and deleting a large amount of files right around the same area, things got wonky. Easiest solution for me was to just remove the Recycling Bin plugin altogether. I just never used it and I'd rather not schedule around it. It's been about 5 days now without shares vanishing so hopefully that was it!

[JorgeB]

17 hours ago, nnhoang said:

It's been about 5 days now without shares vanishing so hopefully that was it!

Goods news, please keeps us posted if you find anything different.

[t3]

May I add, that Recycle Bin is most likely not the main culprit here (or any at all), just a symptom of an underlying problem with fuse.

 

This same problem happens to me regularly for ~two years now, at completely "random" times, but usually when severe activity from various clients happen (Unraid is still a file server, after all). A couple months ago, it seems like I had identified Folder Caching as a possible reason, but even after disabling it, the problem still occurs, though less frequent...