I thought it would be better to make a new thread than attempting to hijack someone elses thread for the same issue I was having. I've attached my diagnostics.

 

On line 3838 in the syslog.txt, I have:

Mar 13 21:32:46 Tower shfs: shfs: ../lib/fuse.c:1402: unlink_node: Assertion `node->nlookup > 1' failed.

 

At some point I had attempted to hard link some files from one share to another but I have since deleted those particular files.

I have also set the "Number of fuse File Descriptors" to 813384, which slowed down this issue from happening every day to every few days.

 

Is there a way to find or locate the problem? I have no idea what is linked where anymore and I'm getting really tired of having to reboot my server every few days to temporarily solve this issue. If I have to wipe and reinstall, so be it, but I'd prefer to fix this issue first if possible.

tower-diagnostics-20250314-0804.zip

Are you using user and disk shares at the same time?

 

Also, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, this seems to help for some users.

16 minutes ago, JorgeB said:

Are you using user and disk shares at the same time?

Is there a CLI command I can run to verify? Everything I map is under the /mnt/user/<share name>/

I have only accessed the direct disk shares to delete files individually.
 

 

17 minutes ago, JorgeB said:

Also, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, this seems to help for some users.

"I have also set the "Number of fuse File Descriptors" to 813384, which slowed down this issue from happening every day to every few days."

42 minutes ago, SkilledAlpaca said:

I have only accessed the direct disk shares to delete files individually.

It's very possible doing that causes that shfs crash, try deleting them from /mnt/user as well.

 

43 minutes ago, SkilledAlpaca said:

"I have also set the "Number of fuse File Descriptors" to 813384

Sorry, missed that.

50 minutes ago, JorgeB said:

It's very possible doing that causes that shfs crash, try deleting them from /mnt/user as well.

I think I got the files that I mapped or linked, but I have no way of knowing without waiting a few days. Is there a way to find them?

33 minutes ago, SkilledAlpaca said:

I think I got the files that I mapped or linked

Not sure what you mean by this, just don't delete more files from /mnt/<disk share> and see if it still happens.

1 minute ago, JorgeB said:

Not sure what you mean by this, just don't delete more files from /mnt/<disk share> and see if it still happens.

I deleted the files that I know for a fact that I used sym/hard links on under /mnt/user/<share>

Unless I'm mistaken on the actual problem here.

OK, so recommend retesting to see whether the issue is resolved or not.

The issue happened again. I've attached the diagnostics. Same error in the syslog.

 

Is there a CLI command that can help me narrow down possible links?

tower-diagnostics-20250316-1425.zip

11 hours ago, SkilledAlpaca said:

Is there a CLI command that can help me narrow down possible links?

Not that I know of, are you still using disk and user shares to access files?

 

 

7 hours ago, JorgeB said:

Not that I know of, are you still using disk and user shares to access files?

 

 

I personally am not using disk shares and all my docker containers are mapped using the /mnt/user/ file paths.

Then that should not be the issue, and if you have already raised Number of fuse File Descriptors, possibly not much else you can do, other than try and find which app is causing the errors and try working around it.

I did a more thorough search through my docker containers and noticed that my Beszel Agent was mapped to /mnt/cache/

If I'm understanding this correctly, this might be the issue?

 

If I remap this to /mnt/user/<cache share>, could it possibly this issue? I'll make that change now and test for a few days.

32 minutes ago, SkilledAlpaca said:

/mnt/user/<cache share>

It would be /mnt/user/<share name>, but worth doing it and testing.

Changing the Beszel container was not the fix. Is my only resolution to perform a full and clean reinstall?

32 minutes ago, SkilledAlpaca said:

perform a full and clean reinstall?

Reinstall of what?

Just now, trurl said:

Reinstall of what?

Of Unraid. Since there appears to be no way to solve this issue without deleting all data and starting over, hoping it doesn't happen again. This wasn't an issue until I upgraded to 7.0.0 but I could be wrong and only noticed it since then.

12 minutes ago, SkilledAlpaca said:

Of Unraid. Since there appears to be no way to solve this issue without deleting all data and starting over

No reason to expect that would solve anything.

 

Unraid is already a clean install at each boot. The OS is contained in archives on flash. Those archives are unpacked fresh into RAM at each boot, and the OS runs completely in RAM. Think of it as firmware.

 

The only time those original archives are changed is when you upgrade to a new version.

 

The flash drive also contains all settings from the webUI (your configuration) in the config folder. These are applied at boot after the OS is unpacked into RAM.

 

9 minutes ago, SkilledAlpaca said:

deleting all data

The only "data" on your disks that can possibly actually do anything is your dockers and VMs. The rest is just data.

 

11 minutes ago, SkilledAlpaca said:

This wasn't an issue until I upgraded to 7.0.0 but I could be wrong and only noticed it since then.

Have you tried rolling back?

2 hours ago, trurl said:

No reason to expect that would solve anything.

 

Unraid is already a clean install at each boot. The OS is contained in archives on flash. Those archives are unpacked fresh into RAM at each boot, and the OS runs completely in RAM. Think of it as firmware.

 

The only time those original archives are changed is when you upgrade to a new version.

 

The flash drive also contains all settings from the webUI (your configuration) in the config folder. These are applied at boot after the OS is unpacked into RAM.

Ah good to know, thank you.

 

2 hours ago, trurl said:

The only "data" on your disks that can possibly actually do anything is your dockers and VMs. The rest is just data.

So if my docker containers and VMs are setup using /mnt/user/<share>, then there shouldn't be any issues? Is that a proper understanding of your statement?

 

2 hours ago, trurl said:

Have you tried rolling back?

I was unaware of how easy rolling back was. I am now back on 6.12.14 and will wait to see if it happens again.

Issue has presented itself again after 2.5 days of being on 6.12.14

 

There has to be a way to figure what is causing this issue. What can I do?

12 hours ago, SkilledAlpaca said:

of being on 6.12.14

This is expected since this has been happening to some users for a long time, the only thing I can think of is to try and find the app that is causing the problem, you can for example, start only half the container you run, if the same try the other half, then keep drilling down.

Upgraded to 7.0.1

 

Going on 3.5+ days of uptime with no crash. Hoping it continues for at least 10 days before I re-enable the following containers one by one and test.

• ArchiveTeam-Warrior

• CodeProject.AI_ServerGPU

• Dozzle

• beszel-agent

• beszel

The above were not the cause.
 

Edit: I'm currently only running the following containers and testing:

• immich
• PostgreSQL_Immich
• Redis_Immich
• Apache-Tika-Server
• gotenberg
• Redis_Paperless
• paperless-ngx
• Plex
• Transmission
• audiobookshelf
• TTVPointsMiner

Edited April 5, 20251 yr by SkilledAlpaca

I am still battling with this problem. I'm slowly narrowing down the containers that are causing this issue and will update further.

 

I do however have a question. Would the Appdata Backup plugin accessing `/mnt/cache/appdata` possibly be the issue? This runs nightly for me, but I don't think I had any issues when I only had minimal critical apps. I've remove that directory for now from it's config and will push the plugin schedule out to every week instead of nightly.

Edited May 15, 20251 yr by SkilledAlpaca

Not sure if it could be related, but worth trying disabling it for a couple of days to test.

I have narrowed it down to five + one containers. I ended up setting up a simple nginx:stable container as a canary page that is monitored from another host. Within twenty seconds of the shfs service crashing, I'll get an alert to review.

• ghcr.io/advplyr/audiobookshelf

• plexinc/pms-docker

• lscr.io/linuxserver/prowlarr

• lscr.io/linuxserver/sabnzbd:latest

• lscr.io/linuxserver/sonarr

• nginx:stable

It doesn't appear the AppData Backup Plugin is the reason either. The crashes happened on days and times that were not at the temporary once a week backup time.

Finally, I've observed (without any backing evidence) when sabnzbd is quickly running through a lot of files, the error seems to follow shortly afterwords.