SMB Shares crashed and no longer in network after 6.11.2 update

Post the output of this command:

ulimit -Sn

 

I think you're running into the soft file limit, not the hard file limit.

7 hours ago, dlandon said:

Post the output of this command:

ulimit -Sn

40960.

However, this is after I was forced to reboot the server.

I did not include it earlier (above) as the command did not return anything meaningful. (i don't recall its exact wording but it something like that command was unknown???).

 

PS: I did restart the SMB service before rebooting, but it didn't have any effect.
PSS: I have more copying/moving via SMB today, so will monitor if the issue reappears.

Edited November 7, 2022 by PhilipJFry

We've had several reports of the soft file limit changing to 16424 causing the issue you are seeing.

 

The soft limit of 40960 is correct.  Something changed or corrupted the value in your case.  If you see it again, post your diagnostis.

3 minutes ago, dlandon said:

We've had several reports of the soft file limit changing to 16424 causing the issue you are seeing.

 

The soft limit of 40960 is correct.  Something changed or corrupted the value in your case.  If you see it again, post your diagnostis.

Just crashed again now.. @dlandon 😞
Let me know if there are any other diagnostic logs that would be helpful.

consolelog 2.txt syslog 2.txt

What does 'ulimit -Sn' show now?

Your log shows:

Nov  8 08:47:35 Futurama  smbd[26576]:   Too many open files, unable to open more!  smbd's max open files = 16424

 

Something is resettig the soft file limit.  Log entry shows 16424.

 

Stop these log messages:

Nov  8 08:47:35 Futurama  smbd[26576]: [2022/11/08 08:47:35.393719,  0] ../../source3/smbd/files.c:1199(synthetic_pathref)
Nov  8 08:47:35 Futurama  smbd[26576]:   synthetic_pathref: opening [events] failed

by entering this:

logging = 0

in your smb-extra.conf file and reboot.

 

Your log might be filling up with these messages causing problems.

 

If you continue have issues, post the complete diagnostics, not just log snippets.

NPS. Will do the above.

 

Morning @dlandon... happened again this morning.

Full logs attached for your reference.

If there is any other helpful info you need, please let me know.

 

PS: Think i found my earlier mysterious "command not found" for ulimit -Sn

I typed unlimit -Sn! LOL.

futurama-syslog-20221108-2008.zip futurama-diagnostics-20221109-0709.zip

Try putting this command in the go file:

ulimit -Sn 40960

and reboot.

 

One user was able to solve their issues with this command in the go file.

25 minutes ago, dlandon said:

Try putting this command in the go file:

ulimit -Sn 40960

Done. Fingers crossed. 🙂

Hi @dlandon, I added the above and rebooted. But sadly the issue re-appeared shortly after. 😞futurama-diagnostics-20221109-1046.zipfuturama-syslog-20221108-2345.zip

Next things to try:

• Update all your plugins.  Some are out of date.
• Fix Common Problems is reporting problems:

Nov  9 10:34:00 Futurama root: Fix Common Problems Version 2022.10.17
Nov  9 10:34:01 Futurama root: Fix Common Problems: Warning: Share media set to not use the cache, but files / folders exist on the cache drive
Nov  9 10:34:01 Futurama root: Fix Common Problems: Warning: Docker Application DiskSpeed has an update available for it
Nov  9 10:34:01 Futurama root: Fix Common Problems: Warning: Docker Application EmbyServer has an update available for it
Nov  9 10:34:01 Futurama root: Fix Common Problems: Warning: Docker Application Jellyfin has an update available for it
Nov  9 10:34:01 Futurama root: Fix Common Problems: Warning: Docker Application sonarr has an update available for it
Nov  9 10:34:01 Futurama root: Fix Common Problems: Warning: Share FetchTV is set for both included (disk4) and excluded (disk1,disk2,disk3,disk5) disks

Fix those issues.

• If the issue peresists, boot in safe mode and try your transfers.  If that works, add plugins one at a time and see if one of them causes the problem.
• If everything works with plugins installed, enable Docker containers one at a time and see if the problem occurs again.

Hi @dlandon. I will do updates as noted above 🙂

FYI: Re docker containers and versions. I actually only use 2 (homeassistant and krusader) which are both set to autostart.

The rest are only started and used when needed - which is rarely (sometimes never. i really should get around to removing them at some point as they dont work/run).

Edited November 9, 2022 by PhilipJFry

Changed Priority to Urgent

Hi @dlandon. No such luck. 😞

After upgrading all plugins/docker etc... the issue persisted.

After booting into safe mode - the issue persisted (even more quickly than before).

Diags attached for your reference.

futurama-syslog-20221109-0957.zipfuturama-diagnostics-20221109-2055.zipfuturama-syslog-20221109-0957.zipfuturama-diagnostics-20221109-2055.zip

 

The problem has now gone from being mildly annoying to completely frustrating!

I would appreciate that the issue is resolved with speed, because as of now, my server is no longer "fit for purpose"

 

I am unfamiliar if you are an employee of limetech or more a helpful community developer?... but I do appreciate your assistance in respect of the issue to date. 👍🏻

Out of curiosity I was checking my server and there are 3 running smb PIDs, and one of then has a lower soft limit (16464) than the others (40960), type in the console:

ps aux | grep smbd

then check the limits for all smd processes with:

cat /proc/<pid>/limits

 

5 hours ago, PhilipJFry said:

Hi @dlandon. No such luck. 😞

 

The problem has now gone from being mildly annoying to completely frustrating!

 

I would appreciate that the issue is resolved with speed, because as of now, my server is no longer "fit for purpose"

 

I am unfamiliar if you are an employee of limetech or more a helpful community developer?... but I do appreciate your assistance in respect of the issue to date. 👍🏻

 

This is classic Unraid. There are critical issues and nobody from Unraid responds. It's great there are many users here who try to help but they don't have access to the code, they don't know what was tested or not tested for each release, they don't have access to any sort of bug reporting database, and the list goes on of how we're all left to just fend for ourselves with a paid closed source product where there's extremely little participation from anyone official at Unraid. Support for Unraid is literally worse than support for a lot of open source software for the reasons I just listed. 

45 minutes ago, dev_guy said:

 

This is classic Unraid. There are critical issues and nobody from Unraid responds. It's great there are many users here who try to help but they don't have access to the code, they don't know what was tested or not tested for each release, they don't have access to any sort of bug reporting database, and the list goes on of how we're all left to just fend for ourselves with a paid closed source product where there's extremely little participation from anyone official at Unraid. Support for Unraid is literally worse than support for a lot of open source software for the reasons I just listed. 

You are completely off base.  @JorgeB and I are here trying to help you.  We are both involved with LT and are doing the best we can to help you.  Instead of complaining, please provide the information asked of you.

15 minutes ago, dlandon said:

You are completely off base.  @JorgeB and I are here trying to help you.  We are both involved with LT and are doing the best we can to help you.  Instead of complaining, please provide the information asked of you.

 

I've been very appreciative of the support I've received from you, JorgeB, and others and have said so many times in my replies. But it's not at all clear who is acting in an official capacity on behalf of Unraid and who is just being a helpful experienced user? I also wasn't asking for help as but more making an observation on the state of things with Unraid these days. The 6.11.2 release is a good example of something that should have never happened.

55 minutes ago, dev_guy said:

 

I've been very appreciative of the support I've received from you, JorgeB, and others and have said so many times in my replies. But it's not at all clear who is acting in an official capacity on behalf of Unraid and who is just being a helpful experienced user? I also wasn't asking for help as but more making an observation on the state of things with Unraid these days. The 6.11.2 release is a good example of something that should have never happened.

@JorgeB and I are here.  When it comes to samba issues, I am one of the lead LT guys.  What more are you wanting to hear?  We are on it and can effect a quick fix once we have a handle on what is wrong.  We have to find it first.

 

If we cannot reproduce an issue in our beta and rc tests, we cannot fix it.  We are not Microsoft with a test lab with thousands of computers to test on, so some things unfortunately will get by us.

 

Take your car to a mechanic and tell the tech it's not running right.  If the tech cannot reproduce the problem, how can he fix it?  Same here with software.

 

I did some research on this problem because others have reported it also, but don't understand at this point how the soft file limit is getting set so low for samba.

 

Let's end this discussion with the commitment from me (I am part of the LT team) that I will do everything I can to shoot this probelm down and get it fixed, but we need your help by providing the information that was asked for.

Just now, dlandon said:

Let's end this discussion with the commitment from me (I am part of the LT team) that I will do everything I can to shoot this probelm down and get it fixed, but we need your help by providing the information that was asked for.

Fair enough. Your help is always appreciated. I'm not the OP or the one having this particular SMB problem but it does make me hesitant to upgrade to 6.11.3 when this issue seems to be outstanding with no official answers.

4 minutes ago, dev_guy said:

Fair enough. Your help is always appreciated. I'm not the OP or the one having this particular SMB problem but it does make me hesitant to upgrade to 6.11.3 when this issue seems to be outstanding with no official answers.

Not everyone is having this issue.  Many people don't have a problem.  That's why it's so hard to find.

7 hours ago, JorgeB said:

and one of then has a lower soft limit (16464) than the others (40960)

This is apparently normal, the 1st smb process has a lower soft limit, at least I downgraded as far back as v6.9.2 and it's the same:

 

root@Tower15:~# ps aux | grep smbd
root      4202  0.0  0.4  87384 18048 ?        Ss   18:52   0:00 /usr/sbin/smbd -D
root      4205  0.0  0.2  85436  9460 ?        S    18:52   0:00 /usr/sbin/smbd -D
root      4206  0.0  0.1  85428  4476 ?        S    18:52   0:00 /usr/sbin/smbd -D
root      5962  0.0  0.0   3904  1988 pts/0    S+   18:52   0:00 grep smbd

root@Tower15:~# cat /proc/4202/limits | grep files
Max open files            16464                40960                files 

root@Tower15:~# cat /proc/4205/limits | grep files
Max open files            40960                40960                files     

 

 

 

@PhilipJFrywhat was the previous release you were running?

Hi guys @JorgeB @dlandon.

I was previously running 6.1.11 I believe. I'm a newbie and have only recently, last month, bought a license.

Re Max open files.... yup.. mine looks similar with 1 PID as 16464.

 

One thing i've noticed when the issue occurs, post reboot, is that all the files/folders that were in transit to the server are all greyed out when viewing the SMB share (from an iMac's perspective).

 

My iMac bombs out with I think an error 24, when I try to merge or copy/replace the same folders/files. They appear "stuck" (locked?), and the only way to start copying again to same, is by going into the WebGUI and deleting them all.