*ALL* shares are *GONE* ... ??? (unRAID 6.6.1)

[digitalformula]

I have unRAID 6.6.1 and everything was fine last night.  I had 12 shares, all mounted via /mnt/user as per usual.  I was browsing unRAID just fine.  I use gFTP now, since FileZilla has stopped working properly since upgrading to 6.6.1 (I posted about that but got no response).

 

Today, when I login, the /mnt/user folder is *inaccessible*.  All the shares that were there in the unRAID GUI are *gone*.  The GUI says "There are no exportable user shares".

 

If I SSH into unRAID and check /mnt/user0, the files are still there so doesn't look like there's been any data loss at this point (I'll do more checking).

 

The only thing that changed recently was the installation of the "rClone" plugin by @Waseh (note that I am *not in any way* saying this plugin caused this issue so please don't think that).

 

Attached are some screenshots of what I see via SSH and GUI.

 

I have not yet stopped and restarted the array, in case something's going on and it doesn't come back.

 

As you can see, there's a notification on the screen right now saying the automatic health check has passed.

 

What's going on?  Can anyone help diagnose this?

[digitalformula]

Found others in the forums with the exact same issue as far as back 2014.  Some mentioning NFS - all my shares are NFS and syslog shows a segfault in libfuse.  Great.

 

Rebooting fixed it for them and it has fixed it for me.

 

I would like to know why, though - how do I go about making sure I know *why* this happened?

 

unRAID was rock solid until I went to 6.6 and then 6.6.1.  Since then I've had this issue + other issues relating to SSH/SFTP.  Not good.

Edited October 5, 2018 by digitalformula

[Hoopster]

32 minutes ago, digitalformula said:

Some mentioning NFS - all my shares are NFS and syslog shows a segfault in libfuse.  Great.

This has been reported as a bug introduced by the Linux kernel in unRAID 6.6.x

 

 

FUSE crashes, which kills shfs which kills nfsd and all NFS shares disappear.

 

 

Limetech has identified the cause of the bug, but, only a patched kernel will fix it.

Edited October 5, 2018 by Hoopster

[digitalformula]

1 minute ago, Hoopster said:

This has been reported as a bug introduced by the Linux kernel in unRAID 6.6.x

 

 

FUSE crashes, which kills sfhs which kills nfsd and all NFS shares disappear.

 

 

Limetech has identified the cause of the bug, but, only a patched kernel will fix it.

 

*facepalm*

 

Thanks for the info - much appreciated.

 

[waveworm]

I also am having issues.  server is good for about 12 to 20 hours and then they just go missing.  I am running on 6.6.1.

 

Thanks

[digitalformula]

2 minutes ago, waveworm said:

I also am having issues.  server is good for about 12 to 20 hours and then they just go missing.  I am running on 6.6.1.

 

Thanks

 

I believe the only "permanent" fix is to roll back to 6.5.3.  Rebooting fixes it, but only temporarily.

[thoster]

Same here. Upgraded yesterday to 6.6.1 and after ~18 hours all shares goes missing. Rebooting fixes it temporary, time to rollback to 6.5.3.

[digitalformula]

2 hours ago, thoster said:

Same here. Upgraded yesterday to 6.6.1 and after ~18 hours all shares goes missing. Rebooting fixes it temporary, time to rollback to 6.5.3.

For the love of <insert preferred deity here> please backup your flash drive before you roll back.

 

I was told by someone (whom I thought knew more than me) that there's nothing important on the flash drive and that re-imaging it wouldn't lose anything.  After 10 minutes I realised that's completely false and I'd lost every bit of configuration I'd spent months setting up.

 

unRAID does have the Flash Backup option in the GUI but ... yeah.  I didn't know about it until it was too late and there's no "backup" of the configuration on the array that I'm aware of (if there and I don't know about it, I'll be sure to use it next time).

[Hoopster]

2 hours ago, digitalformula said:

unRAID does have the Flash Backup option in the GUI but ... yeah.  I didn't know about it until it was too late and there's no "backup" of the configuration on the array that I'm aware of (if there and I don't know about it, I'll be sure to use it next time).

Install the CA Backup/Restore Appdata plugin.  As you can see from the screenshot of my plugin config below, it can automatically backup the flash drive (USB Backup Destination:) to the array, or, another location you specify. 

 

I  have mine configured to backup the flash drive to the cache drive weekly.  Technically, I guess it is also backed up to the array since I backup the flash drive to a folder in appdata and then I backup appdata (including the flash drive folder) to the array.

 

Edited October 10, 2018 by Hoopster

[digitalformula]

10 minutes ago, Hoopster said:

Install the CA Backup/Restore Appdata plugin.  As you can see from the screenshot of my plugin config below, it can automatically backup the flash drive (USB Backup Destination:) to the array, or, another location you specify. 

 

I  have mine configured to backup the flash drive to the cache drive weekly.  Technically, I guess it is also backed up to the array since I backup the flash drive to a folder in appdata and then I backup appdata (including the flash drive folder) to the array.

 

 

Yep, I know there are plugins that can do it and this one is a good one.  I have it installed and configured the same way.

 

My comment was more re unRAID not doing anything like this automatically by default (disclaimer, again: if it does, I didn't know).  IMO it should take a Flash backup and store it on one of the data disks/user shares without the user having to do anything.  E.g. the "system" share that unRAID has when it is installed.

[Hoopster]

2 minutes ago, digitalformula said:

My comment was more re unRAID not doing anything like this automatically by default (disclaimer, again: if it does, I didn't know).  IMO it should take a Flash backup and store it on one of the data disks/user shares without the user having to do anything.  E.g. the "system" share that unRAID has when it is installed.

 

You should submit that as a feature request. 😀

[JonathanM]

8 hours ago, digitalformula said:

IMO it should take a Flash backup and store it on one of the data disks/user shares without the user having to do anything.  E.g. the "system" share that unRAID has when it is installed.

That's a good idea! Fleshing it out a bit more, I can actually see a way to make it relatively foolproof.

 

 Unraid automatically makes a flash backup along with current diagnostics every time the array is started successfully, and saves it as a generational backup in the system folder, limited to a user defined number (last 5 or whatever).

 

A fresh unlicensed install mounts each drive it finds as read only and looks for a backup. If it finds some, it offers the dated list to restore from backup and reboot, if no backup is found alternatively go through the trial key process.

 

Here is where I think it might get a little tricky. On first array start after a restoration, I think we need to force a different startup mode, one that hasn't been defined up to now, but would be useful even if the backup and restore isn't implemented.

 

I propose a startup mode where all array drives are mounted read only, and a parity check is started. If any sync errors are found, it immediately aborts and warns that something is wrong. If no errors in the first arbitrary amount of space, it returns to a screen where a normal array start is offered.

 

Opinions? @bonienl, @trurl, @Squid, @johnnie.black, et. al?

[KinGSiZ3]

I had the same Problem day's ago .. After i reinstall the Network Status Plugin and reboot, unRAID Shares working like a charme again.

 

Greetz

[JorgeB]

40 minutes ago, jonathanm said:

Unraid automatically makes a flash backup along with current diagnostics every time the array is started successfully, and saves it as a generational backup in the system folder, limited to a user defined number (last 5 or whatever).

Seems like a good idea to me, even if at first only the auto backup is done, that's the most important thing, restore options would be great but if needed we can guide an user to manually mount each disk read only to find and restore the config folder to the flash drive.

[Squid]

1 hour ago, KinGSiZ3 said:

After i reinstall the Network Status Plugin and reboot

I hope you mean uninstall

[z0rlac]

As a new unRAID user, a HUGE thank you for mentioning the Network stats plugin, and removing it fixing the issue. My new/trial install just sprung back into life, and as it is my first unRAID install, I didnt have the ability to rollback to 6.5.x.

 

[pluginCop]

2 hours ago, z0rlac said:

HUGE thank you for mentioning the Network stats plugin, and removing it fixing the issue.

Unless you installed it right when it was released, before the problems became apparent (ie: it caused my system no problems prior to it being publicly available), you should have seen a comment within the Apps tab about it when installing.  Additionally, if you also have fix common problems plugin installed, it should have complained about it being installed also.

[z0rlac]

It didnt complain at the time, it didnt show any issues were outstanding (or at least I dont think it did)... but as a new user, maybe I wasnt looking for those comments in my over enthusiastic install frenzy I learned a few lessons:

 

#1 - dont install everything

#2 - check the support threads for the plugins/dockers before installing it

#3 - the community forums (plus possibly reddit) are the place to lurk to watch for this kind of things

 

I am just so happy its all working again!

 

 

[digitalformula]

12 hours ago, KinGSiZ3 said:

I had the same Problem day's ago .. After i reinstall the Network Status Plugin and reboot, unRAID Shares working like a charme again.

 

Greetz

 

Hey - wondering ... are you on 6.6.1 and using NFS shares?  Or are you using SMB shares?

 

Just asking if you can confirm that?  If removing the Network Stats plugin has completely solved the issue, I’d like to upgrade to 6.6.1 again and start troubleshooting this more (I rolled back to 6.5.3).  I don’t think I have Network Stats installed though ...

[KinGSiZ3]

@Squid ... sorry yes mine uninstall 😄

@digitalformula i use NFS Shares 

 

 

[dorgan]

As a new unRAID user, a HUGE thank you for mentioning the Network stats plugin, and removing it fixing the issue. My new/trial install just sprung back into life, and as it is my first unRAID install, I didnt have the ability to rollback to 6.5.x.
 

Network Stats plugin was fixed with today’s release. Sorry for causing all the issues.

[digitalformula]

25 minutes ago, dorgan said:

Network Stats plugin was fixed with today’s release. Sorry for causing all the issues.

Personally I think an acknowledgment is more than enough.  An apology isn't even required.  🙂

 

Honestly, these things happen and I'm totally OK with that - it's when issues get ignored by the devs that I get a little frustrated.

 

@dorgan Good on you for responding!

[BRiT]

As LimeTech mentioned multiple times, please post up your full diagnostics zip file whenever crashes like this happen. The build of unRaid 6.6.2 has additional instrumentation in SHFS (user filesystem) to help them try to get the additional details they need.

[digitalformula]

1 hour ago, BRiT said:

As LimeTech mentioned multiple times, please post up your full diagnostics zip file whenever crashes like this happen. The build of unRaid 6.6.2 has additional instrumentation in SHFS (user filesystem) to help them try to get the additional details they need.

Thanks and I would love to but I was on 6.6.1 when the issue happened.  The additional emhttp instrumentation for FUSE/shfs mentioned in the 6.6.2 release notes wasn't there in 6.6.1.

 

I have rolled back to 6.5.3 and it's not an option to upgrade to 6.6.2 just so I can collect a diagnostics file, knowing NFS is going to crash.

 

The end result is an unRAID system that would be unusable until the issue is resolved (I don't use SMB shares at all - this is a Linux shop).  This could be ages away, considering I'm led to believe the issue is caused by the current in-use kernel version.

Edited October 16, 2018 by digitalformula

[waveworm]

Do we know if 6.6.3 has fixed the issues?