Hi all,

 

I'm sorry in advance if this comes across a little panicky. I'm doing my best to remain calm, but I've got over 10TB of data that I'm worried about losing at the moment.

 

So today, when I got back from work, I was prompted that there was an update to unRAID available. Naturally, I installed it and rebooted. That's when my problem kicked in. As soon as my system came back online, I noticed that I couldn't access any of my server's share's through my desktop's mapped network drive. Only the "flash" folder was visible. I decided to load up the web UI to check on it, and my Shares tab is completely empty. All of my shares are gone, apparently.

 

I checked to make sure that the data still exists on the drives, and it does. I can still see all of my files and folders through /mnt/disk*, but /mnt/user is completely empty.

 

My next thought was "okay, let's just try adding the shares back." So I tried that, only to be told "Share $share has been deleted" immediately after clicking "Done." It doesn't matter if the share is one that existed previously, or one that is completely unique. I can't add anything to the machine.

 

Attached is my diagnostics zip file and a couple of screenshots. Hopefully someone here can help me out and get me access to my data once more.

 

I have all of my data backed up elsewhere as well, but given the speeds and data caps my I$P imposes on me, it would take almost a year to download it all again, and I'd really rather avoid that.

 

I'm sorry, but I wasn't tailing the syslog to a file before the initial shutdown. I know I should be doing that, but I just didn't think of it.

 

 

 

 

 

tower-diagnostics-20181015-1727.zip

Edited October 15, 20187 yr by markswam
Added more screenshots

What version did you upgrade from? 

I'm not entirely sure, but I want to say either 6.6.0 or 6.6.1. I know that I've updated at least once within the last month.

Edited October 15, 20187 yr by markswam

I've always done the updating through the web interface, since I don't know the steps for any other way off the top of my head.

Update: Something appears to be deleting every single folder in /mnt/user upon startup. I manually created /mnt/user/testFolder using mkdir, verified that the folder is there using ls, rebooted, and the folder is now gone.

This is happening to many users - the fuse file system which supports the 'shares' under /mnt/user/ appears to be failing. If you check the forums you'll see that many people have this issue.

My limited observations lead me to believe that people with newer hardware seem to be more affected.

In your observations, have you found anyone who's figured out a fix for this?

6 minutes ago, markswam said:

In your observations, have you found anyone who's figured out a fix for this?

This only seems to affect users who have NFS access to shares enabled (as you do according to your diagnostics).  I have only SMB access to shares and have never experienced the problem in any 6.6.x version.  I am currently running 6.6.2 on both servers.

 

It doesn't matter if you also have AFP or SMB, the problem is NFS.  It kills /mnt/user and so no shares are visible to AFP or SMB either.

 

The only permanent solution (until there is a kernel fix for this) at this time is to roll back to unRAID 6.5.3

Edited October 15, 20187 yr by Hoopster

There's more discussion about similar issue here:

 

I personally think that Linux support for Ryzen/Epyc is about where it would normally be - as in a few years behind. I expect that kernel 4.19+ will sort out some of the AMD issues that people are experiencing.

Well, alrighty. I'll try that. 

 

Just to make sure, is the process still just copying and overwriting bzroot, bzimage, and bzroot-gui on the flash drive?

11 minutes ago, markswam said:

Well, alrighty. I'll try that. 

 

Just to make sure, is the process still just copying and overwriting bzroot, bzimage, and bzroot-gui on the flash drive? 

There are a few extra files that now need to be copied.

 

If the last version you had installed prior to upgrading to a 6.6.x version was 6.5.3, you can roll back to 6.5.3 from the GUI via Tools>Update OS>Restore.

 

If 6.5.3 was not your previous version, you can download it from unraid.net and copy over all of the following files (this is what is stored in the "previous" folder on the flash) The most important are the bz* files:

 

Edited October 15, 20187 yr by Hoopster

Further update:

 

The previous version I had was 6.6.1. I tried rolling back to that, just to see what would happen, and everything is back the way it was before.

 

That said, I'm still going to roll back to 6.5.3, for stability's sake.

2 minutes ago, markswam said:

Further update:

 

The previous version I had was 6.6.1. I tried rolling back to that, just to see what would happen, and everything is back the way it was before.

  

That said, I'm still going to roll back to 6.5.3, for stability's sake.

 

Because after a reboot, everything is generally fine for a few minutes or hours.  With any 6.6.x version, if you have NFS access enabled, shares will eventually disappear when FUSE crashes and takes /mnt/user with it.

 

It's hard to say if this happens on *every* 6.6.x system with NFS enabled, but, those who have reported the problem have all had NFS enabled on a 6.6.x system.

@markswam Also, if you have the Network Stats plugin installed, uninstall it and install the latest version.  It was causing some share-related issues as well and has been updated by the author.

Edited October 16, 20187 yr by Hoopster

Thank you Hoopster. There's something amiss with newer hardware and linux kernels. It's not only AMD as I've seen a similar issue with intel recently.

Definitely do not panic, one of the huge advantages to unraid is that all the data is on disks in a simple manner. No stripes, nothing complicated. The key thing is we have /mnt/disk# 

Rolling back is a great idea and so good this sorted things out. watch updates for kernel 4.19 or greater 

Del

 

The problem appears to be the nerdpack plugin overwriting glibc with an older version.

11 hours ago, johnnie.black said:

The problem appears to be the nerdpack plugin overwriting glibc with an older version.

@markswam you should check this possibility. There are other reports of nerd pack plugins causing shares to disappear in 6.6.2 as noted in the 6.6.2 release thread:

 

 

The NFS issue may still be a problem, but, whether or not it will be an issue for you may be masked by the problem johnnie.black mentioned based on your diagnostics.

On 10/15/2018 at 11:54 PM, johnnie.black said:

The problem appears to be the nerdpack plugin overwriting glibc with an older version.

 

Talking with @eschultz about this he said:

 

"NerdPack (and probably DevPack too) puts packages in /boot/config/plugins/NerdPack/packages/6.6 folder.  NerdPack is using 'upgradepkg --install-new' AFAIK so i'm a bit puzzled how glibc is getting overwritten."

 
Talking with [mention=1875]eschultz[/mention] about this he said:
 
"NerdPack (and probably DevPack too) puts packages in /boot/config/plugins/NerdPack/packages/6.6 folder.  NerdPack is using 'upgradepkg --install-new' AFAIK so i'm a bit puzzled how glibc is getting overwritten."

NerdPack is fairly benign and doesn't include any packages that are present in unRAID. It's the DevPack that's the problem. I put it out there on request as a repository for compiling on a dev unRAID system not for daily use. It uses just installpkg. Going forword I think I'll strip out all the packages inherent to unRAID, include only the headers and add - headers tag to them. Something may not compile right but it won't hose the system. Basically all that is needed is the headers you guys strip out to save space for the preinstalled unRAID packages.