Shares unavailable/non-existent during data rebuild

[karash]

Good Afternoon!

 

Yesterday I had a drive die, I immediately shut the server down and then replaced the drive as soon as I was able to get to the store.

It is currently in the middle of a data rebuild with about 3 hours and 40 minutes left and I am noticing some oddities today.

 

For context: I recently (last Friday) did a huge update of my server going from 6.8.3 to 6.12.10 so these may be new features that are normal but that I didn't know about from skimming release notes. I also didn't notice this yesterday but after the rebuild started I wasn't watching it constantly. 

 

First, and what drew me to notice this: The Mover is saying it is disabled due to a Parity Operation Running. I have never noticed this before, but I haven't had a failed drive in quite a while, but during drive upgrades I always seem to remember Mover running normally?

 

Second, I have some Shares defined but when I go to the Share tab only Movies, TV, and Music are shown the others are not there. And those shares show as being empty and as only having 16.4GB free. The disks all show as disk shares as well. 

This is definitely something new, I have never had the shares not be available for use during a rebuild as far as I can remember.

 

Third, If I click into each disk manually, the shares and their respective files do show. 

 

Is it normal for Shares to disappear or show as empty during a data rebuild in 6.12.10? I have attached the Diagnostics. 

 

The disks:

 

The Rebuild + Mover:

 

No Shares (Except the three mentioned) + Disk Shares:

 

Share Settings (TV and Music are the same):

 

Contents of a Disk:

 

mahbukkit-diagnostics-20240420-1256.zip

[karash]

Update. Rebuild finished, shares still not showing and disks still showing as Disk Shares. 
 

Maybe I can turn off Disk Shares and restart the array.

[karash]

Update 2:

I am home now and seem to have been able to fix it.

 

I attempted to stop the array to turn off disk shares, however it got stuck attempting to unmount the user shares during the array stop so I did a reboot. Once it came back up everything was mounted correctly. 

I was then able to stop the array, turn off disk shares, and then turn the array on. It appears to be working perfectly now. 

 

As for what caused it, I can not be sure but after I got a failed disk error on Friday, in my haste I shut the server down before stopping the array and removing the disk from it. I then plugged in my replacement disk, started the server, stopped the array, and added the replacement disk to the array in the spot of the failed disk.

This may have caused an issue somehow, but again I can not be sure of this. I am replacing the last two 6TB drives on Monday/Tuesday so we will see if any other issues arise.

[karash]

Update 3:

I thought I solved this but it seems to go a bit deeper so I am reopening it.

 

Everything went fine above after rebooting the server, shares came back and everything was happy. Today I received in some 14TBs and decided to replace a 6TB in my system.

I:

1. Stopped the array
2. Unassigned the disk to be replaced
3. Started the array
4. Stopped the array
5. Shut down
6. Replaced the disk physically
7. Started the machine
8. Stopped the array
9. Selected the new disk
10. Started the array

All as outlined here: https://docs.unraid.net/unraid-os/manual/storage-management/#replacing-a-disk-to-increase-capacity

 

As soon as the array came back up, the rebuild started as normal however again, the shares are not available. 3 shares appear just as above with Movies, Music, and TV being in the shares tab but they only show as 16.4GB free with no data contained within. The files are still on the drives as normal. 

The only difference this time is no disk shares as I turned those off completely.

 

I have zero idea why I don't have access to my shares during a data rebuild, I always did while on 6.8.3 and I see no reason why I shouldn't on 6.12.10.

 

Disks:

 

Rebuild Process:

 

Shares which show, all show as empty:

 

The files on the disks:

 

 

At this point the only thing I can think to do is reboot once more, but I am not sure if that is smart during a data rebuild.

 

New Diagnostics file attached.

 

mahbukkit-diagnostics-20240422-2215.zip

 

[karash]

I was able to reboot during the data rebuild and that brought the shares back online.

 

 

[itimpi]

16 minutes ago, karash said:

I was able to reboot during the data rebuild and that brought the shares back online.

 

 

If you did tis before the rebuild completed then the rebuild would be started again from the beginning.

[karash]

7 minutes ago, itimpi said:

If you did tis before the rebuild completed then the rebuild would be started again from the beginning.

 

It did restart the rebuild, I was only 45 minutes into it. But it also remounted the shares so I have access during the full rebuild.

It seems to be a FUSE issue based on what we saw, I just have to add a second reboot into my workflow when replacing a drive.

 

[JorgeB]

There's something creating this folder before array start:

 

Apr 22 22:08:12 MahBukkit emhttpd: error: malloc_share_locations, 7199: Operation not supported (95): getxattr: /mnt/user/Music

 

That will cause the problems you are seeing.

[karash]

On 4/23/2024 at 12:47 AM, JorgeB said:

There's something creating this folder before array start:

 

Apr 22 22:08:12 MahBukkit emhttpd: error: malloc_share_locations, 7199: Operation not supported (95): getxattr: /mnt/user/Music

 

That will cause the problems you are seeing.

 

I am not sure what could be doing that honestly. I have another computer setup with auto.fs mounting my shares but I turned off the service before shutting down Unraid to replace the drive and it doesn't mount Music anyway.

I have no idea what is causing it, I successfully completed my second disk replacement and did a third today with the same thing happening. A reboot after adding the new drive to the array fixed the issue immediately.

 

Obviously now that I know this works, my steps to replace a drive just involve an extra reboot. Solving the issue entirely would be nice but I am at a loss.