[SOLVED] Disabled Disk & Separate Unmountable

[AustinTylerDean]

Hello All,

 

(edited: Diagnostics have been posted below if anyone can take a crack at this for me

 

I have two problems as listed in the title. For some reason I can't get the public image uploaders to work for my screenshot to show you. So I'll just say Disk 2 is disabled, and disk 3 us the unmountable: no file system. I don't really know when this all started happening. I am not a Linux guy. Just getting this box to work with Sonarr/NZBGet/Plex and all that comes with operating that via the helpful guides of SPACEINVADERONE was easily over 40 hours of work for me.

 

What are my first steps into correcting this problem? I stopped the array, started it in maintenance mode, and have clicked on extended SMART test for the disk 3 so far. However it's comfortably sitting at 10% and don't see this being done tonight or maybe even tomorrow. I've searched a bit about these issues, but they seem very specific for the problems that each person is having. I'll be happy to do all the legwork that is asked of me if I can get some help from here.

 

I really appreciate anyone's time in assisting me, and look forward to learning some new things from these problems.

 

Thanks again,

 

Austin

 

Edited January 7, 2021 by AustinTylerDean
Posted Diagnostics

[ChatNoir]

Hello,

First step would be to share your diagnostics in your next post one the Smart test is finished. (Tools / Diagnostics).

[AustinTylerDean]

ChatNoir, I have attached the diagnostics file. To note, disk 3 did finish its SMART test with no problems. Disk 2 is currently sitting at 90% complete, and I will post the results of that as soon as it happens. I appreciate you replying.

tower-diagnostics-20210107-1526.zip

[AustinTylerDean]

Also adding the two SMART reports

tower-smart-20210107-1553.zip tower-smart-20210107-1523.zip

[AustinTylerDean]

I've run the -nv command of xfs_repair and received the following on disk 3. I can't tell if it fixed anything. When I stop the array maintenance mode, the disk looks ready to be included in the array on startup, however since I have disk 2 disabled, I am worried that I will mess something up by trying to put the array online without disk 2 fixed first.

 

Phase 1 - find and verify superblock... - block cache size set to 734312 entries

Phase 2 - using internal log - zero log... zero_log: head block 521875 tail block 521875 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes...

Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 3 - agno = 1 - agno = 4 - agno = 5

No modify flag set, skipping phase 5

Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - traversal finished ... - moving disconnected inodes to lost+found ...

Phase 7 - verify link counts...

No modify flag set, skipping filesystem flush and exiting.

XFS_REPAIR Summary Thu Jan 7 16:37:55 2021 Phase Start End Duration

Phase 1: 01/07 16:37:51 01/07 16:37:51

Phase 2: 01/07 16:37:51 01/07 16:37:51

Phase 3: 01/07 16:37:51 01/07 16:37:54 3 seconds

Phase 4: 01/07 16:37:54 01/07 16:37:54

Phase 5: Skipped

Phase 6: 01/07 16:37:54 01/07 16:37:55 1 second

Phase 7: 01/07 16:37:55 01/07 16:37:55

Total run time: 4 seconds

[trurl]

44 minutes ago, AustinTylerDean said:

-nv command of xfs_repair and received the following on disk 3. I can't tell if it fixed anything.

-n is the nomodify flag so it made no changes

45 minutes ago, AustinTylerDean said:

No modify flag set, skipping phase 5

Did you run this in the webUI or from the command line?

[AustinTylerDean]

I ran it both with -v /dev/md3 in terminal, and in the GUI as just -v. I didn't see any difference.

[trurl]

47 minutes ago, AustinTylerDean said:

since I have disk 2 disabled, I am worried that I will mess something up by trying to put the array online without disk 2 fixed first.

The only way to fix the disabled disk is to rebuild it. It won't hurt anything to start the array with it disabled. It won't even use the disabled disk. It is emulated from the parity calculation.

 

We can work on the rebuild before or after you fix the unmountable disk. Might be better to go ahead and get the array protected again first by rebuilding disk2.

 

SMART for those disks look OK.

 

Looks like disk3 was already unmountable before you booted up, but it looks like it may be having connection problems. I assume disk2 was already disabled before you booted up too since I don't see any errors on it. If it weren't already disabled possibly it would show connections problems too.

 

Shut down, check all connections, power and SATA, including splitters. Then reboot, post new diagnostics and a screenshot.

[trurl]

1 minute ago, AustinTylerDean said:

I ran it both with -v /dev/md3 in terminal, and in the GUI as just -v. I didn't see any difference.

Did you do that after I mentioned the nomodify flag? What was the output from running it without -n?

[AustinTylerDean]

13 minutes ago, trurl said:

Did you do that after I mentioned the nomodify flag? What was the output from running it without -n?

 

[trurl]

Did anything get put in lost+found? (you would have a new user share by that name).

[AustinTylerDean]

6 minutes ago, trurl said:

Did anything get put in lost+found? (you would have a new user share by that name).

I don't see any shares present. Would this only show up if I had the array started?

[trurl]

Yes you have to start the array in normal mode to see any of your shares or files. And hopefully that repaired disk3 will be mountable now.

[AustinTylerDean]

11 minutes ago, trurl said:

Yes you have to start the array in normal mode to see any of your shares or files. And hopefully that repaired disk3 will be mountable now.

I started the array! It doesn't look like I have any shares that were added. Disk 3 looks normal. Disk 2 is obviously still disabled. AND I also see that I have a SMART error on my parity disk now. Both screenshots attached. What is my next step? Getting after the disk 2?

 

I feel like I am having a bunch of these UDMA CRC error counts happen every once in awhile.

 

[AustinTylerDean]

The short SMART report is attached for the parity drive. By the way, it says my parity is bad now. EDITED : I swear I just saw something say my parity was invalid, but now I don't see mention of it anywhere. Sorry for my terrible Unraid skills.

tower-smart-20210107-1909.zip

Edited January 8, 2021 by AustinTylerDean

[trurl]

1 minute ago, AustinTylerDean said:

it says my parity is bad now.

Define "it"

 

And define "bad"

[trurl]

5 minutes ago, AustinTylerDean said:

UDMA CRC error

These are connection problems counted by the disk and not a problem with the disk itself. Many connection problems the disk won't even count since it can't know about them when it is disconnected.

 

You can acknowledge the current count by clicking on the SMART warning on the Dashboard page and it will warn you again if it increases. That is a lot of them though.

 

[AustinTylerDean]

3 minutes ago, trurl said:

Define "it"

 

And define "bad"

I apologize for the lackluster details on my part. I distinctly remember seeing "Parity is invalid" under Array Operations of the Main page, however now I don't see this information anymore. I suppose I can now move to the disk 2 problem?

[trurl]

1 hour ago, trurl said:

Shut down, check all connections, power and SATA, including splitters. Then reboot, post new diagnostics and a screenshot.

 

[AustinTylerDean]

Alright! So before this last check and reboot, the message that stated the parity was invalid was the floating status box which came up in red. NOW I get a much better status box that states the "array turned good". This is much pleasant. My fingers are crossed that we are good to this point! Trurl, I can't thank you enough for your time and expertise!

tower-diagnostics-20210107-1945.zip

[trurl]

Did you check the connections?

 

You can rebuild disk2 to the same disk, but if you have another disk you can use to replace it you could keep the original disk2 in reserve in case there are any problems rebuilding. In any case rebuilding disk2 won't change parity or the other disks.

 

Do you have backups of everything important and irreplaceable?

[AustinTylerDean]

I definitely checked the connections. I'm using a server grade disk controller with 2x4 port fanout cables. I saw that maybe the cable near the connection to the parity drive was a little banged up so I switched it to a fresh unused one. That may be the ticket for the parity UDMA errors. I can't say anything is irreplaceable as it's all movie and TV related. And I don't have another 4TB disk laying around... Do you think there's a sizeable chance that I will lose data? I was hoping this chance was low considering the selling point of Unraid in the first place, however I'm the idiot here.

[trurl]

All disks are mounted, including the emulated disk2. The emulated data is exactly what will be rebuilt so it looks pretty good.

 

To rebuild to the same disk

1. Stop array
2. Unassign disk
3. Start array with disk unassigned
4. Stop array
5. Reassign disk
6. Start array to begin rebuild

During rebuild, you should see lots of Writes to the rebuilding disk, lots of Reads of all other disks, and zeros in the Errors for all disks.

 

I would estimate 4TB to take 8-12 hours. Using the array during that time won't break anything, but it will make everything go slower. 

[AustinTylerDean]

Thank you very much for your help, again! I'll be sure to post results tomorrow! I'm cautiously optimistic, trurl, thank you!

[AustinTylerDean]

Trurl, we seem to be completely out of the woods! Let me know if I can help you in any way! Patreon account or whathaveyou!!

 

You the best!