2 Disks showing as disabled


Recommended Posts

Hi. Yesterday I was preclearing a new 6tb drive in an external esata enclosure. Sometime during the process my enclosure crapped its pants and made a few disks show as offline, including the new 6tb, as well as two other existing 6tb's. After moving the new 6tb drive to an internal bay, as well as another 6tb drive which was in the array already, all the other drives are operating normally, except for the two 6tb's that were in the array already. They're showing as disabled. I ran a xfs_repair on them without destroying the log. The results for one of the disks is below. I'm stuck on what to do next, as I don't want to format the emulated drives and lose my data


Phase 1 - find and verify superblock...
        - block cache size set to 12335824 entries
Phase 2 - using internal log
        - zero log...
zero_log: head block 2294 tail block 2294
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 1
        - agno = 5
        - agno = 3
        - agno = 4
Phase 5 - rebuild AG headers and trees...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...

        XFS_REPAIR Summary    Wed Aug 14 22:19:47 2019

Phase		Start		End		Duration
Phase 1:	08/14 22:19:43	08/14 22:19:44	1 second
Phase 2:	08/14 22:19:44	08/14 22:19:44
Phase 3:	08/14 22:19:44	08/14 22:19:47	3 seconds
Phase 4:	08/14 22:19:47	08/14 22:19:47
Phase 5:	08/14 22:19:47	08/14 22:19:47
Phase 6:	08/14 22:19:47	08/14 22:19:47
Phase 7:	08/14 22:19:47	08/14 22:19:47

Total run time: 4 seconds
done

 

 

diagnostics-20190815-0234.zip

Edited by IDontBelongHere
Uploading diagnostics.zip
Link to comment

After stopping the array, unassigning the disks and re-assigning (did not boot array in between), the filesystems on the emulated disks are now showing as mountable. The disks themselves are still disabled. I ran smart tests on each and both are fine without any errors. 

 

What's the best way to get unRaid to re-enable my old disks back without having to rebuild data to each 6tb drive?

Link to comment

Please don't do anything else without further advice. You are probably already trying to go down the wrong path.

 

Disabled disks need to be rebuilt from parity. Unmountable disks need to be repaired.

 

You definitely have 2 disabled disks, but it is unclear that any are unmountable.

 

Please start the array in normal mode and post a new diagnostics so we can see if the emulated disks will mount.

Link to comment

You posted again while I was posting. Since the emulated disks are mountable then rebuilding should reenable them. In fact, if you 

4 minutes ago, IDontBelongHere said:

unassigning the disks and re-assigning

then start the array they may already be rebuilding.

 

Let us know what is happening now and post a new diagnostics and screenshots of Main - Array Devices and Main - Array Operations.

Link to comment
Quote

You posted again while I was posting. Since the emulated disks are mountable then rebuilding should reenable them. In fact, if you 

  6 minutes ago, IDontBelongHere said:

unassigning the disks and re-assigning

then start the array they may already be rebuilding.

Should I stop the array, unassign disks, start array, stop, and re-assign? Or was unassinging and re-assigning w/o starting the array the correct procedure?

Link to comment

I'm not sure how you got to where you are. Maybe the details are in those diagnostics but I haven't studied them. If you

3 minutes ago, IDontBelongHere said:

stop the array, unassign disks, start array, stop, and re-assign

then start the array, it will begin to rebuild them. But I thought you said you had already done that. You must have left out a step before.

 

Link to comment
Just now, trurl said:

I'm not sure how you got to where you are. Maybe the details are in those diagnostics but I haven't studied them. If you

then start the array, it will begin to rebuild them. But I thought you said you had already done that. You must have left out a step before.

 

My sincerest apologies if I didn't do a good job at explaining everything. I probably shouldn't be up posting on here at this time. 

 

Originally, my array looked like this (this is from 11:26 AM 8/14/19, about 12 hours ago): image.thumb.png.815892f235c39cd5ee64b0a6036c62ce.png

 

After moving two 6tb drives inside my server from my external enclosure, and starting the array in maint. mode, all the drives with missing temps went back to normal, except the two actually disabled drives. At the time, those two emulated drives said they needed to be formatted as the file systems couldn't be mounted. After running the xfs_repair option on them removing the -n flag, it gave me the output I quoted originally but still wouldn't enable the drive, and the emulated drive still needed to be formatted. 

I never removed any of the drives from the array at this point yet. I rebooted to see if that would re-enable the disks to no avail. 

 

I then started the array in maint. mode again, stopped it, unassigned disks 15 and 16, (no starting of the array in between) re-assigned them, then restarted the array in maint. mode. At that point I got the notification that both disks had returned to normal operation. I assume it meant the emulated disks, as I stopped the array, and the disks were still showing as disabled, but when I started the array in normal mode, the emulated disks were mounted with the physical disks still disabled. 

 

I hope this clears any confusion up. 

Link to comment

I can't tell from that last screenshot whether the emulated disks were mounted or not because it is clipped on the right where it would show.

 

The other screenshot in your earlier post seems to show the emulated disks are mounted, and so you say. If that is the case then it should be OK to rebuild them. If you browse the emulate disks you should be able to see their contents.

 

I am concerned though about how all this happened to begin with. In order to rebuild the disks reliably, Unraid must be able to reliably read all of the other disks. It doesn't sound like your hardware configuration is reliable.

Link to comment
1 minute ago, trurl said:

I can't tell from that last screenshot whether the emulated disks were mounted or not because it is clipped on the right where it would show.

 

The other screenshot in your earlier post seems to show the emulated disks are mounted, and so you say. If that is the case then it should be OK to rebuild them. If you browse the emulate disks you should be able to see their contents.

 

I am concerned though about how all this happened to begin with. In order to rebuild the disks reliably, Unraid must be able to reliably read all of the other disks. It doesn't sound like your hardware configuration is reliable.

They were showing as "File system unmountable: Format disk" or something along those lines in my first screenshot. 

 

I'm pretty sure this is all due to my two crappy 4-bay MediaSonic enclosure being crappy. Added a new 6tb drive to one to pre-clear it, and I don't think the supplied power brick is powerful enough for all for 4 drive bays to be used reliably. 


 

Quote

Personally, I would have used the new larger disks to replace some of the smaller disks instead of adding them. Fewer disks means fewer opportunities for problems, and your large number of small disks are certainly giving you problems.

 

Just got a new 6tb Tuesday. The new 6tb drive is going to replace some if not all of the 1tb drives. 

Link to comment
6 minutes ago, IDontBelongHere said:

4-bay MediaSonic enclosure

I assume those enclosures only have one eSATA port. If that is the case, then that is going to be another problem, since it will have to push all the data for all 4 drives through that one port, which will cut your rebuild speed by a lot. If each disk has its own connection then they can be accessed simultaneously but if they are sharing a port then I can imagine a 6TB rebuild taking days instead of hours, if it even works.

Link to comment
Just now, trurl said:

I assume those enclosures only have one eSATA port. If that is the case, then that is going to be another problem, since it will have to push all the data for all 4 drives through that one port, which will cut your rebuild speed by a lot. If each disk has its own connection then they can be accessed simultaneously but if they are sharing a port then I can imagine a 6TB rebuild taking days instead of hours.

Yup. Yet another reason why I'm ditching the enclosures after about 1.5 years and consolidating disks. 

I only have 2 or 3 drives per enclosure at this point, most if not all of them 1tb drives.  (Sorry. Hard for me to remember the specific layout of my disks as my server is being hosted over 600 miles away at a friend's house while I'm working a long term assignment out of state. He's been my remote data center admin for the last few months.)

 

A 6tb rebuild usually takes me about 18-24 hours if I remember correctly. 

Link to comment

It is worth pointing out that unassigning disks and then re-assigning without starting the array is effectively a null operation as it is only at the point of starting the array that any changes are committed.    Also, do NOT attempt to format a drive or it’s contents will be lost.    Running xfs_repair on the emulated drives was the correct thing to do as it corrects any errors in the emulated disks and from what you said that appeared to work as expected and they now mount.

 

if you want to rebuild the drives then as you surmised the correct procedure is stop array; unassign drives; start array; stop array; re-assign drives; start array to begin the rebuild.   The rebuild will put onto the disks exactly what is shown by Unraid when it is emulating the disks before doing the rebuild.

Link to comment

Problem with the enclosure/port multiplier dropped all disks connected:

 

Aug 15 04:50:25 DragonsNest kernel: ata8.15: Port Multiplier detaching
Aug 15 04:50:25 DragonsNest kernel: ata8.00: disabled
Aug 15 04:50:25 DragonsNest kernel: ata8.01: disabled
Aug 15 04:50:25 DragonsNest kernel: ata8.02: disabled
Aug 15 04:50:25 DragonsNest kernel: ata8.03: disabled
Aug 15 04:50:25 DragonsNest kernel: ata8.00: disabled

 

You should really avoid SATA port multipliers with Unraid.

Link to comment
20 minutes ago, IDontBelongHere said:

I'm now getting tons of read errors on multiple drives. 

Of course that is going to compromise the rebuilds. Probably we should have suggested rebuilding to new disks so you didn't mess with the originals.

 

I guess you can check connections of those disks and try again, but

8 hours ago, trurl said:

In order to rebuild the disks reliably, Unraid must be able to reliably read all of the other disks. It doesn't sound like your hardware configuration is reliable.

8 hours ago, trurl said:

Fewer disks means fewer opportunities for problems, and your large number of small disks is certainly giving you problems.

 

Do you have backups of anything important and irreplaceable? If not that should probably take priority over even the rebuilding. You must have another copy of anything important and irreplaceable on another system.

Link to comment
34 minutes ago, trurl said:

Of course that is going to compromise the rebuilds. Probably we should have suggested rebuilding to new disks so you didn't mess with the originals.

 

I guess you can check connections of those disks and try again, but

 

Do you have backups of anything important and irreplaceable? If not that should probably take priority over even the rebuilding. You must have another copy of anything important and irreplaceable on another system.

Yes. I have off-site backups of my most important stuff. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.