Pulled disk: no reaction


Ancan

Recommended Posts

Hi all,
I'm trying out Unraid as a platform for my new NAS, and pulled a disk to see the impact on the array.

 

I'm bit surprised actually, because I pulled the disk 30 minutes ago, and so far nothing has happened except a popup about twenty or so read errors. You would think completely removing a disk from an array would cause some distress, but it's still green in the interface and I can browse the structure (but not download). What's the expected behavious here really? Because I thought I'd get a big red warning about a dead disk.

 

Link to comment
8 minutes ago, Ancan said:

Hi all,
I'm trying out Unraid as a platform for my new NAS, and pulled a disk to see the impact on the array.

 

I'm bit surprised actually, because I pulled the disk 30 minutes ago, and so far nothing has happened except a popup about twenty or so read errors. You would think completely removing a disk from an array would cause some distress, but it's still green in the interface and I can browse the structure (but not download). What's the expected behavious here really? Because I thought I'd get a big red warning about a dead disk.

 

I haven't tested myself, however when removing a disk without shutting array down etc. It should detect this and then emulate the storage using parity data.

 

I would expect it to also show a notification stating something along the lines of "disk not detected"

Edited by Conmyster
Link to comment
14 minutes ago, Conmyster said:

I haven't tested myself, however when removing a disk without shutting array down etc. It should detect this and then emulate the storage using parity data.

Nothing yet. Except a "*" on the disk temperature and up to 32 errors now. Still green otherwise. By now I ideally should have had a mail in my inbox and be on my way to the store for a replacement. Hmm...

 

bild.thumb.png.3be8fa018fec9da7cd7d2f37e313744e.png

 

Edit: I *love* how you can just paste images from the clipboard into the forum here (Ok, I somehow get double attachments when I do it but it 10 x beats saving the image and uploading it)

Edited by Ancan
PS
Link to comment
27 minutes ago, Ancan said:

Nothing yet. Except a "*" on the disk temperature and up to 32 errors now. Still green otherwise. By now I ideally should have had a mail in my inbox and be on my way to the store for a replacement. Hmm...

 

bild.thumb.png.3be8fa018fec9da7cd7d2f37e313744e.png

 

Edit: I *love* how you can just paste images from the clipboard into the forum here (Ok, I somehow get double attachments when I do it but it 10 x beats saving the image and uploading it)

If you setup the notification system under Settings > Notification Settings

 

Then it should of notified you of the errors on the drive... I would need to test myself though

 

Edit: After pulling my disk 5, I got a notification within 5 seconds (Discord) This is what my main screen looks like too:

image.thumb.png.a7c9584ec526a62c4ffda664969ab323.png

 

After this I stopped the array, unassigned the device, started the array. Then stopped the array and reassigned the device and started the array.

 

Data is now rebuilding fine.

Edited by Conmyster
Link to comment
24 minutes ago, Conmyster said:

If you setup the notification system under Settings > Notification Settings

 

Then it should of notified you of the errors on the drive... I would need to test myself though

 

Edit: After pulling my disk 5, I got a notification within 5 seconds (Discord) This is what my main screen looks like too:

image.thumb.png.a7c9584ec526a62c4ffda664969ab323.png

 

After this I stopped the array, unassigned the device, started the array. Then stopped the array and reassigned the device and started the array.

 

Data is now rebuilding fine.

 

Kudos on commitment for pulling a drive!

I've got notifications set to "browser" only while I'm testing, and haven't seen anything except the "errors" popup. Disk still green in the GUI, even though it's sitting here on my desk.

I should mention that my paritity is not fully synchronized yet, and was not when I pulled the disk. I'm being cruel I know. I validate enterprise storage installations as part of my job, and am a bit damaged by that probably.

 

Edited by Ancan
wording
Link to comment
17 minutes ago, Ancan said:

 

Kudos on commitment for pulling a drive!

I've got notifications set to "browser" only while I'm testing, and haven't seen anything except the "errors" popup. Disk still green in the GUI, even though it's sitting here on my desk.

I should mention that my paritity is not fully synchronized yet, and was not when I pulled the disk. I'm being cruel I know. I validate enterprise storage installations as part of my job, and am a bit damaged by that probably.

 

Ah okay, I'm unsure on how it would deal with loosing a drive during a parity sync. As I'm aware most single parity hardware raid (say raid 5) would just die. 

 

And I don't have a system to test that with myself.

Link to comment

Parity is now green as well. Just as green as Toshiba disk on my desk here beside me. Perhaps I should post on the bug-forum instead?

 

image.thumb.png.542742cecdf56af20e00414c7bf67902.png

 

I'll stop the array now, and replace the Toshiba with a new disk instead. Let's hope Unraid will let go of it then.

 

 

Edit: About time!!!

 

image.png.3505137ecfeb0014cbb8922cc87060c7.png

 

Edited by Ancan
Link to comment
8 minutes ago, Ancan said:

Parity is now green as well. Just as green as Toshiba disk on my desk here beside me. Perhaps I should post on the bug-forum instead?

 

image.thumb.png.542742cecdf56af20e00414c7bf67902.png

 

I'll stop the array now, and replace the Toshiba with a new disk instead. Let's hope Unraid will let go of it then.

 

 

Edit: About time!!!

 

image.png.3505137ecfeb0014cbb8922cc87060c7.png

 

Ah so likely due to the parity sync running.

 

Although I don't think that should be the correct course of action by unraid....

Link to comment

While this behaviour is a little unsettling, and possibly should be modified, I can see why it probably happened. Parity is built walking the disk sectors in order. Since the disk you pulled was only a 500GB, and you were building to 6TB of parity, the disk was already not being read because the parity build process was beyond the 500GB mark. Its contents (and all the other drives to that point) were already written to parity and being emulated by the rest of the drives.

 

I suspect if you had read file contents or written to that drive, it would have immediately failed and warned you, but since all you did was browse the TOC that was most likely in RAM, no activity was actually asked of the disk, so unraid didn't know it was missing.

 

Since one of the main selling points of unraid is the ability to keep some drives spun down if they are not needed, I think some effort has been put into not actively poking drives just to be sure they are there.

 

Unraid only fails a drive when a write to it errors out, so it's quite conceivable that a failed drive could hang out in the array for some time without being noticed. Regularly scheduled parity checks (typically monthly) are a way to be sure little used drives are still capable of participating in a rebuild should they be called into action.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.