10TB Disk has read errors. "Device is disabled. Contents Emulated"

[CorserMoon]

Hi all. I've read several threads here on this subject but the general consensus seems to be make a thread and post diagnostics so the experts can take a look. Thanks in advance. 

 

I have a 10TB drive that is now disabled due to read errors. What exactly does "Device is disabled. Contents Emulated" mean? Am I still able to access the data that is/was on the disk?

 

Regardless, I am assuming I will have to RMA that disk, but being relatively new to unraid, I want to make sure I do everything properly to prevent dataloss, corruption of the parity, etc. I don't know the proper procedure to go from here. I have plenty of extra space on the array so I can pull this disk and shrink the array if need be. 

 

Attached are the syslogs. 

executor-diagnostics-20200520-1427.zip

[BRiT]

IIRC, unRaid only Disables Device on Write Errors.

[BRiT]

It looks like you have multiple issues.

 

Your Disk3 has WRITE errors.

 

May 16 22:10:33 Executor kernel: md: disk3 read error, sector=55075048
May 16 22:10:34 Executor kernel: md: disk3 write error, sector=55075048

Mar 14 11:34:27 Executor kernel: md: import disk3: (sde) WDC_WD100EMAZ-00WJTA0_2YJ0ZABD size: 9766436812 

 

You also have a corrupt NTFS filesystem on one of your Unassigned Devices.

Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6427 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5224): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6438 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5905): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6438 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5905): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6419 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5069): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6567 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6034): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6567 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6034): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6465 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5396): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6568 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6033): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6568 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6033): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6443 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5340): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6487 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (1040): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6487 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (1040): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6506 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (1036): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6411 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5055): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6411 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5055): Input/output error

 

[CorserMoon]

1 minute ago, BRiT said:

It looks like you have multiple issues.

 

Your Disk3 has WRITE errors.

 

May 16 22:10:33 Executor kernel: md: disk3 read error, sector=55075048
May 16 22:10:34 Executor kernel: md: disk3 write error, sector=55075048

Mar 14 11:34:27 Executor kernel: md: import disk3: (sde) WDC_WD100EMAZ-00WJTA0_2YJ0ZABD size: 9766436812 

 

You also have a corrupt NTFS filesystem on one of your Unassigned Devices.

Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6427 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5224): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6438 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5905): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6438 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5905): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6419 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5069): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6567 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6034): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6567 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6034): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6465 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5396): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6568 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6033): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6568 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (6033): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6443 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5340): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6487 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (1040): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6487 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (1040): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6506 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (1036): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6411 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5055): Input/output error
Apr  8 18:13:47 Executor ntfs-3g[27436]: Trying to read non-allocated mft records (6411 > 6400): Illegal seek
Apr  8 18:13:47 Executor ntfs-3g[27436]: Inode is corrupt (5055): Input/output error

 

Thanks for helping diagnose. I only said READ because of what I saw in the notifications:  

 

I only have one unassigned device (aside from this 10TB drive that is disabled and now showing in Unassigned devices). The unassigned device is a 4TB WD Purple drive that I use only for surveillance camera footage. It's attached only to a Windows 10 VM that is running Blue Iris. Blue Iris writes to that drive. So now I have 2 questions:

 

1. What is the proper procedure for dealing with the 10TB with read errors?

2. How should I deal with/repair the Unassigned device with corrupt file system? 

[itimpi]

15 minutes ago, CorserMoon said:

have a 10TB drive that is now disabled due to read errors. What exactly does "Device is disabled. Contents Emulated" mean? Am I still able to access the data that is/was on the disk?

This means that Unraid is constructing what it thinks should be on that drive by the combination of parity plus all the other data drives.   You should see if you can browse the ‘emulated’ drive in the GUI and see its contents?   If not then let us know so we can give advice on best way to proceed.   If the emulated contents look OK then you have the choice of rebuilding the emulated disk contents either back to the same disk or to a replacement.   If you cannot see the emulated contents then the next steps will be slightly different.

 

The SMART information for that drive looks OK.     The most frequent cause of a disk being disabled is due to the SATA/power cabling having issues rather than the disk itself being faulty,  so you want to check that out carefully.

 

[CorserMoon]

7 minutes ago, itimpi said:

This means that Unraid is constructing what it thinks should be on that drive by the combination of parity plus all the other data drives.   You should see if you can browse the ‘emulated’ drive in the GUI and see its contents?   If not then let us know so we can give advice on best way to proceed.   If the emulated contents look OK then you have the choice of rebuilding the emulated disk contents either back to the same disk or to a replacement.   If you cannot see the emulated contents then the next steps will be slightly different.

 

The SMART information for that drive looks OK.     The most frequent cause of a disk being disabled is due to the SATA/power cabling having issues rather than the disk itself being faulty,  so you want to check that out carefully.

 

Thanks @itimpi. It seems that I am able to browse the data via the web gui without issues:

 

Regarding SATA cables, I'm running unraid on a supermicro 846 chassis with SAS backplane. I can try replacing the Mini-SAS cables if this issue persists. 

[itimpi]

That is good as the rebuild process just puts what you can see on the ‘emulated’ disk back onto a physical drive.    The process for doing this is documented here in the online documentation.

 

if you have a spare disk it is always safest to rebuild to the spare and keep the original intact in case anything goes wrong during the rebuild process as then you have the original intact for data recovery purposes.   This is not always practical if the drive appears to not actually be faulty, so often as long as the SMART information looks OK rebuilding the disk back to itself is often an eminently reasonable thing to do.

[CorserMoon]

5 minutes ago, itimpi said:

That is good as the rebuild process just puts what you can see on the ‘emulated’ disk back onto a physical drive.    The process for doing this is documented here in the online documentation.

 

if you have a spare disk it is always safest to rebuild to the spare and keep the original intact in case anything goes wrong during the rebuild process as then you have the original intact for data recovery purposes.   This is not always practical if the drive appears to not actually be faulty, so often as long as the SMART information looks OK rebuilding the disk back to itself is often an eminently reasonable thing to do.

Thanks @itimpi. I was also reading THIS about shrinking the array (removing a disk) and maintaining data on the removed disk (via the "Remove Drives Then Rebuild Parity" method). I don't have a spare disk on hand but I do have space on the array to remove that disk. I am wondering if I should do this method, rebuild the array, then if there are problems I still have the original data disk. If everything rebuilds fine, then I can re-insert that disk, pre-clear it, and add back into the array fresh. Thoughts on the best method?

[CorserMoon]

Alright, given that the SMART data seems ok and I pre-cleared the device when I added it, and the drive is not that old, I am going to re-add it to the array using this method outlined in the unraid wiki article @itimpi linked to:

 

Quote

There can be cases where it is determined that the reason a disk was disabled is due to an external factor and the disk drive appears to be fine. In such a case you need to take a slightly modified process to cause Unraid to rebuild a 'disabled' drive back onto the same drive.

1. Stop array

2. Unassign disabled disk

3. Start array so the missing disk is registered

4. Stop array

5. Reassign disabled disk

6. Start array to begin rebuild. If you start the array in Maintenance mode you will need to press the Sync button to start the rebuild.

Presumably this will take a long time since it is rebuilding the disk from parity?

[itimpi]

4 hours ago, CorserMoon said:

Presumably this will take a long time since it is rebuilding the disk from parity?

The rebuild process has to rewrite every sector on the target disk, so you should expect it to take a similar amount of time to a parity check.    In addition, any other activity on the array will slow the process down just as it would slow a parity check.

[CorserMoon]

Thanks all for the help. Rebuilding now.