[6.10.3] UNRAID reported read errors on existing Drive 4 - Replaced it and now I have read errors on many more drives?

[taflix]

Disk 4 started rebuilding after I assigned it to the array.

 

However, UNRAID reported read errors from 2 additional drives, please help!

 

Diagnostics attached.

taflix-unraid-diagnostics-20220803-1841.zip

 

Short self tests on drives 2 and 5 show:

 

Quote

Completed without error.

 

Edited August 4, 2022 by taflix

[trurl]

1 hour ago, taflix said:

Short self tests on drives 2 and 5 show:

Did you do these tests after you took those diagnostics?

 

In diagnostics,

Disk5 shows

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.

 

No SMART report for disk2 because it has disconnected.

 

Did you do anything to fix that connection so you could run self-test on disk2?

 

Post new diagnostics

 

 

[trurl]

1 hour ago, taflix said:

Disk 4 started rebuilding after I assigned it to the array

Also, could you tell us more about what led up to that?

[taflix]

23 minutes ago, trurl said:

Also, could you tell us more about what led up to that?

 

Yes.  This issue popped up again:  

 

 

I tried the same solution:

1. Stop array
2. Unassign drive
3. Start array
4. Stop array
5. Assign drive
6. Start array

Except this time, it would get read errors and stop rebuilding.

 

[taflix]

26 minutes ago, trurl said:

Did you do these tests after you took those diagnostics?

 

In diagnostics,

Disk5 shows

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.

 

No SMART report for disk2 because it has disconnected.

 

Did you do anything to fix that connection so you could run self-test on disk2?

 

Post new diagnostics

 

 

 

I sort of just started doing everything all at once.

 

I've since turned off the machine.  Reseated the cables.  That fixed the issue with Disk 2.  However Disk 5 is still showing tons of errors:

 

 

What should I do?  It's currently rebuilding.

[trurl]

post new diagnostics

[taflix]

1 hour ago, trurl said:

post new diagnostics

 

Thank you, done.

taflix-unraid-diagnostics-20220803-2223.zip

Edited August 4, 2022 by taflix

[JorgeB]

It's not logged as a disk problem and disk5 looks healthy, cancel the rebuild, replace cables/change slot on disk5 and try again.

[taflix]

On 8/4/2022 at 2:19 AM, JorgeB said:

It's not logged as a disk problem and disk5 looks healthy, cancel the rebuild, replace cables/change slot on disk5 and try again.

 

Hello, I replaced 8 cables in my system and now I'm getting read errors from a different drive?

 

Diags attached

taflix-unraid-diagnostics-20220807-1205.zip

[trurl]

Check connections on disk2, both ends, SATA and power, including splitters.

[taflix]

51 minutes ago, trurl said:

Check connections on disk2, both ends, SATA and power, including splitters.

 

Disk 4 is a new disk.  Does it have to be precleared before adding to the array?

 

Does it have to be formatted?

 

I'm getting this message while Disk 4 is being rebuilt? 

 

"Unmountable: Wrong or no file system"

 

fresh daigs attached

taflix-unraid-diagnostics-20220807-1424.zip

 

I unplugged and reseated all data and power cables.  Looked like it was running well for about 30 seconds but Disk 2 is still getting errors?

 

Should I allow the rebuild process to continue or stop?  I'll pause it for now.

Edited August 7, 2022 by taflix

[trurl]

1 hour ago, taflix said:

Disk 4 is a new disk.  Does it have to be precleared before adding to the array?

 

Does it have to be formatted?

Let's clear up some verbage. You replaced a disk, you didn't add a disk.

 

The difference is always relevant, but even more so here with your other questions.

 

There is only one scenario where Unraid requires a clear disk. That is when adding it to a new slot in an array that already has valid parity. This is so parity will remain valid. A clear disk is all zeros, and so has no effect on parity. If you add a disk to a new slot, and it hasn't been precleared, Unraid will clear it before it will let you format it in the array..

 

If you were adding a precleared disk, then formatting the disk before adding it would make it no longer clear, so Unraid would have to clear it again before it would allow you to format it in the array.

 

It must be formatted in the array because format writes an empty filesystem to the disk, and parity needs to update on all write operations.

 

But, you aren't adding a disk, so I will continue in the next post.

[trurl]

42 minutes ago, taflix said:

Does it have to be precleared

 

Does it have to be formatted?

When replacing/rebuilding a disk, it doesn't matter at all what was on the replacement. It can be clear, it can be newly formatted (contains an empty filesystem), it can be completely full of your baby pictures. Doesn't matter, the entire disk is going to be completely overwritten by the rebuild.

[trurl]

45 minutes ago, taflix said:

"Unmountable: Wrong or no file system"

Usually we prefer to repair the filesystem before rebuilding, since the emulated disk is exactly what rebuild will result in, which in this case is an unmountable filesystem. But too late now. You will have to repair the filesystem after rebuild completes.

 

Haven't looked at diagnostics yet so don't know if you still have other problems. More about that next post.

[trurl]

50 minutes ago, taflix said:

Disk 4 is a new disk

Don't remember, and apparently some of this was on another thread, so I will ask.

 

Was there anything actually wrong with the original disk? How do you know? Do you still have the original?

 

The reason I ask is because you are currently rebuilding a corrupt filesystem, which will need to be repaired. The contents of the original might also be useful depending on how well repair works.

[trurl]

Looks like you did format the replacement as an Unassigned Device, which was pointless, but no harm. Where people often get in trouble is formatting a disk in the array then hoping to rebuild it from parity. When you format a disk in the array, parity will agree that is has been formatted so then you can only rebuild a formatted disk.

 

And the situation you are currently in (unmountable) also leads some people to make that critical mistake of formatting a disk in the array. Unraid will give a list of unmountable disks with an option to format, but you must check a box to do that and it gives you a big warning about it. People do it anyway.

 

In your case, disk4 is emulated, but all this about formatting applies whether the disk is missing/disabled/emulated/invalid/rebuilding (red x, yellow triangle), or not (green ball).

 

NEVER format a disk that has data you want to keep.

[trurl]

Those diagnostics show disk4 unmountable, and rebuild had started, but I didn't notice any problems with any disks yet.

1 hour ago, taflix said:

I unplugged and reseated all data and power cables.  Looked like it was running well for about 30 seconds but Disk 2 is still getting errors?

Were those diagnostics taken before all that? Because I didn't notice anything about disk2.

1 hour ago, taflix said:

Should I allow the rebuild process to continue or stop?  I'll pause it for now.

And those diagnostics didn't show the rebuild had been paused.

 

Post new diagnostics.

[taflix]

7 hours ago, trurl said:

Don't remember, and apparently some of this was on another thread, so I will ask.

 

Was there anything actually wrong with the original disk? How do you know? Do you still have the original?

 

The reason I ask is because you are currently rebuilding a corrupt filesystem, which will need to be repaired. The contents of the original might also be useful depending on how well repair works.

 

I still have the original disk.

 

In the other thread, it was giving me read errors.  The advice was to just remove it from the array and have UNRAID rebuild it.

 

That process was successful.

 

Then, recently, it started showing disk read errors again.  I tried the same solution but it wouldn't rebuild without stopping due to read errors.

 

So, that's when I purchased a new disk.

[taflix]

7 hours ago, trurl said:

Those diagnostics show disk4 unmountable, and rebuild had started, but I didn't notice any problems with any disks yet.

Were those diagnostics taken before all that? Because I didn't notice anything about disk2.

And those diagnostics didn't show the rebuild had been paused.

 

Post new diagnostics.

Those diags were taken after the re-seating.

 

Currently, UNRAID is:

 

Fresh diags attached.

 

Thank you so much for those easy to understand and well-organized explanations!

 

taflix-unraid-diagnostics-20220807-2257.zip

[JorgeB]

There were read errors on disk2 during disk4 rebuild:

 

Aug  7 13:25:18 TaFlix-UNRAID kernel: md: disk2 read error, sector=86476512
Aug  7 13:25:18 TaFlix-UNRAID kernel: md: recovery thread: multiple disk errors, sector=86476336

 

So some corruption is expected, you can check filesystem but if you have the old disk you should them compare the data.

[trurl]

Disk4 was going to need repair whether or not rebuild was good, but that is worse.

 

Diagnostics smart folder, and system/vars.txt, still show disk4 invalid, and syslog doesn't show rebuild completed yet. Maybe rebuild could be tried again if disk2 problems are fixed.

 

SMART report for disk2 looks OK, but it has never had extended self-test.

 

Not sure whether syslog entries about disk2 are a problem with the disk or something else.

Aug  7 13:25:18 TaFlix-UNRAID kernel: sd 1:0:4:0: [sdf] tag#948 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=3s
Aug  7 13:25:18 TaFlix-UNRAID kernel: sd 1:0:4:0: [sdf] tag#948 Sense Key : 0x2 [current] 
Aug  7 13:25:18 TaFlix-UNRAID kernel: sd 1:0:4:0: [sdf] tag#948 ASC=0x4 ASCQ=0x0 
Aug  7 13:25:18 TaFlix-UNRAID kernel: sd 1:0:4:0: [sdf] tag#948 CDB: opcode=0x88 88 00 00 00 00 00 05 27 86 70 00 00 01 98 00 00
Aug  7 13:25:18 TaFlix-UNRAID kernel: blk_update_request: I/O error, dev sdf, sector 86476400 op 0x0:(READ) flags 0x0 phys_seg 51 prio class 0
Aug  7 13:25:18 TaFlix-UNRAID kernel: md: disk2 read error, sector=86476336
Aug  7 13:25:18 TaFlix-UNRAID kernel: md: disk2 read error, sector=86476344
Aug  7 13:25:18 TaFlix-UNRAID kernel: md: disk2 read error, sector=86476352
Aug  7 13:25:18 TaFlix-UNRAID kernel: md: disk2 read error, sector=86476360
Aug  7 13:25:18 TaFlix-UNRAID kernel: md: disk2 read error, sector=86476368
...

 

[JorgeB]

It's not logged as a disk problem, and SMART looks good, most likely power/connection issue.

[trurl]

6 hours ago, JorgeB said:

There were read errors on disk2 during disk4 rebuild

And likely the reason emulated disk4 was unmountable.

[trurl]

Just now, trurl said:

And likely the reason emulated disk4 was unmountable.

 

@taflix

 

Maybe if you fix disk2 problems, emulated disk4 will be mountable.

 

How is disk2 connected to power? Have you tried changing SATA cables?

[taflix]

2 hours ago, trurl said:

 

@taflix

 

Maybe if you fix disk2 problems, emulated disk4 will be mountable.

 

How is disk2 connected to power? Have you tried changing SATA cables?

 

Thank you.  Funny thing is that Disk 2 was never originally an issue.

 

Ok, I will double check the power connection to Disk 2. 

 

I did swap in a fresh set of new cables.

 

If the errors on Disk 2 continue after this, what would be the next step?