Read errors while creating parity

[RandomFactoid]

Hi,

 

yesterday I migrated from Truenas Scale to Unraid and am absolutely loving it so far, however things started getting a bit sketchy today.

My setup:

Microserver Gen8/Xeon E3-1260L/12GB ram/2x 4TB WD Red/1x 250GB SSD

 

Whilst using Truenas my both 4TB drives were in a ZFS pool (mirrored).

 

During migration I used the Unraid ZFS plugin to mount the whole pool, tried offlining one drive(Drive1) and checked whether I can still access the online ZFS drive (Drive2), which I could. I then formatted drive1 and manually copied approx 2tb of mostly media from Drive 2 to Drive1 which was the only drive in the array. I then 'destroyed' the zfs pool and set drive2 as my parity drive, letting unraid do parity rebuild/sync.

It's been running for approx 6, however I started getting read errors (approx 30000 at the moment) in Drive1 (the one in array). It did slow down substantially at that point (from 130MB/s to kilobytes), however currently it's still running at a decent, but slower speed (~90MB/s).

I will let it finish, however I have a feeling something's trying to break - I'm just not sure how far from breaking that 'something' is.

 

My SSD is set as a cache drive.

 

After it finishes sync'ing, how do I proceed ? Some pointers to diagnose the issue would be much appreciated. For now I have stopped the dockers i've set up and am just letting it finish building parity.

 

Thanks

[Squid]

Post your diagnostics

[RandomFactoid]

5 minutes ago, Squid said:

Post your diagnostics

Thanks for replying. Here goes.

adulis-diagnostics-20211227-1704.zip

[Squid]

Looks like disk1 is shot

197 Current_Pending_Sector  -O--CK   200   200   000    -    214

 

Problem is that I'm not 100% sure what actually happens when parity is invalid (as it's building) when a read error happens on another drive.

 

If Parity is valid, then the system recalculates what's supposed to be on disk1 via Parity and rewrites the appropriate data.  If the write fails, then the disk gets disabled.

 

But, since in your case if Parity isn't valid the above would result in corruption on disk 1.  I *think* that the system wouldn't attempt to rewrite, but I'm not 100% sure.

 

Either way, your best course of action would be to replace disk 1 and re-copy the data.

 

@JorgeB Do you know what the OS does in this situation?

[JorgeB]

7 minutes ago, Squid said:

But, since in your case if Parity isn't valid the above would result in corruption on disk 1.  I *think* that the system wouldn't attempt to rewrite, but I'm not 100% sure.

AFAIK this is what happens.

[RandomFactoid]

RIght..so it finished succesfully (...i guess?).  I found an old 2TB drive that I'm about to add to an empty slot. What's the best way to proceed ? Thinking to move what I can off the bad 4tb drive to the 2tb at least temporarily. Although if parity is 'Green', do i have to ? Provided I get the bad one replaced.

 

Thanks

Edited December 27, 2021 by RandomFactoid

[JorgeB]

12 hours ago, RandomFactoid said:

What's the best way to proceed ?

 

15 hours ago, Squid said:

your best course of action would be to replace disk 1 and re-copy the data.