Recent corruption issues

[WeeboTech]

ya know, it's understood that reiserfs on those suspect disks was corrupt. The question that comes to mind is why.

If you never ran the betas with the defective filesystem drivers, how did the corruption on the suspect disks occur?

That's the million dollar question.

 

 

It's crucial that you finish the memtest for at least one pass for sanity sake.

 

 

[johnodon]

Another update...

 

Continued to migrate data back to parity protected DISK1 (~820GB total so far) with no issues at all.

 

Also, NZBGet downloaded another ~30GB to DISK4 with no issues.

 

And yes weebo...that is indeed the question.  I can tell you one thing though;  I will not be on RFS long enough to find out. 

[johnodon]

OK guys...I can't take it anymore.

 

Copying data from the spare disk back to the newly XFS formatted DISK1 with parity enabled averaged an awful 16MB/s.  When I copied the other way (DISK1 --> spare disk) I saw an average of 90MB/s with some rates hovering around 125MB/s.  I have no idea what the issue is but it seems to be tied directly to parity.  I have since ordered a new parity drive (4TB HGST NAS drive) to see if it is drive related.

 

Anyway, I will be performing the rest of my migration sans parity.  I understand the risks but I just can't accept a total of ~32 hours to transfer data back and forth (array --> spare = ~6 hours, spare --> array = 26 hours).

 

Please let me know if this process will work OK.  The part that I am struggling with is when to run the New Config tool.  After all disks are transferred and assigned?  After each disk transfer and assignment?

 

- Unassign parity disk

- Format spare disk as XFS (mkfs.xfs /dev/sdl1)

- Mount spare drive via SNAP

- rsync data from DISK2 --> SNAP

- Unmount SNAP disk

- Stop array

- Unassign DISK2

- Assign SNAP disk as DISK2

- Start array

- Format old DISK2 as XFS

- Mount old DISK2 via SNAP

- rsync data from DISK3 --> SNAP

 

----  Continue with remaining disks

 

 

 

 

 

 

[RobJ]

Please let me know if this process will work OK.  The part that I am struggling with is when to run the New Config tool.  After all disks are transferred and assigned?  After each disk transfer and assignment?

 

- Unassign parity disk

- Format spare disk as XFS (mkfs.xfs /dev/sdl1)

- Mount spare drive via SNAP

- rsync data from DISK2 --> SNAP

- Unmount SNAP disk

- Stop array

- Unassign DISK2

- Assign SNAP disk as DISK2

- Start array

- Format old DISK2 as XFS

- Mount old DISK2 via SNAP

- rsync data from DISK3 --> SNAP

 

----  Continue with remaining disks

 

Looks pretty good in general, but as nice as SNAP is, I think I would prefer to let UnRAID take care of all the mounting, unmounting, and formatting.  You have 10 data drives, add the spare as Disk 11, formatted as XFS, then copy from md2 to md11, swap their assignments, format disk 11 as XFS, copy md3 to md11, swap them, etc etc.  This way, you know that UnRAID has prepared the XFS disks just the way UnRAID wants them, and expects to see them.  Of course, there's a bunch of array stops and starts and reassignments in that loop.

 

The one other change I would require, but will add significant time, is some sort of file verification.  I would want to either file compare all of the copies or run an md5 checker on source and destination files or run one of the file verifying rsync commands WeeboTech offered somewhere.

[johnodon]

 

You have 10 data drives, add the spare as Disk 11, formatted as XFS, then copy from md2 to md11, swap their assignments, format disk 11 as XFS, copy md3 to md11, swap them, etc etc.  This way, you know that UnRAID has prepared the XFS disks just the way UnRAID wants them, and expects to see them.  Of course, there's a bunch of array stops and starts and reassignments in that loop

 

That makes completes sense!  Thx Rob.

[johnodon]

Seeing an odd issue that I think I read about in another thread.  I'll see if I can find it.

 

Per Rob's suggestion, I added my spare disk in the DISK11 slot, changed the format to XFS and started the array.  As expected, unRAID saw the drive and unformatted so I formatted it.  I then began the rsync process from DISK3 to DISK11.

 

When rsync was complete, I stopped the array, unassigned both DISK3 and DISK11 and swapped them.  When I went to start the array, there was a note the said disk assignment was wrong and a new assignment would be created is the array is started.  I started the array but now DISK3 showed as unformatted.

 

I put the disks back in their original slots and DISK3 showed as formatted as RFS (as it had been).  I did the swap a few times and DISK3 always came back as unformatted when the disks were swapped into the new assignments.

 

In the end, I just stopped the array, formatted DISK3 as XFS and rsync'd DISK11 to DISK3.

 

When I move onto the next disk, I'll see if I can duplicate.

 

Have you guys seen this before?

 

John

[WeeboTech]

Seeing an odd issue that I think I read about in another thread.  I'll see if I can find it.

 

Per Rob's suggestion, I added my spare disk in the DISK11 slot, changed the format to XFS and started the array.  As expected, unRAID saw the drive and unformatted so I formatted it.  I then began the rsync process from DISK3 to DISK11.

 

When rsync was complete, I stopped the array, unassigned both DISK3 and DISK11 and swapped them.  When I went to start the array, there was a note the said disk assignment was wrong and a new assignment would be created is the array is started.  I started the array but now DISK3 showed as unformatted.

 

I put the disks back in their original slots and DISK3 showed as formatted as RFS (as it had been).  I did the swap a few times and DISK3 always came back as unformatted when the disks were swapped into the new assignments.

 

In the end, I just stopped the array, formatted DISK3 as XFS and rsync'd DISK11 to DISK3.

 

When I move onto the next disk, I'll see if I can duplicate.

 

Have you guys seen this before?

 

John

 

 

You may have to change the File System Type for the disk so the mount command puts in the parameters in when executed.

i.e. click on the new disk.  Main -> Array Devices -> click on the disk # -> change filesystem type to the known version.

[johnodon]

Finally finished migrating all of my data disks from RFS to XFS.  Parity sync is running now.

 

Thank you to everyone for all of your help!

 

John

[WeeboTech]

FWIW, I've moved 4GB to reiserfs disks  from multiple 1TB disks without any detectable corruption.

I've run the md5sum after the initial copy and I'm doing it 1 more time before I move the drive to service.

 

Point here is, there's no corruption on my copies, so something else might be going on that we have not detected yet.

[AmpegV4]

ya know, it's understood that reiserfs on those suspect disks was corrupt. The question that comes to mind is why.

If you never ran the betas with the defective filesystem drivers, how did the corruption on the suspect disks occur?

That's the million dollar question.

 

 

It's crucial that you finish the memtest for at least one pass for sanity sake.

 

Jon did you ever actually finish a memtest & confirm your memory is not causing this issue as Weebo suggested? (I think it's unlikely but would be great if we could rule it out & determine the cause) I've had a single faulty stick on my main rig which caused weeks of unnecessary troubleshooting before discovery.

[WeeboTech]

ya know, it's understood that reiserfs on those suspect disks was corrupt. The question that comes to mind is why.

If you never ran the betas with the defective filesystem drivers, how did the corruption on the suspect disks occur?

That's the million dollar question.

 

It's crucial that you finish the memtest for at least one pass for sanity sake.

 

Jon did you ever actually finish a memtest & confirm your memory is not causing this issue as Weebo suggested? (I think it's unlikely but would be great if we could rule it out & determine the cause) I've had a single faulty stick on my main rig which caused weeks of unnecessary troubleshooting before discovery.

 

 

I'm also curious of the model number for the 3 drives that were having the continual corruption. I wonder if there is a pattern here.

[johnodon]

memtest completed 6 passes without error.

 

All of the drives that had issues were WD green drives although varying models...WD20EARS and WD20EARX.

 

I also had a 1.5TB WDEARS drive that I had to pull from the array some time ago.

 

No more WD drives for me again...ever!

[hackztor]

Well you are using green drives afterall. Green drives are known for issues and problems. The red are way better and worth the extra cost. Seagate has green drive like ones and I have had issues with them. Red's and se or re2 will be much better.

[johnodon]

Just wanted to provide a quick update.

 

It has now been ~2 months since migrating all of my data disks to XFS.  Since that time, I have not had a single corruption issue.

 

Also, any hard crashes that I have experienced since that time I have been able tie directly to Docker apps (namely NZBGet or NZBDrone).  About a month ago, I retired most of my Docker apps (except MariaDB and Headless Kodi) and moved the apps into VMs.  My server has been purring like a kitten! 

 

John

[WeeboTech]

Thanks for the update!