So I recently got 2 10TB HDD to replace my original 2TB HDD, was following the tutorial to replace the Parity first (https://docs.unraid.net/legacy/FAQ/parity-swap-procedure/) which everything seems to go well, and when I try to replace one of the HDD to a newer  one, after data rebuild is finish newer drive suddenly goes empty?

 

 

For me it looks like the Parity Swap went well, but it attempting to rebuild Disk3 seems to go wrong causing all the data on that drive just disappear.( Luckily the old drive still seems to have all the data available so I don't think any data is lost just yet.

 

But now the Disk1 Seems to be throwing a lot of error, Structure needs cleaning, can I safely do XFS Repair on this drive?

 

Also another info as well, initially I was trying to replace the Disk1 with one of the new 10TB Drive right after the Parity Swap, which seems to went well (It shows the new Drive name and everything) but I notice that the storage indicated by UnRAID is remain at 2TB, so I end up restarting the array.

 

That's where the weird thing happened, right after the array stopped the Disk1 just suddenly change back to the old 2TB Drive instead of the 10TB New Drive.

 

 

Update: try todo a xfs repair and completely failed and it looks like all data in there is lost so I was trying to mount the 10TB which was supposed to replace the drive that have the damage xfs but I'm getting this error (Nov 25 13:50:50 Lunar-UnRAID unassigned.devices: Mount of 'sdc1' failed: 'mount: /mnt/disks/IBM-ESXS_ST10000NM0226_E: mount(2) system call failed: Function not implemented.)   Wonderif there's anyway for me to mount this to see if the data is still available?

 

Try to do a fdisk does indicated that there's something in it 

Disk /dev/sdc: 9.03 TiB, 9931038130176 bytes, 2424569856 sectors
Disk model: ST10000NM0226 E 
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 3602AAF0-6DC7-4E9F-BFEC-947C289BD0FD

Device     Start        End    Sectors Size Type
/dev/sdc1      8 2424569850 2424569843   9T Linux filesystem

I think both the 10TB is 4kn drive not sure if that mattere or not

 

 

Other Question is assuming the old drive for disk3 is still functional with all the data, Would it be possible for me to try to bring the Original Disk 3 online and rebuild the Disk1 that way?

 

lunar-unraid-diagnostics-20241124-2058.zip

Edited November 25, 20241 yr by LunarstarPony

you may have grabed the wrong drive to swap in that case d1 had all the data. Sorry but I think that data is gone gone. First i'm seeing that error:
"Structure needs cleaning"

 

Sounds like files system corruption.... General Torubleshotting

 

The issue you are experiencing might be related to file system corruption on the new drive. Here's how you can address it:

 

1. Verify and Back Up

Check the Data on the Old Drive: Since you mentioned the old drive's data is intact, ensure you back up any critical files to a separate location before proceeding further.

Do Not Write to the New Drive: Avoid further writes to the new drive to prevent additional corruption.

 

2. Stop the Array

Log in to the Unraid web interface.

Navigate to Main > Array Operation, and stop the array.

 

3. Check File System on the Problematic Drive

Unraid allows file system checks and repairs. Follow these steps:

Go to Main and locate the problematic drive.

Under the specific drive, click on its Check File System Status option.

Choose the xfs_repair option (assuming your drive uses XFS, which is common in Unraid).

If prompted, select the -n option for a dry-run first (this will not make changes but will show issues).

If issues are detected, re-run without the -n flag to repair.

 

4. Replace or Rebuild Drive Again

If the file system repair doesn’t resolve the issue:

Stop the array again.

Unassign the new drive from the array.

Start the array without the drive to simulate a "missing drive."

Stop the array, reassign the new drive, and start the rebuild process.

 

5. Test the New Drive for Hardware Issues

The "Structure needs cleaning" error may also indicate potential hardware issues. Before reusing the new drive:

Use Preclear to test the drive thoroughly. This will scan for bad sectors and prepare the drive for use in the array.

Check SMART data for signs of potential failure.

 

6. Copy Data Back

If rebuilding or repairing the file system resolves the issue:

Copy the data back from your old drive.

Use rsync or the Krusader plugin for safe, error-checking file transfers.

 

7. Prevent Future Issues

Always verify drive health (SMART tests) before adding new drives to the array.

Keep the array updated with the latest version of Unraid for improved stability and compatibility.

Edited November 25, 20241 yr by bmartino1

15 minutes ago, bmartino1 said:

you may have grabed the wrong drive to swap in that case d1 had all the data. Sorry but I think that data is gone gone. First i'm seeing that error:
"Structure needs cleaning"

 

Sounds like files system corruption.... General Torubleshotting

 

The issue you are experiencing might be related to file system corruption on the new drive. Here's how you can address it:

 

1. Verify and Back Up

Check the Data on the Old Drive: Since you mentioned the old drive's data is intact, ensure you back up any critical files to a separate location before proceeding further.

Do Not Write to the New Drive: Avoid further writes to the new drive to prevent additional corruption.

 

2. Stop the Array

Log in to the Unraid web interface.

Navigate to Main > Array Operation, and stop the array.

 

3. Check File System on the Problematic Drive

Unraid allows file system checks and repairs. Follow these steps:

Go to Main and locate the problematic drive.

Under the specific drive, click on its Check File System Status option.

Choose the xfs_repair option (assuming your drive uses XFS, which is common in Unraid).

If prompted, select the -n option for a dry-run first (this will not make changes but will show issues).

If issues are detected, re-run without the -n flag to repair.

 

4. Replace or Rebuild Drive Again

If the file system repair doesn’t resolve the issue:

Stop the array again.

Unassign the new drive from the array.

Start the array without the drive to simulate a "missing drive."

Stop the array, reassign the new drive, and start the rebuild process.

 

5. Test the New Drive for Hardware Issues

The "Structure needs cleaning" error may also indicate potential hardware issues. Before reusing the new drive:

Use Preclear to test the drive thoroughly. This will scan for bad sectors and prepare the drive for use in the array.

Check SMART data for signs of potential failure.

 

6. Copy Data Back

If rebuilding or repairing the file system resolves the issue:

Copy the data back from your old drive.

Use rsync or the Krusader plugin for safe, error-checking file transfers.

 

7. Prevent Future Issues

Always verify drive health (SMART tests) before adding new drives to the array.

Keep the array updated with the latest version of Unraid for improved stability and compatibility.

Oh no, the data is still there as far as I know but it only exist in the old drive, not the drive that it being replaced with (The drive tht do the data rebuild pretty much)

Pretty much data rebuild seems to have failed or somethin, but I cold still copy the data over from the old one

Edited November 25, 20241 yr by LunarstarPony

a more experienced user may need to assist here again it sounds like your disk 1 and parity disk got swaped.

Sorry, I'm not sure how to further assist here with out data loss.

Aw Thanks eitherway, I think the Parity Swap went well, but when I tried to swap out an old 2TB drive to a new one is what went wrong (The other 2 Drive in the pool seems to still have all the data).

 

(But since I'm just replacing old drive means the data is still in the old drive as far as I know)

 

Could also be that since I didn't check the parity after doing the parity swap so a broken parity cause it to unable to calculate data as well.

Edited November 25, 20241 yr by LunarstarPony

Try to do the xfs_repair will all 3 drive which seems like the Drive 1 have a lot of CRC Error (Do I just repir that one first?

 

Also another info as well, initially I was trying to replace the Disk1 with one of the new 10TB Drive right after the Parity Swap, which seems to went well (It shows the new Drive name and everything) but I notice that the storage indicated by UnRAID is remain at 2TB, so I end up restarting the array.

 

That's where the weird thing happened, right after the array stopped the Disk1 just suddenly change back to the old 2TB Drive.

Edited November 25, 20241 yr by LunarstarPony

There's filesystem corruption on disk1 and disk3, this may suggest an underlying issue, may be a good idea to run memtest first, then check filesystem on both disks.

2 minutes ago, JorgeB said:

There's filesystem corruption on disk1 and disk3, this may suggest an underlying issue, may be a good idea to run memtest first, then check filesystem on both disks.

But it's been running fine until I started the Parity swap and the disk replacement tho But I'll still give memtest a go

10 minutes ago, JorgeB said:

There's filesystem corruption on disk1 and disk3, this may suggest an underlying issue, may be a good idea to run memtest first, then check filesystem on both disks.

Also I already tried the xfs repair which does not work on the disk 1, the utility end up just placing a lost and found folder.  As for Disk 3 I feels like the Parity Rebuild doesn't even work, Because what I was trying to do was replace a older drive to a new drive, but after the Data-Rebuild the new drive just appears empty. (Maybe could because the Disk 1 is already broke at that time but I and UnRAID just didn't notice?)

 

Which is why I ask if it would be possible for me to remount the old drive again? If I'd able to remount the old Disk3 maybe It would be possible for me to just rebuild Disk1 since the parity should be valid again?

 

Or the other options would be the weird bug I experience where I have mounted the new 10TB Drive in Disk1 and complete the data rebuild already(At least UnRAID Said rebuild is complete), but settings seems to got revert as soon as I stop the array, so maybe the full data is already available in the new 10TB Drive? 

Edited November 25, 20241 yr by LunarstarPony

I wonder if it was possible to check the log, maybe there's indication where the data is being rebuild? (sdc is the new 10TB Drive) (It could be either in the original drive (2TB) or the new (10TB) Drive

Edited November 25, 20241 yr by LunarstarPony

44 minutes ago, LunarstarPony said:

maybe there's indication where the data is being rebuild?

Not sure what you mean, there are no disabled disks, data is not being rebuilt anywhere.

26 minutes ago, JorgeB said:

Not sure what you mean, there are no disabled disks, data is not being rebuilt anywhere.

Let me explain what I did in 2 part so it won't be that confusing. Would be easier to figure out a solution as well. There's currently 6 Drives in the system,

A: Old Parity Drive

B: Disk 1

C : Disk 2

D : Disk 3 (Have Smart Error needed to be replaced)

E: One of the new 10TB Drive to replace Parity

F: Another new 10TB Drive to replace one of the Data Drive

Edited November 25, 20241 yr by LunarstarPony

1 hour ago, JorgeB said:

Not sure what you mean, there are no disabled disks, data is not being rebuilt anywhere.

First Part I was just trying to do the parity swap (https://docs.unraid.net/legacy/FAQ/parity-swap-procedure/)

 

Step1: Stop Array

Step2: Unassign B Drive

Step3: Start Array

Step4: Stop Array 

Step5: Unassign A Drive

Step6: Assign E Drive as Parity Drive and A Drive as Disk 1

Step7: Press Copy (For the Parity Info) and wait until copy completes

Step8: Start the array

Step9: Stop data-rebuild and the array instantly 

Step10: Unassign Disk1 and Assign F Drive as Disk1

Step11: Wait for the rebuild Completes.

 

And that's pretty much all, but the weird thing that happened is after the Step11 everything seems to be normal at first with the F Drive as Disk1, but right after I stop the array, The Disk1 is somehow assign to the A Drive again.

 

That's why I'm a little bit confused on where did it actually rebuild the data. Is it on the A Drive or F Drive? (This would be very easy to figure out just by looking at the data but I don't think the new 4kn Drive can be mounted using the simple mount command)

 

Because if it's on the new F Drive, that could explain why there's so many error on Disk1 since this is supposed to have data for the parity drive not files, and the actual data it rebuild is on F Drive which is currently Unassigned.

 

Oh and also Both Drive have been PreCleared before 

Edited November 25, 20241 yr by LunarstarPony

1 hour ago, LunarstarPony said:

The Disk1 is somehow assign to the A Drive again.

That's suggests a flash drive problem

 

1 hour ago, LunarstarPony said:

after the Step11 everything seems to be normal at first with the F Drive as Disk1

If the rebuild was done with that disk assigned, it would have used it as the target for the rebuild.

 

 

 

13 minutes ago, JorgeB said:

That's suggests a flash drive problem

 

If the rebuild was done with that disk assigned, it would have used it as the target for the rebuild.

 

 

 

Then is there a way to force assign without clearning drive then? Sicne if it did rebuild with the new 10TB Drive I should able to assign that but unraid is askign to clear data. If there's a way to force assign drive I might able to fix both problem at the same time. Since If i could force assign the old 2TB Drive back into Disk3 The parity should be valid and I might able to just rebuild Disk1, Since the old Disk3 is known to be fully functional.

Edited November 25, 20241 yr by LunarstarPony

IIRC 4kn XFS drives won't mount outside the array, I would do a new config, don't assign parity, assign all data drives and compare the data, then do another new config this time with parity and the drives with the correct data, and re-sync parity.

36 minutes ago, JorgeB said:

IIRC 4kn XFS drives won't mount outside the array, I would do a new config, don't assign parity, assign all data drives and compare the data, then do another new config this time with parity and the drives with the correct data, and re-sync parity.

Is there a more specific tutorial on how to do that? I neevr use New Config before

 

Another question is if the Disk1 is confirmed to be gone but I could managed to get Disk2, Disk3, and the parity drive online would I able to rebuild the data still if using the new config?

Edited November 25, 20241 yr by LunarstarPony

1 hour ago, LunarstarPony said:

Another question is if the Disk1 is confirmed to be gone but I could managed to get Disk2, Disk3, and the parity drive online would I able to rebuild the data still if using the new config?

 

You can try, stop the array, unassign disk1, start the array and see if the emulated disk mounts, but most likely it won't without at least repairing the filesystem, post new diags after that.

10 hours ago, JorgeB said:

 

You can try, stop the array, unassign disk1, start the array and see if the emulated disk mounts, but most likely it won't without at least repairing the filesystem, post new diags after that.

Emulated disk definitely will not mount now because after the above event without checking I tried to replace disk3 which cause another empty drive, Can only tell that the old drive is still fully functional. So I wonder if I'd able to remount the old Disk3 again, which means Parity, Disk2, and 3 should be functional and hopefully I'd able to bring back 1?

 

Also I've checked most likely only disk1 having xfs issue, which is the drive we trying to replace so should be good there.

Edited November 26, 20241 yr by LunarstarPony

9 hours ago, LunarstarPony said:

So I wonder if I'd able to remount the old Disk3 again, which means Parity, Disk2, and 3 should be functional and hopefully I'd able to bring back 1?

Only if parity is still valid with those disks.

When you change a drive would the Parity data change? Or does it actually just copy all data in the drive

2 minutes ago, JorgeB said:

Only if parity is still valid with those disks.

?

Just mounting the other drives alone will make parity a little out of sync, but it may still be able to recover, also depending on the filesystem.

It's xfs

1 hour ago, JorgeB said:

Just mounting the other drives alone will make parity a little out of sync, but it may still be able to recover, also depending on the filesystem.

I'll take it as a win if I could get back some of it tbh, tho could I try to see if there's data in there first by doing new config? Because I'm not sure if I could do another new config with a missing drive or not later.

 

Because I want to try to see if there's data in the Disk1 First because it there is parity won't even be needed, just kinda worry if it's not there we wouldn't able to try and see if we can rebuild data anymore since the config is lost

Edited November 26, 20241 yr by LunarstarPony