Goof Day Everyone!

I recently purchased 2x 22Tb drives, my largest current drive is my 16tb Parity drive. I have been trying to rebuild the parity on to one of the 22TB drives, but the rebuild is taking a LONG time. At one stage the speed is 30kbps.

I checked the logs and noticeds that one of my 10tb drives is giving a TON of read errors. I was going to get rid of the drive, but I thought that it might be better to do the parity change first and then move it on. Not sure what I can do now because I'm quite sure that this parity rebuild is never going to happen with the 10tb drive still in the system.

Attached are the logs.

TIA

datavault972-diagnostics-20261010-0720.zip

The diagnostics confirm that disk 3 is failing. It has 15,009 pending sectors, and the logs show repeated unreadable-sector errors. These errors explain the extreme slowdown.

I recommend canceling the parity sync, stopping Docker/VMs and other array writes, then stopping the array while we establish a recovery plan. The new parity is incomplete, so you cannot safely rely on it to rebuild disk 3.

Disk 5 also needs attention: it has 65,486 reallocated sectors, although it is not currently reporting Unraid read errors, and that number is a little suspect, can be false.

Before changing any disk assignments:

• Do you have backups of the important data, especially the files on disks 3 and 5?

• Do you still have the original 16 TB parity disk, untouched since its removal?

• Have you copied, changed, or deleted files on the array since replacing parity? Has mover run?

• Did you use New Config or change any data-disk assignments during the upgrade?

Keep the original disks unchanged. Do not format, preclear, or start another rebuild yet. The old parity might provide a recovery option, but we need to establish whether it still matches the data disks.

• Do you have backups of the important data, especially the files on disks 3 and 5?

1. I don't have backups here, btut the data is not critical

• Do you still have the original 16 TB parity disk, untouched since its removal?

2. Yes, this drive is still available.

• Have you copied, changed, or deleted files on the array since replacing parity? Has mover run?

3. I did try to download some files but they did not complete, my mover normally runs at 3 am, so I assume that it has run twice since the start of the parity rebuild.

• Did you use New Config or change any data-disk assignments during the upgrade?

4. I only unassigned my 16tb drive and reassigned the new 22tb drive.

Since the data isn't critical, I recommend copying what you can directly from disk3. Don't try to rebuild it.

You still have the old parity disk, but it is no longer in sync with the array. Mover and the downloads have written to other disks since you removed it (the diagnostics show writes on disks 9, 10 and 11). A disk3 rebuild from it would have corrupted data in places you can't identify. Disk3 itself has had almost no writes, and most of it should still be readable. The new parity is incomplete and invalid wherever disk3 couldn't be read, so you can't use it either.

Suggested plan:

1. If the parity sync is still running, cancel it. Stop any downloads, and disable the mover schedule for now. Also go to Settings > Global Share Settings and add disk3 to the excluded disks, so nothing new is written to it.

2. Install the Unassigned Devices plugin if you don't have it. Format the second 22 TB disk as XFS there and mount it.

3. Copy disk3 to it. Because of the bad sectors, some files will fail and the copy may be slow in places. Run it from the terminal with a log so you get a list of the failed files (replace NEWDISK with the

mount name shown in UD):

nohup rsync -aX /mnt/disk3/ /mnt/disks/NEWDISK/ > /boot/disk3-copy.log 2>&1 &

The copy keeps running if you close the terminal. Any line in the log with "error" or "failed" is a file that could not be read completely.

4. While the copy runs, start an extended SMART test on disk5.

5. When the copy is done and disk5 passes, unmount the new disk in UD. Then go to Tools > New Config, keep all current assignments, and replace disk3 with the new 22 TB disk. Keep the other 22 TB as parity and don't check "parity is already valid". Start the array and let the parity sync run. It should go at normal speed without disk3.

Keep the old 16 TB parity disk and the original disk3 unchanged until the new parity sync has finished. If disk5 fails the SMART test, stop at that point and post new diagnostics before you do anything else.

Thanks for the assistance

1. Done!

2. Done!

3. I did that, but in the terminal, I wasn't seeing any logs being written in real time, Closer to the end of the attached video, you will see what I mean.

4. Disk5's selftest is still ongoing so I will wait.

5. Will do after the above steps is done. Please can you let me know how I can tell if the copying is indeed happening.

Good progress. Not seeing anything in the terminal is normal. The & at the end runs the copy in the background, and its output goes to the log file instead of the screen. Also, rsync only writes to that log when it has a problem, so an empty log means no errors so far.

To check that the copy is running, type these commands in the terminal (replace NEWDISK with the name you used):

pgrep -a rsync

df -h /mnt/disks/NEWDISK

cat /boot/disk3-copy.log

- pgrep should list one or more rsync processes. If it shows nothing, the copy has stopped or finished.

- df should show the 22 TB disk in the "Size" column, and "Used" should increase each time you run it. If the Size is not about 20T (for example, if it shows "rootfs" or a small size), the name was wrong and the copy is going to RAM. In that case, run pkill rsync and post the output before you do anything else.

- You can also watch the Main page: the reads counter on disk3 and the writes counter on the new disk in UD should keep going up.

Some parts will be slow when rsync reaches the bad sectors, so the "Used" value can stop for a while before it continues. That is expected.

When pgrep shows nothing, the copy is finished. Then run:

grep -iE "error|failed" /boot/disk3-copy.log

This lists the files that could not be copied completely. A last line like "some files/attrs were not transferred (code 23)" is normal with a failing disk. Post that output together with the disk5 SMART result, and then continue with the New Config step.

Thanks again for the response.

1. When I ran the pgrep -a rsync command, I got the below response, which indicates that the copying is done.

2. When I ran the df -h /mnt/disks/disk13

   It showed that my drive size is 1mb, could this mean that my drive is not formatted yet?

3. Additionally you mentioned that I can check on the main page, however, how will that work if I have not started the array due to the fact that when I try to add the 2 22tb drivce (one for parity and one forregular) in the Array operations section, it states that, " You may not add new disk(s) and also remove existing disk(s)." Should I put "No device" in the parity drive and then start the drive with the 2nd 22tb drive?

You're right, the 1 MB means the new disk was never mounted, so the copy did not run. Unassigned Devices puts a small 1 MB RAM area on /mnt/disks. When a disk isn't mounted at the name you use, the copy goes into that area, fills it, and stops right away. Nothing was lost, but we need to redo the copy.

About the array: the second 22 TB disk should not go into the array at all. It stays in Unassigned Devices until the New Config step at the end. Your idea to set parity to "No device" is correct, though, because the new parity is not valid yet. Removing parity also stops a sync from starting and reading disk3 at the same time as the copy.

1. Reboot the server. This clears the full 1 MB area. The array is stopped, so this is safe.

2. On the Main page, set Parity to "No device". Make sure every data disk is in its original slot, including the old disk3. Leave the second 22 TB disk unassigned, and don't add any new slots. Start the array. The mover and downloads should still be disabled.

3. In Unassigned Devices, find the second 22 TB disk. Check the size and serial number. Your old 16 TB parity disk probably shows in that list too, so don't touch it. Format the 22TB disk as XFS if UD doesn't show a filesystem on it yet, then click Mount. Note the name in the mount point column.

4. Check that it is mounted (replace NAME with that mount point name):

df -h /mnt/disks/NAME

This must show a size of about 20T. If it shows 1.0M, the disk is still not mounted, so stop and post a screenshot of the UD section.

For the copy, I recommend that you install the tmux plugin from Community Apps (search for "tmux") and run the copy inside a tmux session instead of with nohup. Then you can see the progress, close the browser, and come back to it later:

rsync -aX --info=progress2 /mnt/disk3/ /mnt/disks/NAME/ 2> /boot/disk3-copy.log

You will see the progress on the screen, and the files that fail go to the log.

With the array started, you can also watch the Main page. Reads on disk3 and writes on the UD disk should keep going up. The copy will be slow in some places because of the bad sectors.

When the copy finishes, run:

grep -iE "error|failed" /boot/disk3-copy.log

Post that output with the disk5 SMART result. Then we can do the New Config step: the first 22 TB disk goes back as parity, and the copied 22 TB disk replaces disk3.

My main problem currently is how to change the file system to XFS. The only way that I get the option to change the file system is when I stop the array and then, "place" the drive into the array WITHOUT starting it. Then I click on the disk number (in this case disk13), then I get the option to change to XFS or XFS - Encrypted. After that I click on Apply and then done, but that does not appear to complete the change. I assume that I need to start the array after I do this, but that goes against your instruction of assigning the drive.

Can you guide me from here, apologies if I missed something that you pointed out earlier

Attached is the Test for Disk 5

datavault972-smart-20261011-1051.zip

Edited 21 hours ago21 hr by Tuntira

You didn't miss anything: the disk was never formatted. Formatting it through an array slot won't work, and that extra slot is also why the array won't start.

1. Stop the array. On the Main page, set the disk13 slot back to "No device". If the slot count went up when you added it, set it back to the previous number. Also confirm that Parity is set to "No device" and every data disk is in its original slot, including the old disk3.

2. Go to Settings > Unassigned Devices, set "Enable Destructive Mode" to Yes, and click Apply. UD hides the Format button until you do this, which is why you couldn't find it.

3. On the Main page, find the second 22 TB disk in the Unassigned Devices section. Check the size and serial number first. Your old 16 TB parity disk is also listed there, so don't touch it.

- If the disk shows a partition under it, click the red X next to the disk to clear it first.

- Click Format, choose XFS, confirm, and wait until it finishes.

4. Click Mount. Note the name in the mount point column. It will not be "disk13". It is usually based on the disk's model and serial, and you can rename it before you mount.

6. Start the array. It should now start without parity and without a sync.

7. Check the mount (replace NAME with the mount point name):

df -h /mnt/disks/NAME

It must show a size of about 20T. If it still shows 1.0M, stop and post a screenshot of the UD section. Then continue with the tmux and rsync copy from my last post, using that same

NAME.

The disk5 extended test passed, and it has no pending sectors. Its reallocated sectors and errors are all old, so it's OK to keep for now. It's the next disk I would replace when you

can.

Thank you for the detailed instructions:

I have successfully mounted the drive:

I Started the copy and will let it run as you said.

Once I am done, I run the grep -iE "error|failed" /boot/disk3-copy.log

And provide feedback.