October 10Oct 10 Goof Day Everyone!I recently purchased 2x 22Tb drives, my largest current drive is my 16tb Parity drive. I have been trying to rebuild the parity on to one of the 22TB drives, but the rebuild is taking a LONG time. At one stage the speed is 30kbps.I checked the logs and noticeds that one of my 10tb drives is giving a TON of read errors. I was going to get rid of the drive, but I thought that it might be better to do the parity change first and then move it on. Not sure what I can do now because I'm quite sure that this parity rebuild is never going to happen with the 10tb drive still in the system.Attached are the logs.TIA datavault972-diagnostics-20261010-0720.zip
October 10Oct 10 The diagnostics confirm that disk 3 is failing. It has 15,009 pending sectors, and the logs show repeated unreadable-sector errors. These errors explain the extreme slowdown.I recommend canceling the parity sync, stopping Docker/VMs and other array writes, then stopping the array while we establish a recovery plan. The new parity is incomplete, so you cannot safely rely on it to rebuild disk 3.Disk 5 also needs attention: it has 65,486 reallocated sectors, although it is not currently reporting Unraid read errors, and that number is a little suspect, can be false.Before changing any disk assignments:• Do you have backups of the important data, especially the files on disks 3 and 5?• Do you still have the original 16 TB parity disk, untouched since its removal?• Have you copied, changed, or deleted files on the array since replacing parity? Has mover run?• Did you use New Config or change any data-disk assignments during the upgrade?Keep the original disks unchanged. Do not format, preclear, or start another rebuild yet. The old parity might provide a recovery option, but we need to establish whether it still matches the data disks.
October 10Oct 10 Author • Do you have backups of the important data, especially the files on disks 3 and 5? 1. I don't have backups here, btut the data is not critical• Do you still have the original 16 TB parity disk, untouched since its removal? 2. Yes, this drive is still available.• Have you copied, changed, or deleted files on the array since replacing parity? Has mover run? 3. I did try to download some files but they did not complete, my mover normally runs at 3 am, so I assume that it has run twice since the start of the parity rebuild.• Did you use New Config or change any data-disk assignments during the upgrade? 4. I only unassigned my 16tb drive and reassigned the new 22tb drive.
October 10Oct 10 Since the data isn't critical, I recommend copying what you can directly from disk3. Don't try to rebuild it.You still have the old parity disk, but it is no longer in sync with the array. Mover and the downloads have written to other disks since you removed it (the diagnostics show writes on disks 9, 10 and 11). A disk3 rebuild from it would have corrupted data in places you can't identify. Disk3 itself has had almost no writes, and most of it should still be readable. The new parity is incomplete and invalid wherever disk3 couldn't be read, so you can't use it either.Suggested plan: 1. If the parity sync is still running, cancel it. Stop any downloads, and disable the mover schedule for now. Also go to Settings > Global Share Settings and add disk3 to the excluded disks, so nothing new is written to it. 2. Install the Unassigned Devices plugin if you don't have it. Format the second 22 TB disk as XFS there and mount it. 3. Copy disk3 to it. Because of the bad sectors, some files will fail and the copy may be slow in places. Run it from the terminal with a log so you get a list of the failed files (replace NEWDISK with the mount name shown in UD): nohup rsync -aX /mnt/disk3/ /mnt/disks/NEWDISK/ > /boot/disk3-copy.log 2>&1 & The copy keeps running if you close the terminal. Any line in the log with "error" or "failed" is a file that could not be read completely.4. While the copy runs, start an extended SMART test on disk5. 5. When the copy is done and disk5 passes, unmount the new disk in UD. Then go to Tools > New Config, keep all current assignments, and replace disk3 with the new 22 TB disk. Keep the other 22 TB as parity and don't check "parity is already valid". Start the array and let the parity sync run. It should go at normal speed without disk3.Keep the old 16 TB parity disk and the original disk3 unchanged until the new parity sync has finished. If disk5 fails the SMART test, stop at that point and post new diagnostics before you do anything else.
October 10Oct 10 Author Thanks for the assistanceDone!Done!I did that, but in the terminal, I wasn't seeing any logs being written in real time, Closer to the end of the attached video, you will see what I mean.Disk5's selftest is still ongoing so I will wait.Will do after the above steps is done. Please can you let me know how I can tell if the copying is indeed happening. explorer_8qYOJosnes.mp4
October 11Oct 11 Good progress. Not seeing anything in the terminal is normal. The & at the end runs the copy in the background, and its output goes to the log file instead of the screen. Also, rsync only writes to that log when it has a problem, so an empty log means no errors so far.To check that the copy is running, type these commands in the terminal (replace NEWDISK with the name you used):pgrep -a rsyncdf -h /mnt/disks/NEWDISKcat /boot/disk3-copy.log- pgrep should list one or more rsync processes. If it shows nothing, the copy has stopped or finished.- df should show the 22 TB disk in the "Size" column, and "Used" should increase each time you run it. If the Size is not about 20T (for example, if it shows "rootfs" or a small size), the name was wrong and the copy is going to RAM. In that case, run pkill rsync and post the output before you do anything else.- You can also watch the Main page: the reads counter on disk3 and the writes counter on the new disk in UD should keep going up.Some parts will be slow when rsync reaches the bad sectors, so the "Used" value can stop for a while before it continues. That is expected.When pgrep shows nothing, the copy is finished. Then run:grep -iE "error|failed" /boot/disk3-copy.logThis lists the files that could not be copied completely. A last line like "some files/attrs were not transferred (code 23)" is normal with a failing disk. Post that output together with the disk5 SMART result, and then continue with the New Config step.
October 11Oct 11 Author Thanks again for the response.When I ran the pgrep -a rsync command, I got the below response, which indicates that the copying is done.When I ran the df -h /mnt/disks/disk13It showed that my drive size is 1mb, could this mean that my drive is not formatted yet?Additionally you mentioned that I can check on the main page, however, how will that work if I have not started the array due to the fact that when I try to add the 2 22tb drivce (one for parity and one forregular) in the Array operations section, it states that, " You may not add new disk(s) and also remove existing disk(s)." Should I put "No device" in the parity drive and then start the drive with the 2nd 22tb drive?
October 11Oct 11 You're right, the 1 MB means the new disk was never mounted, so the copy did not run. Unassigned Devices puts a small 1 MB RAM area on /mnt/disks. When a disk isn't mounted at the name you use, the copy goes into that area, fills it, and stops right away. Nothing was lost, but we need to redo the copy.About the array: the second 22 TB disk should not go into the array at all. It stays in Unassigned Devices until the New Config step at the end. Your idea to set parity to "No device" is correct, though, because the new parity is not valid yet. Removing parity also stops a sync from starting and reading disk3 at the same time as the copy.1. Reboot the server. This clears the full 1 MB area. The array is stopped, so this is safe.2. On the Main page, set Parity to "No device". Make sure every data disk is in its original slot, including the old disk3. Leave the second 22 TB disk unassigned, and don't add any new slots. Start the array. The mover and downloads should still be disabled.3. In Unassigned Devices, find the second 22 TB disk. Check the size and serial number. Your old 16 TB parity disk probably shows in that list too, so don't touch it. Format the 22TB disk as XFS if UD doesn't show a filesystem on it yet, then click Mount. Note the name in the mount point column.4. Check that it is mounted (replace NAME with that mount point name):df -h /mnt/disks/NAMEThis must show a size of about 20T. If it shows 1.0M, the disk is still not mounted, so stop and post a screenshot of the UD section.For the copy, I recommend that you install the tmux plugin from Community Apps (search for "tmux") and run the copy inside a tmux session instead of with nohup. Then you can see the progress, close the browser, and come back to it later:rsync -aX --info=progress2 /mnt/disk3/ /mnt/disks/NAME/ 2> /boot/disk3-copy.logYou will see the progress on the screen, and the files that fail go to the log. With the array started, you can also watch the Main page. Reads on disk3 and writes on the UD disk should keep going up. The copy will be slow in some places because of the bad sectors.When the copy finishes, run:grep -iE "error|failed" /boot/disk3-copy.logPost that output with the disk5 SMART result. Then we can do the New Config step: the first 22 TB disk goes back as parity, and the copied 22 TB disk replaces disk3.
October 11Oct 11 Author My main problem currently is how to change the file system to XFS. The only way that I get the option to change the file system is when I stop the array and then, "place" the drive into the array WITHOUT starting it. Then I click on the disk number (in this case disk13), then I get the option to change to XFS or XFS - Encrypted. After that I click on Apply and then done, but that does not appear to complete the change. I assume that I need to start the array after I do this, but that goes against your instruction of assigning the drive.Can you guide me from here, apologies if I missed something that you pointed out earlierAttached is the Test for Disk 5firefox_N4rW8l9A0D.mp4 datavault972-smart-20261011-1051.zip Edited October 11Oct 11 by Tuntira
October 11Oct 11 You didn't miss anything: the disk was never formatted. Formatting it through an array slot won't work, and that extra slot is also why the array won't start. 1. Stop the array. On the Main page, set the disk13 slot back to "No device". If the slot count went up when you added it, set it back to the previous number. Also confirm that Parity is set to "No device" and every data disk is in its original slot, including the old disk3.2. Go to Settings > Unassigned Devices, set "Enable Destructive Mode" to Yes, and click Apply. UD hides the Format button until you do this, which is why you couldn't find it.3. On the Main page, find the second 22 TB disk in the Unassigned Devices section. Check the size and serial number first. Your old 16 TB parity disk is also listed there, so don't touch it. - If the disk shows a partition under it, click the red X next to the disk to clear it first. - Click Format, choose XFS, confirm, and wait until it finishes.4. Click Mount. Note the name in the mount point column. It will not be "disk13". It is usually based on the disk's model and serial, and you can rename it before you mount.6. Start the array. It should now start without parity and without a sync.7. Check the mount (replace NAME with the mount point name): df -h /mnt/disks/NAMEIt must show a size of about 20T. If it still shows 1.0M, stop and post a screenshot of the UD section. Then continue with the tmux and rsync copy from my last post, using that same NAME.The disk5 extended test passed, and it has no pending sectors. Its reallocated sectors and errors are all old, so it's OK to keep for now. It's the next disk I would replace when you can.
October 11Oct 11 Author Thank you for the detailed instructions:I have successfully mounted the drive: I Started the copy and will let it run as you said.Once I am done, I run the grep -iE "error|failed" /boot/disk3-copy.logAnd provide feedback.
October 13Oct 13 Author So the copying has been ongoing for a good while and seems to have gone OVER 100%, is this normal?I see that the 22tb drive has filled up but not by much.
October 13Oct 13 It can be, more concerning is that it's going very slow, but it's kind of expected with a failing disk. Hopefully it will get faster in the remaining good zones.
October 15Oct 15 Author Hi there,Is it normal for the copy to take this long. It's been 4 days and according to the below, 2.3Tb of data has been copied across. The problematic drive is 10Tb, so My worry is that this could take weeks to copy?Were I to stop copying, I assume that I would lose the data on the disk?
October 15Oct 15 Yes, unfortunately this can be normal with a disk in this condition. The time doesn't depend on the amount of data, but on how many bad areas the copy hits. Every unreadable sector makes the disk retry for some time before it gives up, and disk3 had more than 15,000 of them. Healthy areas copy at normal speed, and bad areas can take hours. So yes, it can take weeks. The disk can also get worse while it's being read.Stopping the copy won't lose any data. rsync only reads from disk3, so disk3 stays as it is, and the files already copied stay on the new disk. If you run the same command again later, rsync skips the files that are already complete. It only starts the file it was copying when you stopped it from the beginning again.So I suggest copying your most important data first:1. Post the output of these commands (replace NAME with your UD mount name): df -h /mnt/disk3 /mnt/disks/NAMEgrep -ciE "error|failed" /boot/disk3-copy.logThe first shows how much data is on disk3 and how much has been copied. The second counts the files that have failed so far.2. If there are folders you care about more than others, press Ctrl+C in the tmux session to stop the current copy. Then copy those folders one at a time, for example:rsync -aX --info=progress2 "/mnt/disk3/FolderName/" "/mnt/disks/NAME/FolderName/" 2>> /boot/disk3-copy.logNote the 2>>: it adds to the existing log instead of replacing it.3. When those are done, run the full copy again with the same 2>>. It skips everything that's already there:rsync -aX --info=progress2 /mnt/disk3/ /mnt/disks/NAME/ 2>> /boot/disk3-copy.logAlso, while the copy is running, please post new diagnostics, so I can check if disk3 is getting worse. If it is getting worse quickly, copying the important data first matters even more. The progress going over 100% is normal: rsync finds more files as it goes, so the total keeps growing.
October 16Oct 16 Author Thank you for the clear instructions.... I went through the drive and most of the stuff that I want to keep I have backed up elsewhere or can rip again so I think that I will avoid the copying.With this being the case, how should I progress?
October 16Oct 16 That makes it simpler. Since the data you care about is backed up or can be ripped again, we can stop the copy and use what is already copied as the new disk3.1. Optional, but useful: before you stop the copy, open a second terminal and make a list of the files that are not on the new disk yet (replace NAME with your UD mount name):rsync -an --out-format="%n" /mnt/disk3/ /mnt/disks/NAME/ > /boot/disk3-not-copied.txtThis only compares file names and sizes, so it should be fast. The list is saved on the flash drive. Use it later to know what to restore from your backups or rip again.2. In the tmux session, press Ctrl+C to stop the copy. The file it was copying is discarded, and files that could not be read are not kept, so everything on the new disk is complete.3. On the Main page, unmount the copied 22 TB disk in UD.4. Stop the array. Go to Tools > New Config, choose to preserve all current assignments, and apply. Then on the Main page:- Set Parity to the first 22 TB disk.- Set disk3 to the copied 22 TB disk (check the serial number).- Leave the old 16 TB parity disk and the old disk3 unassigned.- Do not check "Parity is already valid".5. Start the array. The copied disk should mount as disk3 with your files, and the parity sync starts. Without the failing disk it should run at normal speed.6. Go to Settings > Global Share Settings and remove disk3 from the excluded disks. Otherwise nothing new is written to the new disk3.You can turn the mover schedule and downloads back on after the sync finishes. They are safe during the sync, but they make it slower.Keep the old 16 TB parity disk and the old disk3 as they are until the sync completes without errors. After that you can remove the old disk3.
October 17Oct 17 Author OK Great, I have started the Parity sync and it will be done tomorrow sometime.Just a couple more things:The 16Tb drive that WAS my parity, can I add it back to my system as a regular drive?Is there a resource that I can read that lets me know about the state of my drives? I want to be better at managing their health.A friend of mine sourced some 24tb drives at excellent prices (even in this economy), I'd need to make that one my new parity drive, am I right in saying that since most of my drive are fine, I wouldn't have the same issues that I had with drive 3?
October 18Oct 18 1. The old 16 TB parity disk: Yes, you can add it as a data disk, but wait until the current sync has finished without errors. Until then, keep it unchanged as a fallback. To add it, stop the array, assign it to a new data slot and start the array. Unraid will clear it first; this takes many hours, but the array stays usable while it runs. When the clear is done, the disk shows as unmountable, and you click Format on the Main page to make it usable. Its SMART is fine (a few old reallocated sectors, no pending sectors). If you plan to do the 24 TB upgrade soon, read point 3 first.2. Monitoring disk health: The docs have a page on this: https://docs.unraid.net/unraid-os/using-unraid-to/manage-storage/array/array-health-and-maintenance/ What helps the most: - Go to Settings > Notifications and set up an agent (email, Pushover, Discord, etc.). Then you get alerts for disk read errors and SMART changes. Your disk3 already had read errors a few days before you replaced parity, and a notification would have warned you. - Schedule a parity check, for example monthly. It reads every disk completely, so it finds bad sectors before you depend on those disks for a rebuild. - The SMART attributes to watch are Reallocated Sectors (5), Reported Uncorrect (187), Current Pending Sectors (197) and Offline Uncorrectable (198). Pending sectors are the most important, because they are sectors the disk can't currently read. UDMA CRC errors (199) usually mean a cable or connection problem, not a disk problem. - If you're not sure about a disk, run an extended SMART test and post the result or the diagnostics. 3. The 24 TB parity upgrade: It's correct that the 24 TB must be parity, because parity must be at least as large as your largest data disk. But replacing parity reads every data disk from start to end, so any disk with unreadable sectors will cause the same problem again. Disk3 is gone now, but disk5 is the weakest disk left, so check it before you start by runing an extended SMART test.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.