This is related to a recent post of mine, where I have 12 mirror pairs of 16TB drives as part of a larger ZFS mirror set pool. I wanted to replace the 16TB drives with 18TB drives.

I figured it would be easy to remove one drive from each mirror pair, and then select the new 18TB for each set.. It would proceed to copy the data from the original mirror to the new mirror pair drive.

However, I forgot that 'resilvering' in this situation, would virtually scrub the whole (1.3PB) pool, while trying to resilver the drives. (I had this happen a long time ago for a single drive replacement).

This resilver has been working for over 2 weeks now, with about 2 weeks remaining.

---

I am wondering, if I should change the process:

1) Should I stop the resilver, and then remove the mirror pairs manually? Would this be safe, in that it would just copy the data to the still active pool devices (outside of these replacement sets)?

2) If I let this resilver finish, and then proceed with part 2 which is removing the original 16TB drive, and adding in the second 18TB.. I would expect the 1 month resilver process to occur again??

** Why I think this is a terrible process, where the ZFS pool essentially scrubs the entire pool, to replace a single drive or multiple at one time, is that it hits the pool with 100% requests. so all the drives are being hit simultaneously, which hurts performance with all the containers I have running.

I am open to any thoughts and suggestions!

Thank you!

What Unraid version are you running? Latest releases should do sequential resilvering for mirrors, but note that if you replace multiple mirrors at the same time, zfs will only resilver one at a time by default.

Check to see if there are simultaneous writes to the multiple vdevs; if there are not, type

zpool resilver <pool name>

That should trigger parallel resilvering.

Hey @JorgeB thanks for the reply.

Unraid 7.3.1

Well, facepalm moment for me, as I read and typed too quickly. I thought your command would be to check if they were simultaneous, not to re-start the process.

I was monitoring the read/writes and there were matching reads<>writes on the various pairs.

Now, it just restarted the whole thing. :☹️ Back to 40 days ETA. Now I'm curious if the time will reduce quicker since the drives already have data, or will it start fresh again?

I had a thread a couple days ago, which showed screenshots, I believe it was resilvering all drives simultaneously.. Similar to what it's showing now again:

Are you sure it was doing it in parallel already? It shouldn't by default. Also, that command should not restart from the beginning; it would continue from the last checkpoint.

Post a screenshot of main and the diags please.

The only confidence I had, was watching the Main tab read/writes, the drives that were replaced had exact matching reads to write (16tb to 18TB) combinations. Other than that, I have no concrete evidence that it was in parallel. My incorrect assumption was the command above would check that, but as I said, I read too fast and incorrectly assumed that. haha my fault for sure.

For now, I saw the stats on the pool page, reset. Here are two screen shots one from yesterday, one from just after I ran your command. As well as your requests:

June 01

June 02 (after command)

June 02 (now, looks like the process has scanned at a much higher rate than earlier)

Attached a 'print to pdf' of the Main tab, since it would take numerous screens (Let me know if you prefer screenshots, i can do that too)

server_Main.pdfserver-diagnostics-20260602-1456.zip

Edited June 2Jun 2 by shelfactor

@JorgeB What would you take be on my thoughts from the original post. (technically even outside this current issue). Would it had been smart to just 'remove' the mirror pairs (which i did have plenty of room to do so).. and then add new mirror pairs of 18TB?

Would it be benefical, to stop the re-silvering now, and do that? Would it be safe...?

Let me rephrase ***

Original: Two drive mirrors of 16TB (Drives A & B)

Current: One original 16TB and one 18TB (Drives A & C)

Future: Two 18TB Drives (Drives C & D)

--> If I stopped the array right now, removed drive C from resilvering with drive A, (so that only the original 16TB drives are active), then install all new 18TB mirror pairs (using the ones that were being resilvered to)... Then when the array comes online, manually run the command to remove each specific 16TB Mirror pair (which only a one of each is currently active). Thus moving the data to the other drives (any, it doesnt matted to me where) in the array which also includes the new 18TB mirror pairs.

edit for current status:

Watching the Main tab, it appears it's all READ activity currently.

Edited June 3Jun 3 by shelfactor
status update

I'll need to investigate why you ae siieng a different behaviour than in my testing.

9 hours ago, shelfactor said:

--> If I stopped the array right now, removed drive C from resilvering with drive A, (so that only the original 16TB drives are active), then install all new 18TB mirror pairs (using the ones that were being resilvered to)... Then when the array comes online, manually run the command to remove each specific 16TB Mirror pair (which only a one of each is currently active). Thus moving the data to the other drives (any, it doesnt matted to me where) in the array which also includes the new 18TB mirror pairs.

I don't think that would work using the CLI, and could leave the pool in an unexpected status. It should work using the CLI, but then I would expect it would take a similar amount of time moving the data to the new vdevs.

Sounds good, I’ll leave it for now.

Is there a command that just shows resilver information? As in, feedback on if it’s running parallel etc. versus what I ran that restarted it?

Not that I'm aware, the way I see it is if it's only writing to one disk or multiple.

Gotcha. Yea, before I ran that command, I'm positive that I saw read/writes working on multiple pairs.

Update for now + plus main-tab pdf attached again.

server_Main-2.pdf

Only this vdev is showing normal writes, are they still showing in parallel sometimes?

I just checked again. And you’re right only that one pair is active currently, I’ll check it a couple times in the next hour to see if there is any further activity.

Edited June 4Jun 4 by shelfactor

@JorgeB Sorry I am out of town this weekend, so my responses are a little slower than usual. I am watching the process right now, and there seems to be a lot of reads, and occasional writes. But nothing perfectly symmetrical like the last example we just looked at. Here is the current status, and then 3x PDFs of the drives read/writes. (me trying to capture the read/writes in varying states over the last few minutes.

server_Main-5.pdf server_Main-4.pdf server_Main-3.pdf

That behavior seems strange to me, but it's such a large pool, so it may be different than I expect.

I've also done some tests, and sometimes when resilvering multiple devices, it doesn't do any parallel resilver; other times, it does parallel for all; sometimes only a couple of the devices are resilvered in parallel, but not all, still I don't know why the difference; investigating.

In any case, my recommendation for now would be to just let it finish.

That is definitely very curious. I would be interested, if you do find out something later on (and if you still think back to this thread).

But as always, thank you for all the help! Just sent you some well deserved beer money!

@JorgeB Small update today when checking.. Attached you can see the parallel symmetrical read/writes.

edit: Adding second PDF, showing excellent speeds on the read/writes. This is similar to past experiences, once long ago when you and I discussed how ZFS will "scrub" the whole pool during a resilver. And once just before the recent command run... Meaning I think the resilver will start progressing faster as more of the pool as been read.

***** This does remind me, that long ago when we discussed the full pool scrub, We had the question: "If ZFS scrubs the whole pool during this resilver, and it completes... Then If I were to swap in another drive (starting a new resilver) would it then require scrubbing the whool pool again, or would it retain the recent scan information.

server_Main-6.pdf

Edited June 6Jun 6 by shelfactor

I think newer OpenZFS releases should no longer scrub the full pool during a mirror resilver, and I cannot replicate that unless it's related to the number of vdevs you have in the pool, which would be weird.

In my testing, the disks being re-silvered are constantly written at their approximate max speed, and the other disks have small reads (only in the KB/s), which I assume are metadata related during the scrub, but no data appears to be scrubbed outside the disks being resilvered. The highlights of the devices are the ones being resilvered.

Also still haven't found why sometimes it starts a parallel resolver, other times not.

I did find why it can show the "cant_open" errors for some devices. In the example above, you can see it's showing device 14 as "offline/cant_open", it can happen if the disk being resilvered has the same /dev/sdX as a disk that was previously being used on the pool, and also being resilvered (likely only if the previous disk is listed before the new one in the zpool status output)

This is just a display issue, and will correct itself once the resilver finishes, but now that I know what causes it, I'll create a bug report.

I assume your test pool is also set to mirrored pairs? As you have probably seen my pool has 72 mirrored pairs, but I did run into this 'full-scrub'-resilver prior with around half of the number of vdevs.

These drives are installed on DS4246 Disk shelves, most of the pool is made up of Exos 18TB drives, in both x18 and x20 series. The drives that I am replacing are Exos x18 16TB drives, being replaced with x18 18TB drives. These drives are rated for read/write ~250mbps..

however I think that the lower speed for the resilver is related to the backplane in the DiskShelves. Earlier I updated firmware across all of the shelves, but I believe that since the drives are resilvering on the same backplane .... They are maxing out the backplane speed, not the drive speeds...

This has me wondering, if I turn off my server... Then swap the target drives with drives from another diskshelf, (effectively moving the 12 pairs across the 6 disk shelves, so that only 2 pairs are resilvering per shelf).. I wonder if I could get some faster speeds.

Question 1: Can I stop the server, and when it comes back online and the array starts... the resilvering resumes correct? NOT a restart?

Question 2: When reading the status of the pool, it appears to me that only 431T of the full 1PB pool has been 'read/scanned/scrubbed' ... is that correct, or is the 'issued' number?

I ask, as I am looking at the MAIN tab (updated attachment), the resilver/copy numbers we are talking about above, and ONLY that data transfer is occurring. Meaning Drive<>Drive copy to replace the mirrrors. I don't see any other activity (of decent speed) for any other drive. Note, I have basically all my services turned off, so there should be no data movement other than the resilver. BUT If the full pool 1PB of data hasnt been scanned yet, at what point does it start processing the rest of the data?

server_Main-8.pdf

Edited June 7Jun 7 by shelfactor

18 hours ago, shelfactor said:

I assume your test pool is also set to mirrored pairs?

Yep, 8 mirrored vdevs.

18 hours ago, shelfactor said:

Can I stop the server, and when it comes back online and the array starts... the resilvering resumes correct? NOT a restart?

Should resume; it can go back just a little, but it should not restart.

19 hours ago, shelfactor said:

it appears to me that only 431T of the full 1PB pool has been 'read/scanned/scrubbed'

This is the scanned metadata, and the value should keep increasing as the resilver progresses, and it typically reaches the full capacity (1 TB in this case) before the pool is fully resilvered.

@JorgeB for records-sake. I was able to stop the array/pool move a bunch of drives around to isolate the resilvering pairs (I may not have done it to the best isolation, but decent enough)... I can confirm:

1) The resilver picked right back up

2) The resilver copy speed has increased by almost double

3) I got caught up in why the drives appeared in the same place on the MAIN tab, but realized that's just how unraid sees the slots, it has no care where they are plugged into as long as the system can see the drives as they were before.

4) Google Fiber is amazing (this is just a fun extra note, as I got 8g/8g fiber installed today!)

Will be continuing to wait the resilver for now.

REPEAT QUESTION for future clarification:

-- When upgrading mirror pairs with larger drives, is it better to manually remove the pairs from the pool OR perform the two-stage process that I am doing, which will call for 2x scan/scrub of the pool.

server_Main-9.pdf

That looks much better, multiple disks being written to in parallel with a decent speed.

Keep an eye on reads from other disks that are not part of the resilver, and I mean high-speed reads, in the MB/s, not KB/s, because I don't 'think the whole pool is being scrubbed, at least it's not for me in a similar scenario.

It's normal to "scan" the whole pool, but that should only be a metadata scan, not a full data scrub.

@JorgeB Maybe at this point I should create a new topic to discuss this further, but I am still a bit confused on the tasks to rebuild mirrored sets.

My 'resilver' has now slowed down tremendously.

You can see the speeds of scanned and issued are much slower. I am getting much more random reads compared to writes.

** I read this article about why rebuilds could be slow (I didn't attempt anything from it, just read it). And I'm somewhat perplexed by how my pool reacts during this resilver.

Note, this server is mainly a media server, so not a lot of 'hard-hitting' applications, in fact none really at all.

-- However, right now, if I start downloading new content, the resilver will slow even further. I would expect that if a file wanting to be accessed (such as a movie that is located on one of the degraded drives) would become slow due to the drive being access.

I am wondering if some of the tactics shown in that article can be used to identify why this process is taking soo much time. I understand that ZFS has the 'catalog' or whatever it's called that it refers to which files are where. But my pool is made up of Mirrored pairs.. With very static content, that are fairly large files. I can't conceive why a resilver takes 2-3 weeks, to replace 12 pairs.

12 x 18TB drives = 216 TB of data to be restored (theoretically on 16TB as that was the source size).

I am confused on the issued amount during this process.

Article: https://cr0x.net/en/zfs-resilver-rebuild-speedup/

server_Main-13.pdf

Very odd and sporadic read/writes now, see attached

server_Main-14.pdf

Let's first see if the io looks normal or if there's one or more disks slowing the resilver:

zpool iostat -vyl 10 1 > /boot/1.txt

Repeat that command a couple more times (wait a minute or so between runs), changing the destination file name only, e.g., 2.txt and 3.txt, then attach all 3 files here (they will be saved to the flash drive)

Just getting to this. Current state and attached PDF; as well as a few outputs from the above command.

Only 1 or 2 or the mirror pairs are moving data... The expected finish time is now climbing.

See page 2 of this thread to see another result

3.txt 2.txt 1.txt

Edited June 12Jun 12 by shelfactor