I decided to let a major download task occur, and ran the command during that data download.
The drives near the bottom are the most empty of the drives, and not part of the resilver.. But this was just to get another look, with high speed download.

EDIT Adding a few more examples, so really interesting wait times. I'm not sure how to read into it.

4.txt 5.txt 6.txt

Edited June 12Jun 12 by shelfactor

Not seeing anytjong that suggests a disk problem. When there's pool activity, the latency is getting backed up massively inside the asynchronous queues (asyncq_wait up to 1 second) and the rebuild queues (rebuild wait up to 1 second).

Try to avoid any pool activity for a few minutes; also, change apply values:

echo 4 > /sys/module/zfs/parameters/zfs_vdev_rebuild_min_active

echo 12 > /sys/module/zfs/parameters/zfs_vdev_rebuild_max_active

echo 4194304 > /sys/module/zfs/parameters/zfs_rebuild_max_segment

echo 134217728 > /sys/module/zfs/parameters/zfs_rebuild_vdev_limit

Note, these values are not persistent; they will get back to default after a reboot, but use them for now just to see if it makes any difference; let the resilver run for 10 to 15 minutes without any pool activity, ideally, and see if there's any difference.

On 6/13/2026 at 6:39 AM, JorgeB said:

Not seeing anytjong that suggests a disk problem. When there's pool activity, the latency is getting backed up massively inside the asynchronous queues (asyncq_wait up to 1 second) and the rebuild queues (rebuild wait up to 1 second).

Try to avoid any pool activity for a few minutes; also, change apply values:

echo 4 > /sys/module/zfs/parameters/zfs_vdev_rebuild_min_active

echo 12 > /sys/module/zfs/parameters/zfs_vdev_rebuild_max_active

echo 4194304 > /sys/module/zfs/parameters/zfs_rebuild_max_segment

echo 134217728 > /sys/module/zfs/parameters/zfs_rebuild_vdev_limit

Note, these values are not persistent; they will get back to default after a reboot, but use them for now just to see if it makes any difference; let the resilver run for 10 to 15 minutes without any pool activity, ideally, and see if there's any difference.

Got it, I think some of this is what I have learned from that linked article from earlier.. But definitely do not have a strong enough knowledge on how it works. But I'm reading up on it.

I have updated with your commands, will wait a short bit of time and capture some outputs

Thank you so much @JorgeB !!

Current Progress prior to commands:

Edited June 14Jun 14 by shelfactor

For reference, these are the numbers before and after your command changes:



Here are some example outputs. Note, There was some pool activity popping up, but was brief.

Edit: One more as the pool calmed a little

10.txt

7.txt8.txt9.txt

Edited June 14Jun 14 by shelfactor

Read bandwidth increased substantially; the last one shows 743MB/s, but writes still seem low, any improvements in the expected resilver time?

Resilver completion time % has been increasing much faster since running your command changes.

Last night, I had to work on a couple tasks (adding media etc), so I took a few snapshots while performing those tasks > 12.txt

Then just now, I checked it again, and it's back to much calmer performance, and has gained a decent % upgrade > 13.txt

Still chugging along, I still find it quite odd how long this reilver process is; and am trying to decide the best way to perform the next step (replacing the remaining 16TB drives to 18TB).

1) Remove Mirror Pairs and re-add with 2x18TB -- Wondering if this will also help with spreading data across more drives.

2) Perform yet another ~1 month resilver just like this one

3) Replace only 4 or 6 at a time, to remove such a high resilver call on the pool

I am wondering, if after this resilver completes, if performing any of the above options will run faster. Thinking back to an earlier thread of you and I discussing this very topic, after a scrub/resilver > well a subsequent scrub/resilver finish faster?

12.txt13.txt

I think part or most of the performance issues may be related to just how big the pool is, and it's not something I have experience with or can test.

29 minutes ago, shelfactor said:

1) Remove Mirror Pairs and re-add with 2x18TB -- Wondering if this will also help with spreading data across more drives.

This may be worth a shot; I'm not sure if the removal performance will also be affected by the pool size, but I would test with one first to see. It will better redistribute the data to the empty vdevs, and then adding one or more new mirrored vdevs is basically instantaneous.

Just keep in mind that the GUI doesn't support removing a vdev, but you can do it using the CLI, e.g.:

zpool remove space mirror-30

Once one or more mirrors have been removed, you need to reimport the pool.

stop the array

unassign the devices for the removed mirror(s)

Tools - New config - Preserve all assignments - Apply

start the array to reimport the pool

1 hour ago, JorgeB said:

I think part or most of the performance issues may be related to just how big the pool is, and it's not something I have experience with or can test.

Ok, I'll pack up my rack and head your way! We can have a LAN Party and test all sorts of things on 1PB+ pool !!! haha

Quite honestly, I have let the pool grow so that my main share can all be within the same pool (for hardlinking). However, I have started to plan out a new layout, that will separate 'active' media from archived (which should not require hardlinking).

1) Once this resilver completes, I'll try the above command to remove a single mirror pair, as a good test for a large pool.

I assume it's better to leave both drives in for the removal? Or does it not matter?
Thinking about reducing the read/writes off one of the drives if possible.

1 hour ago, JorgeB said:

Just keep in mind that the GUI doesn't support removing a vdev, but you can do it using the CLI, e.g.:

zpool remove space mirror-30

Once one or more mirrors have been removed, you need to reimport the pool.

stop the array

unassign the devices for the removed mirror(s)

Tools - New config - Preserve all assignments - Apply

start the array to reimport the pool

2) If I remember correctly, the 'new config' is only for the array/pool layout, not other settings/apps/plugins etc. correct?

18 minutes ago, shelfactor said:

I assume it's better to leave both drives in for the removal? Or does it not matter?

Yes, and you would need to at least leave one of them connected; if you removed both, the vdev would be unavailable (meaning the whole pool could not be imported)

19 minutes ago, shelfactor said:

2) If I remember correctly, the 'new config' is only for the array/pool layout, not other settings/apps/plugins etc. correct?

Correct, it's only about the assignments, and if you do that in the order mentioned, it will avoid you having to reassign the remaining pool devices, which would be a pain with so many.

SMALL UPDATE: After doing a few tasks, adding media and moving some backups. I noticed that the output of your zpool iostat was giving some interesting info.

Looks like the resilver-ing drives were not being called upon, but other drives were.

Attached are two outputs from when the tasks where touching data on the pool, the last is now when the pool is idle again + PDF output from MAIN tab that matches 17.txt.

15.txt16.txt17.txtserver_Main-19.pdf

ANOTHER SMALL UPDATE: Just to show more ongoing iostat results.

No read/write to the resilver drives, is kinda odd.

18.txt19.txt

It is also odd to me, but I think it may be related to having pool activity with such a large pool, try to avoid new reads and writes as much as possible, and it should again continue the resilvering.

At least I don't see anything that would suggest a slow disk or disks, downgrading the resilver speed.