Diagnostics attached.

Space 44 is the issue, which is part of Mirror Pair 21

I have a drive that is failing, which is causing mirror pair-21 to not complete.

I can start/stop the array.

Is the correct path to replacing it, to set the slot to "no device", start the array and check "Start will remove the missing pool disk and then bring the array on-line."

Then, once it's started, stop the array, and then add in the new disk?

-- Thus far, if I stop the array, and try to add a new disk, the new disk just stays unassigned once the array starts.

nucserver-diagnostics-20250706-2152.zip

Edited May 1May 1 by shelfactor
Solved

There's a single device striped into the pool:

You need to remove it manually, if don't know how to, I can post more details

@JorgeB That is the single 'working' device left from the mirror vdev.

Space 43 and 44 were part of mirror-21.

I need to replace Space 44, so that it can rebuild mirror-21

Thanks for the reply.

Edited July 7, 2025Jul 7 by shelfactor

That wasn't part of a mirror, but looked like a single striped device, post the full output from zpool status (or zpool import if the poos is exported)

That drive was definitely part of a mirror pair vdev prior (less than 24 hours ago). Here is the zpool import (array not started for safety right now) sorry for the split images.

I ran into a drive issue where the sector count started going up quickly, before I could stop the array and replace it, I started having read issues. When I rebooted the server, I had the 'no file system warning'. For which I had made a post, but since deleted it. After a second reboot the array started successfully, just now not reflecting mirror-21.

@JorgeB I'm attaching the 3 diagnostics (prior to the one attached above) that I had pulled during this error phase, I'm sorry I dont have exact notes where in the reboot process they were.

My thoughts right now:

- pool still says 35 vdevs of 2 devices

- pool/array will start and mount correctly (as you believe it sees the drive as a single device

nucserver-diagnostics-20250706-2126.zipnucserver-diagnostics-20250706-2026.zipnucserver-diagnostics-20250706-2004.zip

If it was a mirror, it's not one now, it's a single device vdev, you can try adding a mirror manually, the GUI won't support that:

zpool attach space sde1 /dev/sdX1

Replace X with the correct device letter, it needs to be partitioned first, you can use UD

Once that is added, you can reimport the pool using the GUI:

on main click on the first device for that pool and then "remove pool"

back on main, create a new pool with the same name and number of slots

assign the pool device(s), leave the filesystem set to auto

start the array to import the pool

Very interesting, I'm curious what caused that change, other than the drive failing.

1) Would it be smarter to remove that single drive, (is that even possible?) and then re-added another mirror pair?

2) Does your above suggestion retain data on the 'single' device? (mirroring to the newly added device?

3) If your current suggestion is still the way to go, and it doesn't lose the data on 'single' device. I would do:

single device currently is: sde1 (space 43)
zpool attach space sde1 /dev/sdbd1

This is just for example, showing the available drives, and the one that I will want to be in the new pair:

** Sent over some $$ Beer Money!!

Edited July 7, 2025Jul 7 by shelfactor

31 minutes ago, shelfactor said:

1) Would it be smarter to remove that single drive, (is that even possible?) and then re-added another mirror pair?

That should also work, since there are no raidz vdevs, any data on that vdev would be moved to other ones, assuming there's enough space

zpool remove space sde1

Then you would also need to reimport the pool, same as described above

32 minutes ago, shelfactor said:

2) Does your above suggestion retain data on the 'single' device? (mirroring to the newly added device?

It does retain the data, it just adds another device to it making it a mirror.

33 minutes ago, shelfactor said:

zpool attach space sde1 /dev/sdbd1

Correct, assuming sdbd is already partitioned

34 minutes ago, shelfactor said:

** Sent over some $$ Beer Money!!

Thanks!

@JorgeB You're the best.

Out of an abundance of caution, I'd like to clarify the next steps.

- Currently the array is stopped, and I can not perform the command mentioned above. (As I assumed, it would need to be started)

Do I start the pool, with the missing drive, by clicking to remove the missing drive?

Edited July 7, 2025Jul 7 by shelfactor

20 minutes ago, shelfactor said:

Out of an abundance of caution, I'd like to clarify the next steps.

Sorry, forgot to mention that the pool must be imported first:

zpool import space

run the attach command, then

zpool export space

know you can reimport using the GUI

@JorgeB

*Again note, the pool is shown in the GUI, (all drives filled, except the missing one). Array stopped.

I assume this is because 'space' is present in the GUI still?

IF this is the case do I got back and do this:

39 minutes ago, shelfactor said:

Do I start the pool, with the missing drive, by clicking to remove the missing drive?

1 minute ago, shelfactor said:

I assume this is because 'space' is present in the GUI still?

Nope, that cannot be, possibly there's another previous drive connected that was connected to that pool or another one with the same name, you can copy/paste the ID from zpool import:

zpool import 94611...etc

Then for the attach and zpool export commands, use the name again, id is just needed for the import part when there are multiple device/pools witht he same name.

@JorgeB Understood.

To prepare the new drive in UD, is there a certain way to only partition the drive?

When I go to prepare the new drive (sdbd) it wants me to format it and name the pool for which it will be used.

Some Notes as I have been trying to get to the attach step, I was able to zpool import the pool.

Using Unassigned Devices - if i format with a pool name different than space (in this example i tried 'empty') then it wont attach due to the different name.

If I try to format the drive using 'space' as the name and I get the error that the pool name already exists.

Thanks!

Edited July 8, 2025Jul 8 by shelfactor

12 hours ago, shelfactor said:

When I go to prepare the new drive (sdbd) it wants me to format it and name the pool for which it will be used.

You don't need to format zfs, format it's just to create the partition, you can use xfs for example, for a quicker format, also when using the attach command add -f:

zpool attach -f space sde1 /dev/sdX1

@JorgeB Thank you!

I was getting stuck trying to use the same format (ZFS) to partition the drive using UD. This was causing the conflicts and preventing the attach command from working.

Formatting using XFS in UD, doesn't require a pool name. It just partitions, formats and names the drive from the serial number.

After the attach command completed, I immediately saw the resilvering started on the new drive, as part of mirror-21.

-- Is it ok to export the pool and re-import via the GUI while the resilvering has already started? or should I wait for it to complete?

Thanks Again @JorgeB , without you, I surely would have messed this up.

Edited July 8, 2025Jul 8 by shelfactor

1 hour ago, shelfactor said:

Is it ok to export the pool and re-import via the GUI while the resilvering has already started? or should I wait for it to complete?

Either way should work, as you prefer.

@JorgeB Everything seems to be moving forward!

However, I do have one curiosity. When I watch the progress, as well as the 'main' tab to see the read/writes on the pool, it appears that the resilvering is reading the whole pool.

I would have thought resilvering one mirror would only require reading/copying the other single drive in the mirror?

Here are progression from yesterday when it started, to now in the morning.

Edited July 9, 2025Jul 9 by shelfactor

I would think that only the mirror would be read, but after a quick search, looks like it's normal:

https://www.truenas.com/community/threads/resilvering-reading-all-disks-in-multi-vdev-mirror-set.61954/

13 minutes ago, JorgeB said:

I would think that only the mirror would be read, but after a quick search, looks like it's normal:

https://www.truenas.com/community/threads/resilvering-reading-all-disks-in-multi-vdev-mirror-set.61954/

@JorgeB Haha, I was on that exact same page! I thought I would check in to get your opinion. Thanks for all the help! I'll update this and mark it solved in the title.

UPDATE/EDIT: I was planning to replace a couple other drives that have now been marked as problematic (other vdevs)... Can/Should I replace those at the same time then?

Considering this is ready ALL data, similar to scrub, but not he same.

If this is possible, can you suggest the command progression that I would follow to replace those drives. I can have all drives online, I have the slot space

Edited July 9, 2025Jul 9 by shelfactor

The GUI allows you to replace multiple drives from different mirrors at the same time (7.1+), but I'm not sure if the resilvering is done in series or parallel.

If you can have the old and new drives connected, you could do a manual zpool replace, that would be safer since the old disks will be kept online during the process (Unraid currently wipes the old disk(s)), but it also means you will you in the end, need to reimport the pool to bring it to sync with the GUI.

@JorgeB

1) I suppose I should let this resilver finish, since it was replacing a missing drive. Then move on to the other drives ->

2) With the manual zpool replace, does the pool need to be removed from the GUI first? Then re-add it to the GUI as you mentioned after the replace?

3) To you knowledge does manual zpool replace work in parallel, as in, can I run multiple replace at the same time?

1 hour ago, shelfactor said:

With the manual zpool replace, does the pool need to be removed from the GUI first?

No, you can leave the array started, GUI will show the pool as it was at start, do the replacements, then stop the array and reimport the pool.

1 hour ago, shelfactor said:

To you knowledge does manual zpool replace work in parallel, as in, can I run multiple replace at the same time?

I believe it would be one at a time, but the server could remain running with the array started and the pool accessible during all.

@JorgeB Currently at about 75% done resilvering on the one drive. It's been interesting watching the pool "scrub" (yes I know it's not a scrub) but read the whole pool to rebuild a single mirror.

After doing a bunch of reading during this week, I now understand a little more about why it reads through all the TXs etc to rebuild the single drive. Mainly to due with verifying the data before copying it back to the drive.

I found this comment blurb and wanted to get your take on it. For this pool, the data is very static media files. Therefore, data integrity isn't a mission critical thing.

I think being able to issue this command, which will replace the drive and copy the data back quickly. I can verify the files during the monthly scrub.

found @ https://serverfault.com/questions/1007438/zfs-scrub-vs-resilver-are-they-equivalent

That may be worth a try, especially since you intend to replace multiple drives. You can do the various replacements , and then do a scrub in the end