I just had this error pop up today, Device 1 (sdd) on Cache pool (Edit: Raid0 it's RAID1, copy, I mistyped that) is not working. It was initially throwing Errors with every write, so I shut down the server and changed the SATA cable. Now I restarted and it has no reads/writes.

Uploaded diagnostics. Is that sdd drive (HFS500) toast? What else can I try?

Thanks in advance!

untower-diagnostics-20261010-2207.zip

Edited October 11Oct 11 by mattz
mistyped raid0, it was a raid1 / full copy setup

The first device (the HFS500) failed. SMART reports it as failing, and it also lost its partition, so it's no longer part of the pool. The pool is running degraded using only the ADATA NVMe. All data is still there, because with raid1 the NVMe has a full copy.

Important: the ADATA SMART also shows it's failing. It's worn out (140% of its rated life used, critical warning set). Before anything else, back up anything important from the pool, for example to the array.

Then you can fix the pool. With the array running, type:

btrfs balance start -f -dconvert=single,soft -mconvert=dup,soft /mnt/cache

when that's done type:

btrfs device remove missing /mnt/cache

Then stop the array and re-import the pool with the remaining device:

- on Main click on the first device for that pool and then "remove pool"

- back on Main, create a new pool with the same name and 1 slot

- assign the ADATA NVMe, leave the filesystem set to auto

- start the array to import the pool

The pool should be imported as a single-device pool. Don't re-use the HFS500. To get redundancy back, stop the array, change the slots to 2 and add a new device to create a mirror. Since the ADATA is also worn out, plan to replace it soon too.

Thank you, @JorgeB . Good points. Those drives are ancient, in SSD terms. I will get a new drive and get back with my results.

9 hours ago, JorgeB said:

Then you can fix the pool. With the array running, type:

btrfs balance start -f -dconvert=single,soft -mconvert=dup,soft /mnt/cache

when that's done type:

btrfs device remove missing /mnt/cache

How long will it take for the first command to run?? I left it for over 30-minutes, and it just sat at the command line. I finally canceled it with Ctr+C and the command line recovered, but I didn't see any changes.

It should take a few minutes; 30 seems a lot. Please post new diags assuming you didn't yet reboot.

OK, @JorgeB - I decided to grab two new 1 TB Evo 870 SSDs. How do I get the cache back on the right track?

1. My SATA slots were full, but removing the 1 broken drive, I can swap it out for 1 new 1 TB drive.

2. I then need to transfer all the data on the existing/old cache drive to the new 1 TB drive

3. Then I need to decommission the existing/old cache drive

4. Then install the second new 1 TB drive

5. Reestablish the RAID1 (mirrored) cache with both drives for 1 TB of storage

Does that cover it? What steps do I need to take to get there. Do I start again with the balance command above?

Really appreciate the help, because I feel like I'm flying close to the sun with the 1 old SSD and no mirror. And if this is a standard swap operation, I'm glad to RTFM if you would point me to the correct scenario. Thank you!

Yes, that's the right idea, and you do start with the balance again. Sorry, the "few minutes" I gave you was too low. With this much data it can take considerebly more time, and it shows nothing while running, so it was likely working when you cancelled it. Nothing was lost, and it will skip whatever it already converted.

One thing to check first: the ADATA is an NVMe device, so removing it won't free a SATA port. You'll need a port for each 870 EVO while the ADATA is still installed. Your October diagnostics show only 5 of the 8 onboard SATA ports in use, so you may already have free ones, unless you've added disks since or the board disables some when M.2 slots are used. If you can't connect both, let me know before you start.

1. Back up anything important from the pool to the array. The ADATA is also worn out, and the balance rewrites its data.

2. With the array running, start the balance in the background:

btrfs balance start -f --bg -dconvert=single,soft -mconvert=dup,soft /mnt/cache

Check progress with this command, it's done when it reports no balance found:

btrfs balance status /mnt/cache

3. Remove the missing device:

btrfs device remove missing /mnt/cache

4. Stop the array and re-import the pool with the ADATA only:

- on Main, click on the first device for that pool and then "remove pool"

- back on Main, create a new pool with the same name and 1 slot

- assign the ADATA, leave the filesystem set to auto

- start the array and confirm your data is there

5. Stop the array and shut down. Replace the HFS500 with the first 870 EVO, connect the second one too, and boot.

6. With the array stopped, change the pool to 2 slots and assign the first 870 EVO. Start the array. Unraid adds it and converts the pool to RAID1 automatically. Wait until this finishes, it can take a few hours. You can check by clicking on the pool device on Main, the balance status should show only RAID1.

7. Stop the array, assign the second 870 EVO to the ADATA's slot, and start the array. Unraid replaces the ADATA with it. Wait for this to finish, then the ADATA can be removed.

The result will be a RAID1 pool on the two 870 EVOs with about 1 TB usable. Post new diagnostics when it's done so we can confirm all is good.

If any step gives an error, or the pool shows as unmountable or offers to format, stop there and post diagnostics. Don't format anything.

Awesome! I'll get on this and report back. Great info. TY 🙏

25 minutes ago, JorgeB said:

ADATA is an NVMe device

After I posted, I re-read your post and did a full mouth-open double-take. That didn't even register in my mind when I bought the new SSDs. You're right, I should be able to find some SATA slots for that, and I would be fine with that. I added nothing new, and even though the NVME eats up a SATA slot, especially if I remove it I'll have those available again.

20 hours ago, JorgeB said:

btrfs balance start -f --bg -dconvert=single,soft -mconvert=dup,soft /mnt/cache

I got this message when I ran the btrfs balance:

# btrfs balance start -f --bg -dconvert=single,soft -mconvert=dup,soft /mnt/cache
WARNING:

        Conversion with missing device(s) can be dangerous.
        Please use 'btrfs replace' or 'btrfs device remove' instead.
        Safety timeout skipped due to --force

it looks like it did not run because it went right back to the prompt after that warning. You have the Remove command at 3. Attached the latest diagnostic output. I will await your help on this.

untower-diagnostics-20261106-0707.zip

---

Oh, EDIT: Went to the Cache page and I see the Balance is running and 92% running. It must have kicked off and started. I'll continue

Edited 5 hours ago5 hr by mattz
balance is running edit

That warning is normal, but it should still run.

Ok, I also checked, and I don't think I have enough SATA slots to do it in one go.

I have 6 SATA slots total and 5 available since 1 of them is used by the M.2 NVME.

• I'm not sure if that M2_2 slot the ADATA_480GB working cache drive or the ADATA_1TB unassigned drive I use as a VM system disks.

So, I am going to have to First add 1 of the 1TB 870s in, do the cache transfer from the ADATA_480GB, then I can turn off and remove the ADATA_480GB and install the second 1TB 870.

How does that change the steps here?

At the moment I am at Step 4, stopping the array (which is taking forever), but I have done the balance and removed the missing /mnt/cache drive.untower-diagnostics-20261106-0901.zip

Edited 2 hours ago2 hr by mattz
added second image from mobo manual

Good news, the balance finished without errors and the missing device is gone. The new diagnostics show the pool as a single device on the ADATA, as expected, so that part is done.

The array stop is slow because it can't unmount /mnt/user, and that looks like your backup_cache user script. It started last night and its rsync was still running when you took the diagnostics. The stop will finish by itself once rsync ends. Check its log in User Scripts. Since that's your backup of a worn device, it's best to let it complete. If you need to stop it sooner, use Abort Script there, but then the backup will be incomplete. Don't force a reboot.

About the SATA ports, the steps don't change, and sorry, I was wrong about 8 ports, your board has 6. The manual note only applies to a SATA M.2 SSD in M2_2. Both of your ADATA drives are NVMe, so SATA3 is not disabled and you have all 6 ports. It doesn't matter which M.2 slot the ADATA is in. You're using 5 now, the HFS500 and 4 array disks, so one is free. The first 870 EVO can use the HFS500's port and the second one the free port, with the ADATA still installed.

So continue with step 4 once the array stops:

- Step 5: when you shut down, replace the HFS500 with the first 870 EVO and connect the second 870 EVO to the free port. Check that both are detected.

- Step 6: add the first 870 EVO and wait for the conversion to RAID1 to finish.

- Step 7: assign the second 870 EVO to the ADATA's slot and wait for the replace to finish. Then shut down and remove the ADATA.

Keep Docker and VMs disabled until this is all done. If the second 870 EVO isn't detected, or anything gives an error, stop there and post diagnostics. Don't format anything.

There's something weird going on with the /mnt/cache and subsequently the rsync backup. It looks like the /mnt/cache/docker has some sort of recursiveness to it, because it is far too large. The cache only had about 301 GB USED, but the docker folder is 686G.

If I have the appdata stuff, should I just cancel it and move on to the next steps?

139G    /mnt/cache/appdata
686G    /mnt/cache/docker
11G     /mnt/cache/domains
314M    /mnt/cache/photos
1.0G    /mnt/cache/system
0       /mnt/cache/tm-media

That's not recursion, and nothing is wrong. Your pool only has 280 GB used, so 686G can't really be there. Your Docker data is about 125 GB. With a Docker folder on Btrfs, the image layers share the same data, but du counts it again for each layer. rsync does the same and copies each one in full, which is why the backup is taking so long. This has no effect on the pool steps, since they work with the real data, and it will easily fit on the 870 EVOs.

So yes, abort the script. The array stop should then finish, and images and containers don't need a backup, because you can recreate them from Previous Apps. However, since rsync was stuck in docker, it may not have reached the folders after it, and any Docker named volumes are stored in that folder, not in appdata. Your compose stacks may use them, for example the superset database.

1. Use Abort Script in User Scripts and let the array stop.

2. Finish step 4. When the array is started and you've confirmed your data is there, keep Docker and VMs disabled and run this:

rsync -a /mnt/cache/domains /mnt/cache/photos /mnt/cache/system /mnt/cache/tm-media /mnt/cache/docker/volumes /mnt/user/backup_arrayonly/

This creates a folder for each one inside backup_arrayonly. Use that share because it's array only, so the copy doesn't end up on the ADATA.

3. When that finishes, continue with step 5.

You can delete the partial docker copy from the backup share later to get the space back, but only from the backup share. Don't delete anything from /mnt/cache/docker. Before the next backup run, exclude the docker folder from the script, but keep docker/volumes if it has anything you need.

Once the pool is on the 870 EVOs and everything works, you may want to consider moving Docker to a vdisk (docker.img). It's a single file, so du shows its real size and this problem goes away. You'd need to recreate the containers, so that's optional and can wait.

If the array still doesn't stop a few minutes after aborting the script, or if anything gives an error, stop there and post diagnostics. Don't force a reboot.

Ok, 30 min later it's still stopping. Anything else I should close?

untower-diagnostics-20261106-1044.zip

Yes, rsync is still running. Abort Script only stopped the script itself, not the rsync it started, and that's what is keeping the array from stopping. Sorry, I should have included this in my last post.

Open a terminal (SSH or the web terminal) and type:

pkill -x rsync

Then check that it's gone. This command should show nothing:

pgrep -a rsync

The array should stop within a few seconds after that. Then continue with step 4 and the rest of the plan from my last post, nothing changes.

If the array is still stopping a couple of minutes after rsync is gone, type this and post the output with new diagnostics:

fuser -vm /mnt/user

Don't force a reboot.

Unrelated, but worth fixing once this is done: your server can't send email notifications, because Gmail has been rejecting its login since Nov 4. You'll want those alerts working, so check the SMTP settings in Settings > Notifications. Gmail normally needs an app password there.