I am attempting to finish upgrading 2 cache pool drives.  I had dual NVMA M.2 480GB ADATA drives in RAID1 for my cache pool.  I was running out of space and wanted to upgrade them both to 1TB drives.  I started following these directions from the FAQ thread:

 

The first drive seemed to upgrade fine, but every time I attempt to replicate the instructions for upgrading the second drive it gives me errors about a missing disk.  

 

I have attached Diagnostics from the current status which is with 1 drive upgraded to 1TB, and the 2nd drive still being the 480GB.  I am hoping someone can take a look and let me know if it seems to be in the correct state for me to replicate the instructions to replace the second drive.  Once I attempt it I can take additional diagnostics if the operation fails.

 

Thank you!

 

milkyway-diagnostics-20250514-1251.zip

Post the output from

btrfs fi show

 

Thank you for helping out!

 

Edited May 15, 20251 yr by Rhuarc

on main click on the first device for that pool and then "remove pool"
back on main, create a new pool with the same name and 1 slot
assign only nvme0n1 device, leave the filesystem set to auto
start the array to import the pool degraded, if it mounts, run a scrub and post the results, since I'm seeing some corruption errors for that pool

Is this going to delete the stuff on the pool, or should I be fine?

 

Theoretically both devices should have all of the files from the pool, correct?

 

Here are the results of the scrub with only the 480GB drive, no errors found.  Also grabbed an update Diagnostics incase that helps.

 

 

Edit:  I did just get this error in the log.  No idea if this applies or is something completely different.

milkyway-diagnostics-20250515-0021.zip

Edited May 15, 20251 yr by Rhuarc

45 minutes ago, Rhuarc said:

No idea if this applies or is something completely different.

That is the docker image, you can recreate it later, for now, stop the array, assign the other pool device to replace the missing device, and start the array, once it's done balancing post new diags.

This is what I got when I assigned the 1TB that had been in the pool to the second slot in the pool.

 

It will not let me start the Array with the second (1TB) device in slot 2 of the pool.

My bad, we have to remove the missing device and reimport the pool, unassign that device, start the array, then type:

 

btrfs dev remove missing /mnt/cache

 

Then stop the array and reimport the pool again, same steps as above, but it should now reimport as a single device pool, then add the other device.

 

 

I think I understand, but are you saying basically to stop the array, remove the second device, reimport the pool, then...  actually I'm not exactly sure what I am supposed to do.  I don't want to screw this up.  I apologize for the confusion!

 

I tried just removing the second device and starting the array and it says too many missing or unassigned devices.  I am guessing maybe I have to drop teh number of devices on the pool back down to 1 for now?

Edited May 15, 20251 yr by Rhuarc

I believe I followed the instructions correctly, and I seemed to get a lot of errors while it was adding the second larger drive.  I have included diagnostics and a screenshot of the cache pool info.  I think it is running the balance now, but I am concerned with all of the errors that I saw when it added the second drive.

 

Edit: These errors continue to show up in the logs as it runs the balancing operation I think.  The number of write errors under pool device stats continues to go up as well.

milkyway-diagnostics-20250515-0206.zip

Edited May 15, 20251 yr by Rhuarc

The new NVMe device dropped offline, is this a new device?

 

May 15 02:01:35 MILKYWAY kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
May 15 02:01:35 MILKYWAY kernel: nvme nvme1: Abort status: 0x371
### [PREVIOUS LINE REPEATED 2 TIMES] ###
May 15 02:03:43 MILKYWAY kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1
May 15 02:03:43 MILKYWAY kernel: nvme nvme1: Disabling device after reset failure: -19

 

You can reboot and try again.

I had gotten 2 new NVME drives, I thought maybe this drive was bad so I am trying the other one, but now I am having trouble re doing the steps to get it to import the original 480GB as a single cache.  This is what I have now,  then I am trying to run the command.

 

This is what I get now trying to run the command you gave me.

 

Edit: Latest Diagnostics attached

milkyway-diagnostics-20250515-0337.zip

Edited May 15, 20251 yr by Rhuarc

Diags are missing some data, post the output from

 

btrfs fi usage -T /mnt/cache

 

This is with the 1 drive showing as missing.

Thank you for all of your help!

 

With the array still running type:

 

btrfs balance start -dconvert=single,soft -mconvert=single /mnt/cache

 

When that's done, you can remove the missing device with 

 

btrfs dev remove missing /mnt/cache

 

And then reimport the pool again.

Putting in the first command this is what it says, and this showed up in the logs.  Do I want to force it by adding something?

 

11 hours ago, Rhuarc said:

Do I want to force it by adding something?

Yes, forgot about that, use:

 

btrfs balance start -f -dconvert=single,soft -mconvert=single /mnt/cache

 

Ok, that worked!  So here is the current state.  The array is started and only the one cache drive is showing up.

 

Do I now go through the original process of adding the new drive and letting it go through the balance process and then hopefully adding in the second new drive to replace the current one?

 

Edit: Added a current diagnostics.

milkyway-diagnostics-20250516-0017.zip

Edited May 16, 20251 yr by Rhuarc

1 hour ago, Rhuarc said:

Do I now go through the original process of adding the new drive and letting it go through the balance process and then hopefully adding in the second new drive to replace the current one?

Correct.

I did this and seem to be getting a lot of write errors in the logs.  It looks like the btrfs process is running on the UI, but tons of errors are going through the logs.  The drive is a brand new drive so I highly doubt it is an issue with the new one.  Although last time I tried this the btrfs process did not really run, so maybe this is what is supposed to happen?

milkyway-diagnostics-20250516-0137.zip

Edited May 16, 20251 yr by Rhuarc

The process finished, the stop button is now available.  There were a ton of errors in the logs.  Any way to confirm that the cache pool is correct now and I can start replacing the second drive?

 

Edit:  This is what is currently showing on the cache pool status.

milkyway-diagnostics-20250516-0211.zip

Edited May 16, 20251 yr by Rhuarc

Problems again with the new NVMe device:

 

May 16 01:32:52 MILKYWAY kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
May 16 01:32:52 MILKYWAY kernel: nvme nvme0: Abort status: 0x371
### [PREVIOUS LINE REPEATED 2 TIMES] ###
May 16 01:35:00 MILKYWAY kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
May 16 01:35:00 MILKYWAY kernel: nvme nvme0: Disabling device after reset failure: -19

 

Is this an error that I will need to keep trying to reimport the new drive, or will restarting the server potentially fix it?  I do have a second new drive that I can install, but I am guessing that I will have to do this whole process all over again with the other new drive?

7 minutes ago, Rhuarc said:

I do have a second new drive that I can install, but I am guessing that I will have to do this whole process all over again with the other new drive?

Most likely, but reboot and post new diags first.