Unmountable: Invalid pool config

[Fiala06]

One of my cache drives (2 total) is failing. So I went to replace it now I'm getting 

"Unmountable: Invalid pool config"

 

I've tried setting the failed disk to not installed, start and stop the array but doesn't seem to matter what I do I can't get this to start again. 

 

Any ideas?

unraid-diagnostics-20230320-1639.zip

[Fiala06]

So if I remove the cache pool and create a new one with a single drive, it works fine. As soon as I add a new 2nd drive I get the same error. Even tried formatting the new ssd before adding it to the cache. 

[JorgeB]

Post the output of

btrfs fi show

 

[Fiala06]

4 hours ago, JorgeB said:

Post the output of

btrfs fi show

 

 

root@UNRAID:~# btrfs fi show
Label: none  uuid: 8259fa8c-4b09-4304-aed8-6fe58d49323c
        Total devices 2 FS bytes used 581.62GiB
        devid    1 size 931.51GiB used 589.03GiB path /dev/nvme0n1p1
        devid    4 size 0 used 0 path /dev/sdq1 MISSING

Label: none  uuid: cd6041b9-7b7e-4203-8bd5-38bc9167f1e4
        Total devices 2 FS bytes used 4.11GiB
        devid    1 size 465.76GiB used 7.03GiB path /dev/sdt1
        devid    3 size 931.51GiB used 7.03GiB path /dev/sdh1

Label: none  uuid: c854bdcc-ee55-4a4f-bd72-a29f5222a437
        Total devices 2 FS bytes used 834.26GiB
        devid    1 size 931.51GiB used 839.03GiB path /dev/sdm1
        devid    3 size 1.82TiB used 839.03GiB path /dev/sdy1

Label: none  uuid: 3e52a2e4-0036-41a0-bdd7-4e133ef6acb1
        Total devices 1 FS bytes used 22.38GiB
        devid    1 size 80.00GiB used 26.02GiB path /dev/loop2

the nvme011p1 is the drive was replacing. The other drive on the cache pool I did replace about a week ago also which is sdq

Here you can see the cache pool is working with the single disk (sdq)

Edited March 21, 2023 by Fiala06

[JorgeB]

Mar 20 16:26:15 UNRAID kernel: BTRFS: device fsid 8259fa8c-4b09-4304-aed8-6fe58d49323c devid 1 transid 65222504 /dev/nvme0n1p1 scanned by udevd (1051)
Mar 20 16:26:15 UNRAID kernel: BTRFS: device fsid 8259fa8c-4b09-4304-aed8-6fe58d49323c devid 4 transid 65213478 /dev/sdq1 scanned by udevd (1051)

 

Not sure what happened here exactly but they are out of sync, note the different trasnids, the NVMe device has a more recent fileystem, also I'm not quite following this:

 

41 minutes ago, Fiala06 said:

the nvme011p1 is the drive was replacing. The other drive on the cache pool I did replace about a week ago also which is sdq

Which devices were part of the original pool and in what order did you attempt to replace them? Pool is kind of a mess at the momento, it might be easier to just backup current data and re-format.

[Fiala06]

This was the original pool:


I replaced the 870 EVO (sds) with the CT1000MX500SSD1 (sdq) about a week ago. Everything went well. 

Fast forward, 2 days ago the nvmeOn1 started throwing all sorts of errors (tested and is failing). Ordered a new drive, attempted to put it in yesterday and here we are.

[JorgeB]

Any idea then why sdq is devid 4? A two member pool with have devices 1 and 2, and the replacements would inherit the same ids, 1 and 2.

[Fiala06]

Maybe from me trying to get them to work, starting and stopping the array and moving them around? I'm really not sure. 

Edit: Since it's working now with a single cache drive, could I just do a new config? Then add the 2nd cache? Would that reset the ids?

Edited March 21, 2023 by Fiala06

[JorgeB]

No, but if you wipe the other device first (with for example blkdiscard) it should work.

[Fiala06]

5 minutes ago, JorgeB said:

No, but if you wipe the other device first (with for example blkdiscard) it should work.

I've never used that before so  blkdiscard /dev/nvme0n1? Since that's the drive I will no longer be using?

 

[JorgeB]

Add -f to force since there will be a filesystem there:

 

 

blkdiscard -f /dev/nvme0n1

 

[JorgeB]

Also and before doing anything make sure backups are up to date, just in case.

[Fiala06]

3 hours ago, JorgeB said:

Also and before doing anything make sure backups are up to date, just in case.

Just ran it after backing everything up

 

root@UNRAID:~# blkdiscard -f /dev/nvme0n1
blkdiscard: Operation forced, data will be lost!
root@UNRAID:~# ls -lh^C
root@UNRAID:~# btrfs fi show
Label: none  uuid: 8259fa8c-4b09-4304-aed8-6fe58d49323c
        Total devices 2 FS bytes used 583.02GiB
        devid    1 size 931.51GiB used 593.03GiB path /dev/nvme0n1p1
        devid    4 size 0 used 0 path /dev/sdq1 MISSING

Label: none  uuid: cd6041b9-7b7e-4203-8bd5-38bc9167f1e4
        Total devices 2 FS bytes used 4.11GiB
        devid    1 size 465.76GiB used 6.03GiB path /dev/sdt1
        devid    3 size 931.51GiB used 6.03GiB path /dev/sdh1

Label: none  uuid: c854bdcc-ee55-4a4f-bd72-a29f5222a437
        Total devices 2 FS bytes used 834.35GiB
        devid    1 size 931.51GiB used 839.03GiB path /dev/sdm1
        devid    3 size 1.82TiB used 839.03GiB path /dev/sdy1

Label: none  uuid: 3e52a2e4-0036-41a0-bdd7-4e133ef6acb1
        Total devices 1 FS bytes used 22.14GiB
        devid    1 size 80.00GiB used 26.02GiB path /dev/loop2

 

[JorgeB]

It's still showing that fs, and possibly that one was the one getting mounted, reboot and post that output again.

[Fiala06]

Rebooted and now the array didn't auto start. 

[JorgeB]

Start without any pool device assigned, need to check "I want to do this" next to start button, then stop array, re-assign pool device and start array.

[Fiala06]

Well they both now say

Unmountable: Wrong or no file system

[JorgeB]

I think you should just reformat and restore from the backup, something weird happened there.

[Fiala06]

6 hours ago, JorgeB said:

I think you should just reformat and restore from the backup, something weird happened there.

Thanks again for all your help and quick replies! 

I managed to format both new drives and just finished restoring everything.