(SOLVED) How to verify cache pool is working?

[walle]

I have recently bought a new 500 GB SSD in order to create a cache pool with my existing encrypted btrfs 500 GB SSD cache device I installed a while back.

 

What I have basically done is to shut down the server, installed the drive, start the server and lastly followed this guide:

The thing is that I don't think it started to balance the drives when I started the server. I had to trigger a full balance manually and after balancing I'm not sure if the pool is working or not. See attachment for screenshot. As you can see, it still have "new drive" status and have basically no writes to it. I also SSH to the machine and ran "btrfs filesystem show" with the following output:

Label: none  uuid: fd9abfd5-7e13-487f-ba5d-419b90608d6b
        Total devices 1 FS bytes used 263.25GiB
        devid    1 size 465.76GiB used 264.03GiB path /dev/mapper/sdg1

Label: none  uuid: bf870768-3cdb-4f9e-836b-4b1ed2c4c253
        Total devices 1 FS bytes used 384.00KiB
        devid    1 size 238.47GiB used 1.02GiB path /dev/sdk1

Label: none  uuid: e15f3b51-09b3-4cab-bbee-13670824960d
        Total devices 1 FS bytes used 10.86GiB
        devid    1 size 30.00GiB used 20.02GiB path /dev/loop2

How can I test or otherwise verify the pool is working? If not, how can I fix it?

 

Edited March 18, 2020 by walle

[JorgeB]

Cache2, is not part of the pool, please post the diagnostics: Tools -> Diagnostics

[walle]

Ok, here you go.

walleserver-diagnostics-20200318-0901.zip

[JorgeB]

I don't see why on the syslog, but for some reason the second device wasn't added to the pool, it should be if you try again, just make sure cache backups are up date in case something goes wrong, then:

 

stop array

unassign cache2

start array

stop array

re-assign cache2

start array and hopefully it will work now.

[walle]

I did that, it seams that it's still the same issue. I have attached fresh diagnostics zip.

 

But something I noted was that when I stopped and unassigned the drive and started the array again, the docker service failed to start. I connected to the server via SSH and saw that the /mnt/user didn't exist. This was fixed by starting and stopping the array without any changes. When I later re-assigned cache2 it happen again, and was solved in the same way. Do you think this odd behavior have something to do with including cache2 into the cache pool?

walleserver-diagnostics-20200318-1142.zip

[JorgeB]

I still don't see any errors, or why it's failing to add, try wiping the device before adding back again, check cache2 is still sdf, and after:

 

stop array

unassign cache2

start array

 

run:

blkdiscard /dev/sdf

 

If no errors continue:

 

stop array

re-assign cache2

start array

 

[walle]

I don't think that helped. Seams to be the same issue. See attachment for fresh diagnostics.

 

I was not sure exactly how you wanted me to wipe the device, so what I did in that case was to remove the partition and formatted to unencrypted BTRFS (if that matters at all?). Also it has been consistent and called it sdf.

walleserver-diagnostics-20200318-1317.zip

Edited March 18, 2020 by walle

[JorgeB]

1 minute ago, walle said:

I was not sure exactly how you wanted me to wipe the device,

By typing

 

1 hour ago, johnnie.black said:

blkdiscard /dev/sdf

 

[walle]

Ah ok, yes I did that also. I didn't help.

 

[JorgeB]

Can you try adding a different device, even a regular HDD if you don't have another SSD, just to see if there's any difference.

[walle]

I think I found the problem.

 

I use a script from this topic in order to have the encryption key stored on another server:

I think the issue is that the unlock encryption key was removed too early in the process and therefor couldn't start the balance operation.  After I disabled the key deletion script and re-added the drive to the pool, then the balance operation with the start of the array. Now I just have to wait and see if the balance operation can be completed successfully or not.

 

 

Edited March 18, 2020 by walle
Clerify what I did in order to solve it.

[JorgeB]

That's a strong possibility.

[walle]

The balance operation is done, and I think this looks like that everything is in working order (correct me if I'm wrong):

Label: none  uuid: fd9abfd5-7e13-487f-ba5d-419b90608d6b
        Total devices 2 FS bytes used 262.61GiB
        devid    1 size 465.76GiB used 293.03GiB path /dev/mapper/sdg1
        devid    2 size 465.75GiB used 293.03GiB path /dev/mapper/sdf1

Label: none  uuid: bf870768-3cdb-4f9e-836b-4b1ed2c4c253
        Total devices 1 FS bytes used 384.00KiB
        devid    1 size 238.47GiB used 1.02GiB path /dev/sdk1

Label: none  uuid: e15f3b51-09b3-4cab-bbee-13670824960d
        Total devices 1 FS bytes used 10.89GiB
        devid    1 size 30.00GiB used 20.02GiB path /dev/loop2

Thank you for your help @johnnie.black!

[JorgeB]

Yes, looks fine.