Secondary cache drive not working as expected

[tbonedude420]

My backup/testing server, Loki, is causing me some grief as he always does.

I've got two NVME 250gb cache drives, one a samsung 960, one a samsung 970.
 

GOAL: Have two separate fast drives, for different things. IE, one for docker/appdata use, and one for my vmdisks

Reality: Docker, appdata, and vmdisks appear on both drives.

 

I believe btrfs or something else is 'pooling' the devices together. Under 'Pool Devices' one is listed as 'cache' and one as 'vmdrive' but they don't appear independent of each other. I've even tried removing the pool, changing from btrfs to xfs, and redoing things. Currently it wont mount the second drive since its XFS.

Am I doing something wrong here? Did I misunderstand 'Add a second pool' or am I finally going nutty?

 

Everyone loves pictures, so heres one https://prnt.sc/Hnrg_minXscy
https://prnt.sc/2VhboljUQJzZ (Array offline)

https://prnt.sc/RiexvgJLSOYh (Unassigned Devices)
Shows like its apart of 'pool' and the data that exists on it when viewing, is identical to 'cache'

This server, and everything about it are expandable. I'm willing to try anything. 

 

Edit: This is working as expected on my other server 'PhatMicro'. (Can be seen in signature) appdata/system/isos on first ssd, and only domains on the second ssd. GRRR. I don't have much hair left, but I'm willing to pull it all!

Edited March 12, 2023 by tbonedude420
updates, typos.

[itimpi]

I suggest you post your system's diagnostics zip file to your next post in this thread so we can check what is happening.

[tbonedude420]

28 minutes ago, itimpi said:

I suggest you post your system's diagnostics zip file to your next post in this thread so we can check what is happening.

Yikes, I was afraid of that 🤣 So currently i've been nulling out my syslog and mcelog due to having a slight ecc error. I could null both, and then post in theory? I've got new ram for the server ready to go, im just not looking forward to unracking it. Lol, you know how it is right?!

 

Hopefully you can see past the ECC errors. bwahahaha

loki-diagnostics-20230312-0625.zip

[itimpi]

I cannot see any sign in the diagnostics of a pool called ‘vmdrive’.    The only pool I can see is one called ‘cache’ that has both devices assigned to it in a RAID1 type configuration.   Interestingly your last screenshot does not show the pool vmdrive either!

 

Were they part of the same pool in the past?    You might have to wait for @JorgeB to get best advice on how to rectify this.

[tbonedude420]

Were they part of the same pool in the past?

 

Yes, they were. I removed it after realizing 2x250s = 250gb and no real world gain in performance at least in my use-case. I attempted to switch it from btrfs to xfs and it gave me the option to format, but that was unsuccessful. 

My guess is they are still linked somehow. Not sure how to proceed. Again, its all expendable and theres no risk of anything getting lost. This is a server made from spare parts. I use it for a lot of testing. Can and will copy pasta nuclear destructive lines of text into command line 🙈 hahahah

 

Edit: Stopped array, added second pool and that drive, started array. Made a new diagnostics.zip after those changes

 

loki-diagnostics-20230312-0714.zip

Edited March 12, 2023 by tbonedude420
updates, pics

[JorgeB]

Both devices are in the same pool, to separate, stop array, unassign one of the devices (leave it unassigned for now), start array, once the pool is balanced to the remaining device you can stop array and assign the other device to a new pool, then start array to format it.

[tbonedude420]

Hiya @JorgeBand thanks for the like on U.C.D
Removed nvme#2 from vmdisks, started the array with the check mark. Ran mover. Made new diagnostics. 

I see no change, however, I think I need to go back to how it was. I think I should add both devices to the same pool, then remove one, then balance? How do I go about doing that? Its only 28gb, should take seconds on nvme. 

 

I feel as if I've goofed something up, and im not sure how to proceed lol. Should I just make a new hw config and nuke everything? Even after removing it, it shows under UD as a 'pool' device. 

loki-diagnostics-20230312-0737.zip

 

EDIT: Readded it to 'cache' array. Its spinning its wheels and thinking. Next step is plan to turn array off, remove from 'cache' and do nothing else, and start array. Balance I guess should happen on its own? Then shutdown array again, and move it to vmdrive pool. Hopefully then start array, and format. (Appears its done thinking, gonna proceed as I typed above) will stop, remove drive, but leave 2 slots, and start array. 

 

 

EDIT2: Did as I said, and now its really angry bwahaha.

 

 

I checked format, and it errors out saying no pool uuid, Making a new diag.zip now (0753.zip)

loki-diagnostics-20230312-0753.zip

Edited March 12, 2023 by tbonedude420
updates.

[tbonedude420]

As it stands right now, theres no pool uuid, the disk cannot be formated, and im stuck. Its 8am MST, and I need some shuteye. I think my next plan of action is nuke the array, and add both nvme's to the normal array so they format as xfs. Stop, remove, and try again? Shrug. 

 

I knew I should've changed his name from Loki.... bwahahhaa. Thanks @itimpiand @JorgeB. Ill check in on this in a few hours after some much needed sleep. Latest diagnostics in the .zip above (0753). I put new ram on my desk too, so hopefully no one has to weed through mcelog errors 🤣

 

Hope to hear back about any possible solutions. Again, theres nothing here, literally. Just docker.img and libvert.iso. And maybe dozzle is installed. I am 100% sure I screwed this up somehow, just hoping I can figure it out and never do it again, and maybe help someone else in the future. 😛

[JorgeB]

Unfortunately can't see anything in your diags since the syslog is spammed with MCE related errors, reboot and post new diags after array start.