A few days ago, i got some 10TB drives and decided to switch from my current 8tb drives array to the new drives. My setup is 1 Parity and 1 Data disk.

 

I was able to switch out the parity drive successfully no problems. Once the 10TB parity drive was good, I stopped the array, replaced the data disk from 8tb to 10tb. I was expecting that this operation will result in a data-rebuild once i would start the array. But while it did start the data-rebuild, the array also refused to mount my cache drives with the error: "Unmountable: pool config invalid".

Due to this, docker and vm services are also not working.

 

I thought maybe once the data-rebuild would complete, the error would go away with a restart. but it didnt.

 

TLDR;

 

1. Stop the Array
2. Replace Parity Drive from 8TB to 10TB and Start the array
3. Parity was built successfully
4. Stop the Array
5. Next replace the data disk from 8TB to 10TB
6. Start the Array
7. Data-Rebuild Starts but cache pool has invalid config now.
8. Data Rebuild Finished
9. Server restart
10. Cache still unmountable with pool config invalid error.

 

The data on the cache specially the appdata which is set to cache only is important and needs to be recovered. The cache is setup of two 1TB nvme drives. both are different brands(Samsung EVO + Force MP600 ) 

 

While looking at the logs after the restart i can find these about the cache:

 

# grep '/mnt/cache' /var/log/syslog
Aug  6 06:09:50 Ark emhttpd: shcmd (29): mkdir -p /mnt/cache
Aug  6 06:09:50 Ark emhttpd: /mnt/cache uuid: 40177e1b-9e71-4d3f-9ba4-3b487adeefda
Aug  6 06:09:50 Ark emhttpd: /mnt/cache WARNING: adding device /dev/sde1 gen 851206 but found an existing device /dev/sdb1 gen 851211
Aug  6 06:09:50 Ark emhttpd: /mnt/cache ERROR: cannot scan /dev/sde1: File exists
Aug  6 06:09:50 Ark emhttpd: /mnt/cache ERROR: cannot scan /dev/sde1: File exists
Aug  6 06:09:50 Ark emhttpd: /mnt/cache Label: none  uuid: 40177e1b-9e71-4d3f-9ba4-3b487adeefda
Aug  6 06:09:50 Ark emhttpd: /mnt/cache Total devices 2 FS bytes used 99.80GiB
Aug  6 06:09:50 Ark emhttpd: /mnt/cache devid    1 size 931.51GiB used 330.03GiB path /dev/nvme0n1p1
Aug  6 06:09:50 Ark emhttpd: /mnt/cache devid    2 size 931.51GiB used 330.03GiB path /dev/nvme1n1p1
Aug  6 06:09:50 Ark emhttpd: /mnt/cache mount error: Invalid pool config
Aug  6 06:09:50 Ark emhttpd: shcmd (30): umount /mnt/cache
Aug  6 06:09:50 Ark root: umount: /mnt/cache: not mounted.
Aug  6 06:09:50 Ark emhttpd: shcmd (31): rmdir /mnt/cache

Any help to recover the cache would be highly appreciated.

 

Im also attaching the diagnostic reports as well.

ark-diagnostics-20220806-0613.zip

This happens because old disk1 is confusing the pool, it's this line:

 

3 hours ago, Saad said:

Aug  6 06:09:50 Ark emhttpd: /mnt/cache WARNING: adding device /dev/sde1 gen 851206 but found an existing device /dev/sdb1 gen 851211

Wipe or disconnect that disk and the pool will mount again.

Thank you so much, detaching the old disks did help. 👍