cache "Unmountable: pool config invalid" error after data disk replacement


Go to solution Solved by JorgeB,

Recommended Posts

A few days ago, i got some 10TB drives and decided to switch from my current 8tb drives array to the new drives. My setup is 1 Parity and 1 Data disk.

 

I was able to switch out the parity drive successfully no problems. Once the 10TB parity drive was good, I stopped the array, replaced the data disk from 8tb to 10tb. I was expecting that this operation will result in a data-rebuild once i would start the array. But while it did start the data-rebuild, the array also refused to mount my cache drives with the error: "Unmountable: pool config invalid".

Due to this, docker and vm services are also not working.

 

I thought maybe once the data-rebuild would complete, the error would go away with a restart. but it didnt.

 

TLDR;

 

  1. Stop the Array
  2. Replace Parity Drive from 8TB to 10TB and Start the array
  3. Parity was built successfully
  4. Stop the Array
  5. Next replace the data disk from 8TB to 10TB
  6. Start the Array
  7. Data-Rebuild Starts but cache pool has invalid config now.
  8. Data Rebuild Finished
  9. Server restart
  10. Cache still unmountable with pool config invalid error.

 

The data on the cache specially the appdata which is set to cache only is important and needs to be recovered. The cache is setup of two 1TB nvme drives. both are different brands(Samsung EVO + Force MP600 ) 

 

While looking at the logs after the restart i can find these about the cache:


 

# grep '/mnt/cache' /var/log/syslog
Aug  6 06:09:50 Ark emhttpd: shcmd (29): mkdir -p /mnt/cache
Aug  6 06:09:50 Ark emhttpd: /mnt/cache uuid: 40177e1b-9e71-4d3f-9ba4-3b487adeefda
Aug  6 06:09:50 Ark emhttpd: /mnt/cache WARNING: adding device /dev/sde1 gen 851206 but found an existing device /dev/sdb1 gen 851211
Aug  6 06:09:50 Ark emhttpd: /mnt/cache ERROR: cannot scan /dev/sde1: File exists
Aug  6 06:09:50 Ark emhttpd: /mnt/cache ERROR: cannot scan /dev/sde1: File exists
Aug  6 06:09:50 Ark emhttpd: /mnt/cache Label: none  uuid: 40177e1b-9e71-4d3f-9ba4-3b487adeefda
Aug  6 06:09:50 Ark emhttpd: /mnt/cache Total devices 2 FS bytes used 99.80GiB
Aug  6 06:09:50 Ark emhttpd: /mnt/cache devid    1 size 931.51GiB used 330.03GiB path /dev/nvme0n1p1
Aug  6 06:09:50 Ark emhttpd: /mnt/cache devid    2 size 931.51GiB used 330.03GiB path /dev/nvme1n1p1
Aug  6 06:09:50 Ark emhttpd: /mnt/cache mount error: Invalid pool config
Aug  6 06:09:50 Ark emhttpd: shcmd (30): umount /mnt/cache
Aug  6 06:09:50 Ark root: umount: /mnt/cache: not mounted.
Aug  6 06:09:50 Ark emhttpd: shcmd (31): rmdir /mnt/cache

Any help to recover the cache would be highly appreciated.

 

Im also attaching the diagnostic reports as well.

ark-diagnostics-20220806-0613.zip

Link to comment
  • Solution

This happens because old disk1 is confusing the pool, it's this line:

 

3 hours ago, Saad said:
Aug  6 06:09:50 Ark emhttpd: /mnt/cache WARNING: adding device /dev/sde1 gen 851206 but found an existing device /dev/sdb1 gen 851211

Wipe or disconnect that disk and the pool will mount again.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.