Since I ran the reboot to upgrade to 6.9.1 my cache drive is unmountable.  I read in a few posts that sounded simmilar making sure that they are running the same Filesystem would be a good first step.  Still no dice.  The real problem is it seems to have defaulted my unifi controller server.... I'd really like to get that back, aaaaand run it off cache in the future.

 

Any suggestions?

Are you sure your cache drive isn't actually the drive listed as Dev 1 in UD?

Yup.  I have the 250GB SSD as the cache drive.

And it was working fine until I clicked on "reboot to update".

Tools, diagnostics, download zip file and attach it to your NEXT post in this thread.

Thank you for quick replies....  This is SUCKING....

ice-metal-diagnostics-20210401-1252.zip

Do you know what the filesystem is supposed to be? Your screenshot shows XFS but your syslog seems to indicate it decided btrfs instead and failed to mount:

Apr  1 12:24:52 ICE-METAL emhttpd: shcmd (52): mkdir -p /mnt/cache
Apr  1 12:24:52 ICE-METAL emhttpd: shcmd (53): blkid -t TYPE='xfs' /dev/nvme1n1p1 &> /dev/null
Apr  1 12:24:52 ICE-METAL emhttpd: shcmd (53): exit status: 2
Apr  1 12:24:52 ICE-METAL emhttpd: shcmd (54): blkid -t TYPE='btrfs' /dev/nvme1n1p1 &> /dev/null
Apr  1 12:24:52 ICE-METAL emhttpd: shcmd (55): mount -t btrfs -o noatime,space_cache=v2 /dev/nvme1n1p1 /mnt/cache
Apr  1 12:24:52 ICE-METAL kernel: BTRFS info (device nvme1n1p1): enabling free space tree
Apr  1 12:24:52 ICE-METAL kernel: BTRFS info (device nvme1n1p1): using free space tree
Apr  1 12:24:52 ICE-METAL kernel: BTRFS info (device nvme1n1p1): has skinny extents
Apr  1 12:24:52 ICE-METAL kernel: BTRFS info (device nvme1n1p1): enabling ssd optimizations
Apr  1 12:24:52 ICE-METAL kernel: BTRFS info (device nvme1n1p1): creating free space tree
Apr  1 12:24:53 ICE-METAL kernel: BTRFS critical (device nvme1n1p1): corrupt leaf: block=319438848 slot=6 extent bytenr=525130129408 len=65536 invalid generation, have 18446612817294090488 expect (0, 8296690]
Apr  1 12:24:53 ICE-METAL kernel: BTRFS error (device nvme1n1p1): block=319438848 read time tree block corruption detected
Apr  1 12:24:53 ICE-METAL kernel: BTRFS: error (device nvme1n1p1) in btrfs_create_free_space_tree:1189: errno=-5 IO failure
Apr  1 12:24:53 ICE-METAL kernel: BTRFS warning (device nvme1n1p1): failed to create free space tree: -5
Apr  1 12:24:53 ICE-METAL kernel: BTRFS error (device nvme1n1p1): commit super ret -30
Apr  1 12:24:53 ICE-METAL root: mount: /mnt/cache: can't read superblock on /dev/nvme1n1p1.
Apr  1 12:24:53 ICE-METAL kernel: BTRFS error (device nvme1n1p1): open_ctree failed
Apr  1 12:24:53 ICE-METAL emhttpd: shcmd (55): exit status: 32
Apr  1 12:24:53 ICE-METAL emhttpd: /mnt/cache mount error: not mounted
Apr  1 12:24:53 ICE-METAL emhttpd: shcmd (56): umount /mnt/cache
Apr  1 12:24:53 ICE-METAL root: umount: /mnt/cache: not mounted.
Apr  1 12:24:53 ICE-METAL emhttpd: shcmd (56): exit status: 32
Apr  1 12:24:53 ICE-METAL emhttpd: shcmd (57): rmdir /mnt/cache

 

I had changed it to XFS to match the array.

And back to btrfs and regardless it doesn't mount.  I've been certain NOT to format.

Just now, MendoRugger said:

I had changed it to XFS to match the array.

When?

Do you have any diagnostics or screenshots from before when things were running properly?

After it was deemed unmountable.  Everything was working perfectly until I clicked the "reboot to upgrade" link in the control panel and it rebooted.  That's when I found that first screen shot on the display connected to it.

The first screenshot of the CLI is from before any changes were made.  It was configured as btrfs at that point.

I've read elsewhere on the forum that the newer kernel is less tolerant of BTRFS corruption, so possibly the best way forward would be to temporarily roll back the OS using the Tools, Update OS, Unraid previous, Restore function.

 

If that gets your cache mounted, you should immediately do a backup in preparation for restoring it after a clean format of the cache drive.

Thank you!!  Trying right now.  Fingers crossed.  FIrst thing I will do is unload that cache if it works.

YES!!!  You're the man!!  It's up, dockers are working and mover is running!!!  Going to swap out that SSD with one I had already planned on putting in anyway.

15 minutes ago, MendoRugger said:

YES!!!  You're the man!!  It's up, dockers are working and mover is running!!!  Going to swap out that SSD with one I had already planned on putting in anyway.

Mover can't move open files, so if the docker and vm service are enabled, files will be left on the cache. You need to make sure not just that the containers are stopped, but the entire docker service is disabled so the mover can complete properly.

Will do!!  After I get an hard backup of unifi controller.  

4 hours ago, jonathanm said:

the newer kernel is less tolerant of BTRFS corruption

 

Looking at that from a slightly different angle, the older kernel was perhaps less able to detect existing corruption, which remained hidden until the upgrade revealed it. I think rolling back is only a temporary measure to give the opportunity to rescue the user's data. Ultimately, the only way forward is to fix the corruption which, in btrfs's case, usually means re-formatting.

 

On 4/2/2021 at 2:13 AM, John_M said:

Looking at that from a slightly different angle, the older kernel was perhaps less able to detect existing corruption, which remained hidden until the upgrade revealed it.

Correct, newer kernel can detect previously undetected corruption, user should downgrade, backup cache and the upgrade and re-format.

I was able to rescue the most important data.  Thank you for your help guys!!