Weird behaviors in Unraid, cache pool not mounting after reboot

[Maxime Lacelle]

Hi,

 

For the last month or so I have been having a lot of trouble with my unraid installation.

It initially started a while after upgrading to the 6.10 rc2 but stayed around while trying to rollback to 6.9.2.

 

Things like a not responsive Web UI, crashing nginx service, devices disappearing from additional cache pools I have setup or not being detected from time to time when rebooting the server.

 

As of now, I would say that my biggest issue is that one of my pools is seen as "Unmountable : No file system" after booting up. This occurs about a third of the time and prevents me from running my docker containers (since my appdata share is hosted on them).

I use two adata nvme m.2 ssd in raid 1.

 

 

What I have tried:

• Removing one device at a time from the pool, formatting that device, adding it back to the pool and do the same with the second one.
• Changing the "balance mode" from raid 1 to raid 0
• I ran several tests on the two ssds outside of unraid and they don't seem to be at their end of life or anything.
• This also happened with one of my nvme drives from the array cache pool.
• Ultimately, I have taken out my unraid flash drive from the server, wiped it out and restored one of the backups I had onto it to start fresh. It did a big difference in loading times and just the overall experience on the web ui. Some times it took as much as 2 minutes before being able to load the main page or any other pages and this is not a networking issue.

 

The diagnostics file I am joining was taken after wiping my flash drive, but I also took one before, when I had other issues.

 

 

Thanks in advance!

 

 

rawpower-diagnostics-20220217-2352.zip

Edited February 18, 2022 by Maxime Lacelle
typos

[JorgeB]

Feb 17 23:35:45 Rawpower kernel: BTRFS: device fsid f3246c47-c90d-4662-aef8-0197be2da3f7 devid 6 transid 162676 /dev/nvme0n1p1 scanned by udevd (7431)
Feb 17 23:35:45 Rawpower kernel: BTRFS: device fsid f3246c47-c90d-4662-aef8-0197be2da3f7 devid 7 transid 142467 /dev/nvme1n1p1 scanned by udevd (7431)

 

Transid is way off for one of the devices, they should be the same for all members of the same pool, besides that the superblock also appears to be damaged:

 

Feb 17 23:37:21 Rawpower kernel: BTRFS error (device nvme0n1p1): super_num_devices 1 mismatch with num_devices 1 found here

 

There are some recovery options here you can try but not sure they will work with this kind of damage:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=543490

 

 

[Maxime Lacelle]

Thanks a lot for the quick reply!

 

Any ideas or guesses on why it would've done that in the end? I know the nvme1n1p1 was not detected some times when booting up unraid (I had to reboot the server and it would eventually show up), but I thought it could have been a glitch in the RC release, as the drive itself seems to be working just fine, unless you see something else from the logs?

 

So then, after reading your reply, I guess my best bet is to move the data from that pool somewhere else, delete the entire pool, format the drives and rebuild it.

What I am worried about is that this happens again after a while.

 

 

Thanks again

 

 

 

[JorgeB]

If the one device drops trasn id will be out of sync, if it's a raid1 pool it will sync with a scrub, but of course that should not be a regular occurrence, or other issues might occur, the problem here is is that the superblock is also damaged, if any the recovery options work, yes, best bet is to backup and re-format the pool.

[Maxime Lacelle]

I will try this out and come back with some news afterwards.

 

 

Thanks a lot!