Good day yall,

 

Noticed my home assistant instance not functioning this morning, so opened up the Unraid UI and found that my brfs cache pool was reporting "Unmountable: Unsupported or no file system". I have two Samsung nvme drives in a raid1 (see attached photo). I don't think I've made changes to the server recently, although perhaps I'm forgetting... If anyone has ideas on steps to try to fix I'd very much appreciate it.

unraidy-diagnostics-20241207-1555.zip

So after looking at the logs, it seemed like running

 

btrfs rescue zero-log /dev/nvme0n1p1

 

In the terminal was a good step to take, given some browsing on this forum. I then restarted the array, and things look okay. My only questions are: how might this have happenened, and should I be doing something to prevent it in the future?

13 hours ago, celticGreen said:

btrfs rescue zero-log

Yes, that was the correct command for the error the pool had, this can sometimes happen without a specif reason, at least a known one, but if it happens again in the near future, I would recommend backing up and recreating the pool.

@JorgeB and sure enough it's happened again. Attached the log in case there's anything notably different than yesterday's case. Is there a reason why the redundancy of the 2nd drive isn't allowing for things to move ahead? Should I pull the 2nd ssd and use it elsewhere?

 

 

Will plan to rebuild cache pool.

 

adding:

rebuilt cache pool, will report back (or mark as solved) in 24hrs or so, when it seems the issue is unlikely to repeat.

unraidy-diagnostics-20241208-0721.zip

Edited December 8, 20241 yr by celticGreen
rebuilt cache pool

The pool is going read only, there are other issues, redundancy cannot help with filesystem issues, recommend backing up and recreating the pool

On 12/8/2024 at 8:28 AM, JorgeB said:

The pool is going read only, there are other issues, redundancy cannot help with filesystem issues, recommend backing up and recreating the pool

 

Recreated the pool yesterday, and same result this morning. Any further suggestions that might solve the problem?

Post new diags, but if a new pool is getting corrupt so quickly, there's likely an underlying hardware issue.

7 minutes ago, JorgeB said:

Post new diags, but if a new pool is getting corrupt so quickly, there's likely an underlying hardware issue.

Roger that. I'll post logs in the AM if the problem occurs again tonight. I've updated the docker containers, removed a few containers I'm not using, updated the OS, and run smart tests on the two nvme drives. Everything running great at the moment.

 

19 hours ago, JorgeB said:

Post new diags, but if a new pool is getting corrupt so quickly, there's likely an underlying hardware issue.

 

Thought maybe it was going to last... Was still functioning around 04:00 03:00 and then by 06:00 the problem reappeared. New logs attached - I do see a ton of BTRFS errors begin right at 03:00 in the logs. 03:00 is the time the appdata backup runs.

unraidy-diagnostics-20241210-0635.zip

Edited December 10, 20241 yr by celticGreen
update time

Btrfs is detecting data corruption on both devices, start by running memtest.

On 12/10/2024 at 7:10 AM, JorgeB said:

Btrfs is detecting data corruption on both devices, start by running memtest.

 

Going to mark this as solved - I ended up re-creating the cache pool as zfs, as I'd intended to do that for awhile anyway.

 

Have daily backups for the Home Assistant vm, so that was easy to redeploy.

Docker containers were easy to bring back as well, since the config stuff is saved (on the flash storage, I think?)

Did some more cleanup also, seemed like a good time.

 

Scrub'd and ran SMART tests.

I'll still run memtest86, I just haven't dragged a monitor down to the unRaid server - but I shall.

 

Appreciate your assistance @JorgeB - not just with this particular incident, but many of the threads I've referenced for all sorts of questions I've had with unRaid end with you providing a solution.

 

...I do very much hope to avoid having to reference this forum for at least a few weeks  

1 hour ago, celticGreen said:

I'll still run memtest86, I just haven't dragged a monitor down to the unRaid server - but I shall

Although it does not test all the memory it is worth pointing out that there is now a plugin that test most of it.   The advantage of the plugin is that can be run whiie Unraid is running which can be more convenient and if it shows any errors no need to run memtest86 for a more complete check.