Hi,

Over the the past couple days I discovered an issue which prevented my cache drive pool (2x 1TB nvme ssd's) from working - it was on a read-only state. After rebooting it said there was no file system found. I was able to remedy this and get it going by performing this command:

btrfs rescue zero-log /dev/nvme1n1p1

 

After performing this command and rebooting, it works for maybe 10min and goes back in to a faulty state. I then removed the drive nvme1n1 thinking it was faulty - ran the rescue command above, rebooted and all was well - albeit I only have 1 cache drive in my raid1 pool so I wanted to replace it with a new drive.

 

I inserted a brand new Samsung 980 PRO in to that slot, booted up, added the Samsung drive to the cache pool, and now I'm back to having a faulty cache pool again.

 

So... I guess the drive was never bad? But I'm not sure what is the issue here. I've attached diagnostic logs as well to the post.

 

Appreciate any help and input!

Thanks

jupiter-diagnostics-20230705-2052.zip

7 hours ago, chaosclarity said:

After performing this command and rebooting, it works for maybe 10min and goes back in to a faulty state.

In that case recommend backing up the pool and re-formatting.

3 hours ago, JorgeB said:

In that case recommend backing up the pool and re-formatting.

Well, new issue...

I replaced the drive with a Samsung 980 PRO and had intermittent issues getting this drive to show up in Unraid. Initially it did, and I added it to the cache pool, started the array but again I received these BTRFS errors and the cache pool went to a Read Only state.

 

I can't really tell if this is a hardware issue or a software issue in Unraid that's causing the issue. My real concern is that if I restore, this issue will come right back.

 

I replaced back to the original m.2 drive, but if I start the array, my docker/vm's are all gone. Is there some way to recover what was on there? Sadly, my Veeam backup license expired 30 days ago and now all backups are about 30 days old. Better than nothing, but am I really SOL here? I was backing up the appdata folder - if I want to restore from my backup, do I just dump all that in to the appdata folder? And what's the best way of going about restoring the dockers?

 

 

Edited July 6, 20232 yr by chaosclarity

13 minutes ago, chaosclarity said:

I can't really tell if this is a hardware issue or a software issue in Unraid that's causing the issue.

Diags might show something.

6 minutes ago, JorgeB said:

Diags might show something.

Hi Jorge,

Thanks for your help - I've uploaded the latest diagnostics

jupiter-diagnostics-20230706-0740.zip

Not seeing any issues with an NVMe device, or was it already missing at boot? Only see that the pool is missing a device.

9 minutes ago, JorgeB said:

Not seeing any issues with an NVMe device, or was it already missing at boot? Only see that the pool is missing a device.

Ok - I've re-added the original m.2 drive to the pool and formatted the drives, everything mounted now.

I've gone ahead and started restoring a backup of all appdata content from back in June. From there I just need to re-add the docker apps from the Previous Apps section?

Edited July 6, 20232 yr by chaosclarity

@JorgeB Should I be regularly scrubbing the cache drive pool? I just wonder how I got in to this situation. But it is a good exercise in knowing how to restore and keeping good backups.

13 minutes ago, chaosclarity said:

Should I be regularly scrubbing the cache drive pool?

Once a month is good, more important that that see here for better pool monitoring.