[SOLVED] - cannot start containers, cache mismatch issue?

[BurntOC]

I'd really, really appreciate any help here, as my internal Unraid server has created a big problem.  I got a warning that my cache drive has gone read only or is full, but it's only showing 8% utilized with almost 450GB free.  I'd gotten an error same time that Docker image could not be written to and I'd started by deleting that to recreate it.  It won't start because of the cache drive issue, though.   I'm down and I don't know how to fix it.

 

I've attached diagnostics, and I would really appreciate any assistance or thoughts.

 

 

unraid2-diagnostics-20201221-0739.zip

Edited December 22, 2020 by BurntOC

[BurntOC]

I found this in the log, which appears to point to the issue:

 

Dec 21 01:48:59 unraid2 kernel: BTRFS critical (device sde1): corrupt leaf: root=10 block=43443863552 slot=331, bad key order, prev (18413108283400951808 255 4096) current (284853731328 199 4096)
Dec 21 01:48:59 unraid2 kernel: BTRFS info (device sde1): leaf 43443863552 gen 185307 total ptrs 467 free space 4360 owner 10
Dec 21 01:48:59 unraid2 kernel: BTRFS error (device sde1): block=43443863552 write time tree block corruption detected
Dec 21 01:48:59 unraid2 kernel: BTRFS: error (device sde1) in btrfs_commit_transaction:2338: errno=-5 IO failure (Error while writing out transaction)
Dec 21 01:48:59 unraid2 kernel: BTRFS info (device sde1): forced readonly
Dec 21 01:48:59 unraid2 kernel: BTRFS warning (device sde1): Skipping commit of aborted transaction.
Dec 21 01:48:59 unraid2 kernel: BTRFS: error (device sde1) in cleanup_transaction:1901: errno=-5 IO failure
Dec 21 07:33:15 unraid2 kernel: BTRFS info (device sde1): scrub: started on devid 1
Dec 21 07:33:15 unraid2 kernel: BTRFS info (device sde1): scrub: not finished on devid 1 with status: -30
Dec 21 07:48:08 unraid2 emhttpd: read SMART /dev/sde

 

Any ideas on how I can recover this?  The cache drive is btrfs and while I've repaired an xfs system on my other Unraid server, this btrfs looks a little riskier. Any suggestions, and any idea what would cause this?  SMART looks fine, so I don't think it's a bad drive but I'm no expert, obviously.

[JorgeB]

Cache filesystem is corrupt, best bet is to backup, re-format and restore the data.

[BurntOC]

11 hours ago, JorgeB said:

Cache filesystem is corrupt, best bet is to backup, re-format and restore the data.

Thanks.  That's what I was afraid of.   I did actually get it back up.  Not sure if running the balance or scrub fixed it, but glad it came back - if even for a bit.  

 

Any idea what would cause that?  Again, the drive looks fine - at least judging by the SMART properties and it's only a couple years old at mosts.   I think I'm going to try XFS if I have to go through a reformat and restore.

[JonathanM]

12 minutes ago, BurntOC said:

 I think I'm going to try XFS if I have to go through a reformat and restore.

Keep in mind that whatever caused your BTRFS issues may still effect your data, but you may not find out immediately. BTRFS is, as I like to put it, fragile.

 

BTRFS will reveal issues in a much more dramatic way, and doesn't seem to be as recoverable as XFS, but that doesn't mean your issues were CAUSED by BTRFS.

 

A stable well running system should have no problems with BTRFS. So beware of masking an issue vs fixing the actual problem, be it flaky RAM, power, whatever.

[BurntOC]

12 hours ago, jonathanm said:

Keep in mind that whatever caused your BTRFS issues may still effect your data, but you may not find out immediately. BTRFS is, as I like to put it, fragile.

 

BTRFS will reveal issues in a much more dramatic way, and doesn't seem to be as recoverable as XFS, but that doesn't mean your issues were CAUSED by BTRFS.

 

A stable well running system should have no problems with BTRFS. So beware of masking an issue vs fixing the actual problem, be it flaky RAM, power, whatever.

Fair enough.  I have an older HP Z240 that I slapped some drives, memory, and a 1050 Ti in and maybe it's bumping up against something in there.  It does on the rare occasion freeze up, but it's mostly pretty stable.  New rig is coming before year-end so hopefully that will eliminate these issues.  I'll mark as solved for now, just because at least I'm back up and running.