Lots of BTRFS error entries in Log

[Cessquill]

Hi - this morning I've noticed that I'm getting lots of entries for...

Dec 20 09:10:19 unRAID1 kernel: BTRFS error (device loop2): bad tree block start 0 431685632

 

...together with some dockers stopped and giving a server execution error when trying to start.

 

Here's what's currently happening on my server...

• I'm putting a new parity drive in, and it's slowed right down
• The old parity is currently pre-clearing to be used as a data drive
• I have a couple of unmounted drives in there that are only in the rack to aid airflow
• I do not have a BTRFS drive - my cache WAS up until about a week or so ago, when with the help of you guys I ended up reformatting to XFS
• The mover is currently running

 

As a side issue (that may be related?), I no longer get notifications if my cache drive fills up.  This has been since I reformatted to XFS.  Planning on looking into that today, but this has taken precedence.

 

Diags file attached.

 

Any help appreciated - parity sync is currently due to finish in 18.5 days...!

unraid1-diagnostics-20171220-0913.zip

Edited April 8 by Cessquill
removed diagnostics for safety

[JorgeB]

Your docker image is corrupt, because your cache disk got full:

 

Dec 20 06:32:01 unRAID1 shfs: cache disk full
Dec 20 06:32:03 unRAID1 shfs: cache disk full
Dec 20 06:32:04 unRAID1 shfs: cache disk full
Dec 20 06:32:04 unRAID1 shfs: cache disk full
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 439799808, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 858880
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 96, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 439812096, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 859008
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 97, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 439959552, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 859296
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 98, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 440008704, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 859392
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 99, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 440074240, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 859520
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 100, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 1513472000, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 2956000
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 101, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 1513488384, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 2956032
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 102, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 1513553920, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 2956160
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 103, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 1513701376, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 2956448
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 104, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: loop: Write error at byte offset 1513750528, length 4096.
Dec 20 06:32:05 unRAID1 kernel: print_req_error: I/O error, dev loop2, sector 2956544
Dec 20 06:32:05 unRAID1 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 105, rd 0, flush 0, corrupt 1, gen 2
Dec 20 06:32:05 unRAID1 kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2253: errno=-5 IO failure (Error while writing out transaction)
Dec 20 06:32:05 unRAID1 kernel: BTRFS info (device loop2): forced readonly
Dec 20 06:32:05 unRAID1 kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Dec 20 06:32:05 unRAID1 kernel: BTRFS: error (device loop2) in cleanup_transaction:1873: errno=-5 IO failure
Dec 20 06:32:05 unRAID1 kernel: BTRFS info (device loop2): delayed_refs has NO entry

Make sure it's not full anymore and delete and recreate the docker image.

[Cessquill]

Cool - thanks.  Makes sense now I think about it.

 

The mover had freed up some space, so for now I stopped and started the docker service and all appears fine again (dockers started and errors gone).  I'm connected remotely at the moment, but if I need to rebuild I'll do it later when I'm back in the house.

 

This is where my knowledge ends - is a docker image kind of mounted as a BTRFS file system then?

 

Any idea why I'm not getting cache disk notifications since switching it to XFS?

[JorgeB]

3 minutes ago, Cessquill said:

is a docker image kind of mounted as a BTRFS file system then?

yes

 

3 minutes ago, Cessquill said:

Any idea why I'm not getting cache disk notifications since switching it to XFS?

nope, try disabling and re-enabling

[Cessquill]

Thanks.  I'll let the parity-sync and preclear die down then turn the cache off and on again.