Issue with Docker not being able to write.

[[email protected]]

I cant update my dockers.  Plex stopped working. I think it cant write to the transcode folder?

Based on my googling it is a BFRS cache pool issue. But I am a little out of my element on how to fixit.

 

I tried to do a shutdown. but it couldn't get past syncing so i did an unclean shutdown and the cache drive was writable for a while.  I deleted ~400Gb of stuff off of it.  Then started a parity check, btrfs balance, and btrfs scrub with repair corrupted blocks checked.  A couple of hours later the issue was back.

 

Balance status

Data, RAID1: total=742.00GiB, used=347.72GiB System, RAID1: total=32.00MiB, used=160.00KiB Metadata, RAID1: total=4.00GiB, used=1.23GiB GlobalReserve, single: total=369.34MiB, used=0.00B

No balance found on '/mnt/cache'

 

btrfs scrub status:

UUID: f2db4662-24b4-4b09-8f87-932fdcd369dc Scrub started: Thu Oct 29 22:39:46 2020 Status: aborted Duration: 0:19:05 Total to scrub: 697.90GiB Rate: 228.39MiB/s Error summary: csum=2 Corrected: 0 Uncorrectable: 2 Unverified: 0

 

 

Oct 30 00:04:12 UNRAID kernel: loop: Write error at byte offset 587075584, length 4096.
Oct 30 00:04:12 UNRAID kernel: print_req_error: I/O error, dev loop2, sector 1146632
Oct 30 00:04:12 UNRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Oct 30 00:04:12 UNRAID kernel: loop: Write error at byte offset 581763072, length 4096.
Oct 30 00:04:12 UNRAID kernel: print_req_error: I/O error, dev loop2, sector 1136256
Oct 30 00:04:12 UNRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Oct 30 00:04:12 UNRAID kernel: loop: Write error at byte offset 583282688, length 4096.
Oct 30 00:04:12 UNRAID kernel: print_req_error: I/O error, dev loop2, sector 1139224
Oct 30 00:04:12 UNRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Oct 30 00:04:12 UNRAID kernel: loop: Write error at byte offset 587075584, length 4096.
Oct 30 00:04:12 UNRAID kernel: print_req_error: I/O error, dev loop2, sector 1146632
Oct 30 00:04:12 UNRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Oct 30 00:04:12 UNRAID kernel: loop: Write error at byte offset 581763072, length 4096.
Oct 30 00:04:12 UNRAID kernel: print_req_error: I/O error, dev loop2, sector 1136256
Oct 30 00:04:12 UNRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Oct 30 00:04:12 UNRAID kernel: loop: Write error at byte offset 587075584, length 4096.
Oct 30 00:04:12 UNRAID kernel: print_req_error: I/O error, dev loop2, sector 1146632
Oct 30 00:04:12 UNRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Oct 30 00:04:12 UNRAID kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2267: errno=-5 IO failure (Error while writing out transaction)

unraid-syslog-20201030-1551.zip

[JorgeB]

There's filesystem corruption on the pool, best bet is to backup, re-format and restore data.

 

P.S: next time please post full diags instead.

[[email protected]]

I don't see any smart failures in pool.

unraid-diagnostics-20201030-1035.zip

[JorgeB]

12 minutes ago, [email protected] said:

I don't see any smart failures in pool.

I said there's filesystem corruption, not failing devices.

[[email protected]]

Well backup and rebuilt of the cached pool worked.  Pain in the ass. second time i have had to do it.

 

Last time was due to an SSD biting the dust by having a ton of errors.  This time, not so sure.

Edited October 31, 2020 by [email protected]