I'm not sure what is wrong.  I was having difficulty with pulling down a docker container so I thought it was a network issue.  I read on the forum that my docker.img file could be corrupt so I ran the scrub utility and still I had problems pulling down a new container.  I checked my logs and found it flooded with these:

Apr  9 04:40:01 Oni kernel: loop: Write error at byte offset 6505435136, length 4096.

Apr  9 04:40:01 Oni kernel: blk_update_request: I/O error, dev loop0, sector 12705928

Apr  9 04:40:01 Oni kernel: BTRFS: bdev /dev/loop0 errs: wr 248990, rd 0, flush 0, corrupt 0, gen 0

Apr  9 04:40:01 Oni kernel: loop: Write error at byte offset 6505562112, length 4096.

Apr  9 04:40:01 Oni kernel: blk_update_request: I/O error, dev loop0, sector 12706176

Apr  9 04:40:01 Oni kernel: BTRFS: bdev /dev/loop0 errs: wr 248991, rd 0, flush 0, corrupt 0, gen 0

Apr  9 04:40:01 Oni kernel: blk_update_request: I/O error, dev loop0, sector 12706424

Apr  9 04:40:33 Oni kernel: lo_write_bvec: 10 callbacks suppressed

Apr  9 04:40:33 Oni kernel: loop: Write error at byte offset 6505172992, length 4096.

Apr  9 04:40:33 Oni kernel: blk_update_request: 9 callbacks suppressed

 

I'm currently running an extended S.M.A.R.T. test against the cached disk which is where my docker container .img file is located.  Unraid doesn't have any indication that drive is bad and it's rather new with no history of failures.  I'm also running a parity check at this time since the drive was recently replaced and this was not done yet.

 

Other symptoms are:

• the http://[iPAddress]/Dashboard page takes 2+ minutes to load and present information.
  
• Creating new docker containers at the moment seems to result in failure.  Even small simple ones like HaProxy.
  

  Error response from daemon: Some kind of disk I/O error occurred: disk I/O error

 

Towards the end of writing this I noticed a problem in the logs pointing to networking.  Not sure what to do about it to fix it.

Apr  6 21:20:51 initech avahi-daemon[10917]: New relevant interface virbr0.IPv4 for mDNS.

Apr  6 21:20:51 initech avahi-daemon[10917]: Registering new address record for 192.168.122.1 on virbr0.IPv4.

Apr  6 21:20:51 initech dnsmasq[12445]: started, version 2.72 cachesize 150

Apr  6 21:20:51 initech dnsmasq[12445]: compile time options: IPv6 GNU-getopt no-DBus i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect

Apr  6 21:20:51 initech dnsmasq-dhcp[12445]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h

Apr  6 21:20:51 initech dnsmasq-dhcp[12445]: DHCP, sockets bound exclusively to interface virbr0

Apr  6 21:20:51 initech dnsmasq[12445]: reading /etc/resolv.conf

Apr  6 21:20:51 initech dnsmasq[12445]: using nameserver 192.168.111.1#53

Apr  6 21:20:51 initech dnsmasq[12445]: read /etc/hosts - 1 addresses

Apr  6 21:20:51 initech dnsmasq[12445]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses

Apr  6 21:20:51 initech dnsmasq-dhcp[12445]: read /var/lib/libvirt/dnsmasq/default.hostsfile

Apr  6 21:20:51 initech kernel: virbr0: port 1(virbr0-nic) entered disabled state

Apr  6 21:20:55 initech kernel: loop: Write error at byte offset 6505431040, length 4096.

Apr  6 21:20:55 initech kernel: blk_update_request: I/O error, dev loop0, sector 12705920

Apr  6 21:20:55 initech kernel: BTRFS: bdev /dev/loop0 errs: wr 220890, rd 0, flush 0, corrupt 0, gen 0

Apr  6 21:20:55 initech kernel: loop: Write error at byte offset 6505558016, length 4096.

Apr  6 21:20:55 initech kernel: blk_update_request: I/O error, dev loop0, sector 12706168

 

My log attachment size exceeds the allowed for this forum.  I'll post it elsewhere and make it available here somehow should it be needed.

All those BTRFS (loop0) messages indicates either a corrupt docker.img file or your docker.img file is 100% full.    Settings - Docker will show the usage of the docker.img

Nothing looks out of place that I can tell from Docker Settings, see attachment.

 

Also, why is this happening and could it be related?  It never took that long to load that page before.

the http://[iPAddress]/Dashboard page takes 2+ minutes to load and present information.

Looks like the Dashboard issue with time to load went away when the Parity check was completed.

 

Here's a link to the logs since I couldn't upload them directly from the forum:

https://app.box.com/s/l4sjrdad1polx8v7f6gyf16rguubfzgh

 

So the error with virbr0 shouldn't be a concern?

The change for the better of Dashboard loading speed was short lived and the pain of 2+ minute loading has returned.

 

Anyway, if I were to have Docker image corruption how does one recover from that?  I'd prefer not the boot and nuke option, but if that is the only way then I'd be willing to do so if there is a way to extract configurations to re-use.

 

Also, how is it possible that I have a corrupted Docker image?  The drive S.M.A.R.T. test indicates no fault.  I've read that something to do with BTRFS file system + violating Docker image maximum size = formula for corruption.  I don't believe that my image pushed the boundaries of size limitations so I'm not sure how I got here, but I know very little about this topic really.

The only path of success for recovering from corruption that I found was to nuke the Unraid docker image and start over.  I hope that is not the only strategy available in the future...