Plex docker randomly not working, log shows BTRFS errors.

[-Daedalus]

Full logs attached.

 

Errors and warnings marked (***)

 

Jun 22 16:14:01 server login[20195]: ROOT LOGIN on '/dev/pts/4' from '192.168.1.101'
Jun 22 17:55:50 server in.telnetd[10605]: connect from 192.168.1.101 (192.168.1.101)
Jun 22 17:55:57 server login[10606]: ROOT LOGIN on '/dev/pts/4' from '192.168.1.101'
*** Jun 22 18:48:39 server kernel: blk_update_request: I/O error, dev loop0, sector 0
Jun 22 18:48:39 server kernel: BTRFS: bdev /dev/loop0 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
*** Jun 22 18:48:39 server kernel: BTRFS: error (device loop0) in write_all_supers:3501: errno=-5 IO failure (errors while submitting device barriers.)
Jun 22 18:48:39 server kernel: BTRFS info (device loop0): forced readonly
*** Jun 22 18:48:39 server kernel: BTRFS warning (device loop0): Skipping commit of aborted transaction.
Jun 22 18:48:39 server kernel: ------------[ cut here ]------------
*** Jun 22 18:48:39 server kernel: WARNING: CPU: 1 PID: 7549 at fs/btrfs/super.c:260 __btrfs_abort_transaction+0x4d/0x10e()
*** Jun 22 18:48:39 server kernel: BTRFS: Transaction aborted (error -5)
Jun 22 18:48:39 server kernel: Modules linked in: kvm_intel kvm vhost_net vhost macvtap macvlan md_mod xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat ipmi_devintf igb i2c_algo_bit ptp i2c_i801 ahci pps_core libahci ipmi_si acpi_cpufreq [last unloaded: md_mod]
*** Jun 22 18:48:39 server kernel: CPU: 1 PID: 7549 Comm: btrfs-transacti Tainted: G W 4.1.18-unRAID #1
Jun 22 18:48:39 server kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./C2750D4I, BIOS P2.80 12/04/2014
Jun 22 18:48:39 server kernel: 0000000000000009 ffff88024e407c58 ffffffff815f2403 ffffffff8182e140
Jun 22 18:48:39 server kernel: 0000000000000292 ffff88024e407c58 ffff88024e407ca8 ffff88024e407c98
Jun 22 18:48:39 server kernel: ffffffff8104778b ffff88024e407ce8 ffffffff81286d44 00000000fffffffb
*** Jun 22 18:48:39 server kernel: Call Trace:
Jun 22 18:48:39 server kernel: [<ffffffff815f2403>] dump_stack+0x65/0x85
Jun 22 18:48:39 server kernel: [<ffffffff8104778b>] warn_slowpath_common+0x97/0xb1
Jun 22 18:48:39 server kernel: [<ffffffff81286d44>] ? __btrfs_abort_transaction+0x4d/0x10e
Jun 22 18:48:39 server kernel: [<ffffffff810477e6>] warn_slowpath_fmt+0x41/0x43
Jun 22 18:48:39 server kernel: [<ffffffff81286d44>] __btrfs_abort_transaction+0x4d/0x10e
Jun 22 18:48:39 server kernel: [<ffffffff812ac3ad>] cleanup_transaction+0x80/0x21d
Jun 22 18:48:39 server kernel: [<ffffffff810724fb>] ? wait_woken+0x7d/0x7d
Jun 22 18:48:39 server kernel: [<ffffffff812ad617>] btrfs_commit_transaction+0xa6c/0xa81
Jun 22 18:48:39 server kernel: [<ffffffff812a93ce>] transaction_kthread+0xfa/0x1cb
Jun 22 18:48:39 server kernel: [<ffffffff812a92d4>] ? btrfs_cleanup_transaction+0x461/0x461
Jun 22 18:48:39 server kernel: [<ffffffff8105c74a>] kthread+0xd6/0xde
Jun 22 18:48:39 server kernel: [<ffffffff8105c674>] ? kthread_create_on_node+0x172/0x172
Jun 22 18:48:39 server kernel: [<ffffffff815f7f12>] ret_from_fork+0x42/0x70
Jun 22 18:48:39 server kernel: [<ffffffff8105c674>] ? kthread_create_on_node+0x172/0x172
Jun 22 18:48:39 server kernel: ---[ end trace 98c8eca6951a7289 ]---
*** Jun 22 18:48:39 server kernel: BTRFS: error (device loop0) in cleanup_transaction:1692: errno=-5 IO failure
Jun 22 18:48:39 server kernel: BTRFS info (device loop0): delayed_refs has NO entry
Jun 22 21:00:02 server php: /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker 'restart' 'Plex'
Jun 22 21:04:17 server php: /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker 'start' 'Plex'

 

Plex was throwing 'Unexpected error loading database' messages, so after trying some of the options in the GUI, I restarted the docker, and it's not refusing to start, with the above errors in the log.

 

While I'm happy to reinstall the docker, I'm planning to move some sensitive data to a BTRFS cache pool shortly, and I'm a little uneasy about that after seeing the above.

 

Can anyone shed any light?

server-diagnostics-20160622-2107.zip

[RobJ]

It looks to me as if all the BTRFS errors are with loop0, which I believe is just docker.img.  So I *think* all you have to do is remove and recreate docker.img, plus add your containers back from the templates.  Should be quick!

[-Daedalus]

Cool, thanks for that! Will see how this goes...

 

 

Edit: Went to modify a docker (CAdvisor), just changing a port, and when hitting save, I get this error at the bottom

 

Warning: file_put_contents(/var/lib/docker/unraid-update-status.json): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 291 
Warning: file_put_contents(/var/lib/docker/unraid-update-status.json): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 438

 

And no docker. Not even inactive, it just got erased.

 

At the moment it's just running on a single 1TB laptop drive, but I have two 500GB SSDs ready to go, but as I was planning on keeping documents, pictures, etc. on the pool (figuring it would be more protected than the main array), I'm a little hesitant now. If it's just docker screwing up that's fine - annoying, but fine - if it's something lower-level, I'd rather not lose things.

[Squid]

That path is within the docker.img file.  Just do what JoeL suggests and delete the image file, then recreate it and readd your apps via CA's previous apps section and you'll be back up and running like nothing ever happened.

[-Daedalus]

Yup, did that, and worked nicely! Cheers lads!