Jump to content
tooviral

Errors in log, Docker not working

5 posts in this topic Last Reply

Recommended Posts

Had this crop up in my system today.

 

Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 715571200, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1397600
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 715591680, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1397632
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 1789317120, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3494752
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 1789333504, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3494784
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2411786240, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4710512
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2411913216, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4710767
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2520440832, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4922736
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2556149760, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4992480
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2556280832, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4992735
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2411782144, length 4096.
Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4710512
Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 11, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:48 Tower kernel: BTRFS: error (device loop0) in btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing out transaction)
Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): forced readonly
Oct 13 16:06:48 Tower kernel: BTRFS warning (device loop0): Skipping commit of aborted transaction.
Oct 13 16:06:48 Tower kernel: ------------[ cut here ]------------
Oct 13 16:06:48 Tower kernel: WARNING: CPU: 11 PID: 21502 at fs/btrfs/transaction.c:1850 cleanup_transaction+0x8c/0x238
Oct 13 16:06:48 Tower kernel: BTRFS: Transaction aborted (error -5)
Oct 13 16:06:48 Tower kernel: Modules linked in: md_mod iptable_mangle ip_tables bonding ixgbe ptp pps_core mdio x86_pkg_temp_thermal coretemp kvm_intel mpt3sas i2c_i801 ahci i2c_smbus kvm raid_class i2c_core libahci megaraid_sas scsi_transport_sas wmi ipmi_si [last unloaded: pps_core]
Oct 13 16:06:48 Tower kernel: CPU: 11 PID: 21502 Comm: dockerd Not tainted 4.9.30-unRAID #1
Oct 13 16:06:48 Tower kernel: Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.2a 06/18/2016
Oct 13 16:06:48 Tower kernel: ffffc9001a7a3c78 ffffffff813a4a1b ffffc9001a7a3cc8 ffffffff8196b262
Oct 13 16:06:48 Tower kernel: ffffc9001a7a3cb8 ffffffff8104d0d9 0000073a1a7a3d30 ffff883fc58adef0
Oct 13 16:06:48 Tower kernel: ffff881fefbf6800 ffff883fd63d6aa0 00000000fffffffb 0000000000000000
Oct 13 16:06:48 Tower kernel: Call Trace:
Oct 13 16:06:48 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Oct 13 16:06:48 Tower kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Oct 13 16:06:48 Tower kernel: [<ffffffff8104d13a>] warn_slowpath_fmt+0x46/0x4e
Oct 13 16:06:48 Tower kernel: [<ffffffff812eeff1>] cleanup_transaction+0x8c/0x238
Oct 13 16:06:48 Tower kernel: [<ffffffff8107c0fd>] ? wake_up_bit+0x25/0x25
Oct 13 16:06:48 Tower kernel: [<ffffffff812f0d59>] btrfs_commit_transaction.part.11+0x912/0x927
Oct 13 16:06:48 Tower kernel: [<ffffffff812f0db4>] btrfs_commit_transaction+0x46/0x4d
Oct 13 16:06:48 Tower kernel: [<ffffffff81304934>] btrfs_sync_file+0x2ca/0x2f5
Oct 13 16:06:48 Tower kernel: [<ffffffff81146f6e>] vfs_fsync_range+0x87/0x99
Oct 13 16:06:48 Tower kernel: [<ffffffff81146f97>] vfs_fsync+0x17/0x19
Oct 13 16:06:48 Tower kernel: [<ffffffff81146fc5>] do_fsync+0x2c/0x45
Oct 13 16:06:48 Tower kernel: [<ffffffff811471d7>] SyS_fdatasync+0xe/0x12
Oct 13 16:06:48 Tower kernel: [<ffffffff8167f537>] entry_SYSCALL_64_fastpath+0x1a/0xa9
Oct 13 16:06:48 Tower kernel: ---[ end trace 54a3fd7b9446e8d1 ]---
Oct 13 16:06:48 Tower kernel: BTRFS: error (device loop0) in cleanup_transaction:1850: errno=-5 IO failure
Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): delayed_refs has NO entry
Oct 13 16:06:48 Tower emhttp: shcmd (111): umount /var/lib/docker |& logger
Oct 13 16:06:48 Tower kernel: BTRFS error (device loop0): cleaner transaction attach returned -30
Oct 13 16:06:48 Tower emhttp: nothing to sync
Oct 13 16:06:48 Tower emhttp: shcmd (123): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 60 |& logger
Oct 13 16:06:48 Tower root: truncate: failed to truncate '/mnt/cache/system/docker/docker.img' at 64424509440 bytes: No space left on device
Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): disk space caching is enabled
Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): has skinny extents
Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): bdev /dev/loop0 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:48 Tower root: Resize '/var/lib/docker' of 'max'
Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): new size for /dev/loop0 is 64424509440
Oct 13 16:06:48 Tower emhttp: shcmd (124): /etc/rc.d/rc.docker start |& logger
Oct 13 16:06:48 Tower root: starting docker ...
Oct 13 16:06:49 Tower kernel: lo_write_bvec: 201 callbacks suppressed
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1146310656, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: 201 callbacks suppressed
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2238880
Oct 13 16:06:49 Tower kernel: btrfs_dev_stat_print_on_error: 191 callbacks suppressed
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 22, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1441157120, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2814752
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 23, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1492357120, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2914752
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 24, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1510035456, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2949280
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 25, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1611321344, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3147104
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 26, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1618743296, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3161600
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 27, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1629851648, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3183296
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 28, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 573804544, length 4096.
Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1120704
Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 29, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:50 Tower kernel: loop: Write error at byte offset 622366720, length 4096.
Oct 13 16:06:50 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1215552
Oct 13 16:06:50 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 30, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:50 Tower kernel: loop: Write error at byte offset 622448640, length 4096.
Oct 13 16:06:50 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1215712
Oct 13 16:06:50 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 31, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:52 Tower kernel: BTRFS: error (device loop0) in btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing out transaction)
Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): forced readonly
Oct 13 16:06:52 Tower kernel: BTRFS warning (device loop0): Skipping commit of aborted transaction.
Oct 13 16:06:52 Tower kernel: ------------[ cut here ]------------
Oct 13 16:06:52 Tower kernel: WARNING: CPU: 30 PID: 21982 at fs/btrfs/transaction.c:1850 cleanup_transaction+0x8c/0x238
Oct 13 16:06:52 Tower kernel: BTRFS: Transaction aborted (error -5)
Oct 13 16:06:52 Tower kernel: Modules linked in: md_mod iptable_mangle ip_tables bonding ixgbe ptp pps_core mdio x86_pkg_temp_thermal coretemp kvm_intel mpt3sas i2c_i801 ahci i2c_smbus kvm raid_class i2c_core libahci megaraid_sas scsi_transport_sas wmi ipmi_si [last unloaded: pps_core]
Oct 13 16:06:52 Tower kernel: CPU: 30 PID: 21982 Comm: dockerd Tainted: G        W       4.9.30-unRAID #1
Oct 13 16:06:52 Tower kernel: Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.2a 06/18/2016
Oct 13 16:06:52 Tower kernel: ffffc9001b757c78 ffffffff813a4a1b ffffc9001b757cc8 ffffffff8196b262
Oct 13 16:06:52 Tower kernel: ffffc9001b757cb8 ffffffff8104d0d9 0000073a1b757d30 ffff883fccc85658
Oct 13 16:06:52 Tower kernel: ffff881fdb092800 ffff883fcce84d20 00000000fffffffb 0000000000000000
Oct 13 16:06:52 Tower kernel: Call Trace:
Oct 13 16:06:52 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Oct 13 16:06:52 Tower kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Oct 13 16:06:52 Tower kernel: [<ffffffff8104d13a>] warn_slowpath_fmt+0x46/0x4e
Oct 13 16:06:52 Tower kernel: [<ffffffff812eeff1>] cleanup_transaction+0x8c/0x238
Oct 13 16:06:52 Tower kernel: [<ffffffff8107c0fd>] ? wake_up_bit+0x25/0x25
Oct 13 16:06:52 Tower kernel: [<ffffffff812f0d59>] btrfs_commit_transaction.part.11+0x912/0x927
Oct 13 16:06:52 Tower kernel: [<ffffffff812f0db4>] btrfs_commit_transaction+0x46/0x4d
Oct 13 16:06:52 Tower kernel: [<ffffffff81304934>] btrfs_sync_file+0x2ca/0x2f5
Oct 13 16:06:52 Tower kernel: [<ffffffff81146f6e>] vfs_fsync_range+0x87/0x99
Oct 13 16:06:52 Tower kernel: [<ffffffff81146f97>] vfs_fsync+0x17/0x19
Oct 13 16:06:52 Tower kernel: [<ffffffff81146fc5>] do_fsync+0x2c/0x45
Oct 13 16:06:52 Tower kernel: [<ffffffff811471d7>] SyS_fdatasync+0xe/0x12
Oct 13 16:06:52 Tower kernel: [<ffffffff8167f537>] entry_SYSCALL_64_fastpath+0x1a/0xa9
Oct 13 16:06:52 Tower kernel: ---[ end trace 54a3fd7b9446e8d2 ]---
Oct 13 16:06:52 Tower kernel: BTRFS: error (device loop0) in cleanup_transaction:1850: errno=-5 IO failure
Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): delayed_refs has NO entry
Oct 13 16:06:52 Tower emhttp: shcmd (126): umount /var/lib/docker |& logger
Oct 13 16:06:52 Tower kernel: BTRFS error (device loop0): cleaner transaction attach returned -30
Oct 13 16:06:52 Tower emhttp: shcmd (138): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 60 |& logger
Oct 13 16:06:52 Tower root: truncate: failed to truncate '/mnt/cache/system/docker/docker.img' at 64424509440 bytes: No space left on device
Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): disk space caching is enabled
Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): has skinny extents
Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): bdev /dev/loop0 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Oct 13 16:06:52 Tower root: Resize '/var/lib/docker' of 'max'
Oct 13 16:06:52 Tower emhttp: shcmd (139): /etc/rc.d/rc.docker start |& logger
Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): new size for /dev/loop0 is 64424509440

Have already deleted the docker img and keep getting the error.

 

Logs are attached from before I tried deleting the docker img and after

tower-diagnostics-20171013-1554.zip

tower-diagnostics-20171013-1653.zip

Edited by tooviral

Share this post


Link to post

Seems like this was the problem

 

 

Had to delete some files, and ran the commands @johnnie.black mentioned.

 

Haven't seen an error yet.

 

Any way to prevent this from happening.  This has been the second time I've had to do a balance.

Edited by tooviral

Share this post


Link to post
On 10/14/2017 at 11:33 AM, tooviral said:

 

 

Any way to prevent this from happening.  This has been the second time I've had to do a balance.

 

Any ideas?

Share this post


Link to post
12 minutes ago, tooviral said:

 

Any ideas?

 

Run this balance once a week, you can use the user scripts plugin:

 

btrfs balance start -dusage=75 /mnt/cache

 

Once unRAID gets to kernel 4.14 this issue should get fixed or at least improve.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.