tooviral Posted October 13, 2017 Share Posted October 13, 2017 (edited) Had this crop up in my system today. Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 715571200, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1397600 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 715591680, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1397632 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 1789317120, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3494752 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 1789333504, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3494784 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2411786240, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4710512 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2411913216, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4710767 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2520440832, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4922736 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2556149760, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4992480 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2556280832, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4992735 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:44 Tower kernel: loop: Write error at byte offset 2411782144, length 4096. Oct 13 16:06:44 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4710512 Oct 13 16:06:44 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 11, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:48 Tower kernel: BTRFS: error (device loop0) in btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing out transaction) Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): forced readonly Oct 13 16:06:48 Tower kernel: BTRFS warning (device loop0): Skipping commit of aborted transaction. Oct 13 16:06:48 Tower kernel: ------------[ cut here ]------------ Oct 13 16:06:48 Tower kernel: WARNING: CPU: 11 PID: 21502 at fs/btrfs/transaction.c:1850 cleanup_transaction+0x8c/0x238 Oct 13 16:06:48 Tower kernel: BTRFS: Transaction aborted (error -5) Oct 13 16:06:48 Tower kernel: Modules linked in: md_mod iptable_mangle ip_tables bonding ixgbe ptp pps_core mdio x86_pkg_temp_thermal coretemp kvm_intel mpt3sas i2c_i801 ahci i2c_smbus kvm raid_class i2c_core libahci megaraid_sas scsi_transport_sas wmi ipmi_si [last unloaded: pps_core] Oct 13 16:06:48 Tower kernel: CPU: 11 PID: 21502 Comm: dockerd Not tainted 4.9.30-unRAID #1 Oct 13 16:06:48 Tower kernel: Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.2a 06/18/2016 Oct 13 16:06:48 Tower kernel: ffffc9001a7a3c78 ffffffff813a4a1b ffffc9001a7a3cc8 ffffffff8196b262 Oct 13 16:06:48 Tower kernel: ffffc9001a7a3cb8 ffffffff8104d0d9 0000073a1a7a3d30 ffff883fc58adef0 Oct 13 16:06:48 Tower kernel: ffff881fefbf6800 ffff883fd63d6aa0 00000000fffffffb 0000000000000000 Oct 13 16:06:48 Tower kernel: Call Trace: Oct 13 16:06:48 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e Oct 13 16:06:48 Tower kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3 Oct 13 16:06:48 Tower kernel: [<ffffffff8104d13a>] warn_slowpath_fmt+0x46/0x4e Oct 13 16:06:48 Tower kernel: [<ffffffff812eeff1>] cleanup_transaction+0x8c/0x238 Oct 13 16:06:48 Tower kernel: [<ffffffff8107c0fd>] ? wake_up_bit+0x25/0x25 Oct 13 16:06:48 Tower kernel: [<ffffffff812f0d59>] btrfs_commit_transaction.part.11+0x912/0x927 Oct 13 16:06:48 Tower kernel: [<ffffffff812f0db4>] btrfs_commit_transaction+0x46/0x4d Oct 13 16:06:48 Tower kernel: [<ffffffff81304934>] btrfs_sync_file+0x2ca/0x2f5 Oct 13 16:06:48 Tower kernel: [<ffffffff81146f6e>] vfs_fsync_range+0x87/0x99 Oct 13 16:06:48 Tower kernel: [<ffffffff81146f97>] vfs_fsync+0x17/0x19 Oct 13 16:06:48 Tower kernel: [<ffffffff81146fc5>] do_fsync+0x2c/0x45 Oct 13 16:06:48 Tower kernel: [<ffffffff811471d7>] SyS_fdatasync+0xe/0x12 Oct 13 16:06:48 Tower kernel: [<ffffffff8167f537>] entry_SYSCALL_64_fastpath+0x1a/0xa9 Oct 13 16:06:48 Tower kernel: ---[ end trace 54a3fd7b9446e8d1 ]--- Oct 13 16:06:48 Tower kernel: BTRFS: error (device loop0) in cleanup_transaction:1850: errno=-5 IO failure Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): delayed_refs has NO entry Oct 13 16:06:48 Tower emhttp: shcmd (111): umount /var/lib/docker |& logger Oct 13 16:06:48 Tower kernel: BTRFS error (device loop0): cleaner transaction attach returned -30 Oct 13 16:06:48 Tower emhttp: nothing to sync Oct 13 16:06:48 Tower emhttp: shcmd (123): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 60 |& logger Oct 13 16:06:48 Tower root: truncate: failed to truncate '/mnt/cache/system/docker/docker.img' at 64424509440 bytes: No space left on device Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): disk space caching is enabled Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): has skinny extents Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): bdev /dev/loop0 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:48 Tower root: Resize '/var/lib/docker' of 'max' Oct 13 16:06:48 Tower kernel: BTRFS info (device loop0): new size for /dev/loop0 is 64424509440 Oct 13 16:06:48 Tower emhttp: shcmd (124): /etc/rc.d/rc.docker start |& logger Oct 13 16:06:48 Tower root: starting docker ... Oct 13 16:06:49 Tower kernel: lo_write_bvec: 201 callbacks suppressed Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1146310656, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: 201 callbacks suppressed Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2238880 Oct 13 16:06:49 Tower kernel: btrfs_dev_stat_print_on_error: 191 callbacks suppressed Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 22, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1441157120, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2814752 Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 23, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1492357120, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2914752 Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 24, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1510035456, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 2949280 Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 25, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1611321344, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3147104 Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 26, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1618743296, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3161600 Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 27, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 1629851648, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 3183296 Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 28, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:49 Tower kernel: loop: Write error at byte offset 573804544, length 4096. Oct 13 16:06:49 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1120704 Oct 13 16:06:49 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 29, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:50 Tower kernel: loop: Write error at byte offset 622366720, length 4096. Oct 13 16:06:50 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1215552 Oct 13 16:06:50 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 30, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:50 Tower kernel: loop: Write error at byte offset 622448640, length 4096. Oct 13 16:06:50 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1215712 Oct 13 16:06:50 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 31, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:52 Tower kernel: BTRFS: error (device loop0) in btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing out transaction) Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): forced readonly Oct 13 16:06:52 Tower kernel: BTRFS warning (device loop0): Skipping commit of aborted transaction. Oct 13 16:06:52 Tower kernel: ------------[ cut here ]------------ Oct 13 16:06:52 Tower kernel: WARNING: CPU: 30 PID: 21982 at fs/btrfs/transaction.c:1850 cleanup_transaction+0x8c/0x238 Oct 13 16:06:52 Tower kernel: BTRFS: Transaction aborted (error -5) Oct 13 16:06:52 Tower kernel: Modules linked in: md_mod iptable_mangle ip_tables bonding ixgbe ptp pps_core mdio x86_pkg_temp_thermal coretemp kvm_intel mpt3sas i2c_i801 ahci i2c_smbus kvm raid_class i2c_core libahci megaraid_sas scsi_transport_sas wmi ipmi_si [last unloaded: pps_core] Oct 13 16:06:52 Tower kernel: CPU: 30 PID: 21982 Comm: dockerd Tainted: G W 4.9.30-unRAID #1 Oct 13 16:06:52 Tower kernel: Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.2a 06/18/2016 Oct 13 16:06:52 Tower kernel: ffffc9001b757c78 ffffffff813a4a1b ffffc9001b757cc8 ffffffff8196b262 Oct 13 16:06:52 Tower kernel: ffffc9001b757cb8 ffffffff8104d0d9 0000073a1b757d30 ffff883fccc85658 Oct 13 16:06:52 Tower kernel: ffff881fdb092800 ffff883fcce84d20 00000000fffffffb 0000000000000000 Oct 13 16:06:52 Tower kernel: Call Trace: Oct 13 16:06:52 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e Oct 13 16:06:52 Tower kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3 Oct 13 16:06:52 Tower kernel: [<ffffffff8104d13a>] warn_slowpath_fmt+0x46/0x4e Oct 13 16:06:52 Tower kernel: [<ffffffff812eeff1>] cleanup_transaction+0x8c/0x238 Oct 13 16:06:52 Tower kernel: [<ffffffff8107c0fd>] ? wake_up_bit+0x25/0x25 Oct 13 16:06:52 Tower kernel: [<ffffffff812f0d59>] btrfs_commit_transaction.part.11+0x912/0x927 Oct 13 16:06:52 Tower kernel: [<ffffffff812f0db4>] btrfs_commit_transaction+0x46/0x4d Oct 13 16:06:52 Tower kernel: [<ffffffff81304934>] btrfs_sync_file+0x2ca/0x2f5 Oct 13 16:06:52 Tower kernel: [<ffffffff81146f6e>] vfs_fsync_range+0x87/0x99 Oct 13 16:06:52 Tower kernel: [<ffffffff81146f97>] vfs_fsync+0x17/0x19 Oct 13 16:06:52 Tower kernel: [<ffffffff81146fc5>] do_fsync+0x2c/0x45 Oct 13 16:06:52 Tower kernel: [<ffffffff811471d7>] SyS_fdatasync+0xe/0x12 Oct 13 16:06:52 Tower kernel: [<ffffffff8167f537>] entry_SYSCALL_64_fastpath+0x1a/0xa9 Oct 13 16:06:52 Tower kernel: ---[ end trace 54a3fd7b9446e8d2 ]--- Oct 13 16:06:52 Tower kernel: BTRFS: error (device loop0) in cleanup_transaction:1850: errno=-5 IO failure Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): delayed_refs has NO entry Oct 13 16:06:52 Tower emhttp: shcmd (126): umount /var/lib/docker |& logger Oct 13 16:06:52 Tower kernel: BTRFS error (device loop0): cleaner transaction attach returned -30 Oct 13 16:06:52 Tower emhttp: shcmd (138): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 60 |& logger Oct 13 16:06:52 Tower root: truncate: failed to truncate '/mnt/cache/system/docker/docker.img' at 64424509440 bytes: No space left on device Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): disk space caching is enabled Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): has skinny extents Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): bdev /dev/loop0 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Oct 13 16:06:52 Tower root: Resize '/var/lib/docker' of 'max' Oct 13 16:06:52 Tower emhttp: shcmd (139): /etc/rc.d/rc.docker start |& logger Oct 13 16:06:52 Tower kernel: BTRFS info (device loop0): new size for /dev/loop0 is 64424509440 Have already deleted the docker img and keep getting the error. Logs are attached from before I tried deleting the docker img and after tower-diagnostics-20171013-1554.zip tower-diagnostics-20171013-1653.zip Edited October 13, 2017 by tooviral Quote Link to comment
tooviral Posted October 14, 2017 Author Share Posted October 14, 2017 (edited) Seems like this was the problem Had to delete some files, and ran the commands @johnnie.black mentioned. Haven't seen an error yet. Any way to prevent this from happening. This has been the second time I've had to do a balance. Edited October 14, 2017 by tooviral Quote Link to comment
tooviral Posted October 17, 2017 Author Share Posted October 17, 2017 On 10/14/2017 at 11:33 AM, tooviral said: Any way to prevent this from happening. This has been the second time I've had to do a balance. Any ideas? Quote Link to comment
JorgeB Posted October 17, 2017 Share Posted October 17, 2017 12 minutes ago, tooviral said: Any ideas? Run this balance once a week, you can use the user scripts plugin: btrfs balance start -dusage=75 /mnt/cache Once unRAID gets to kernel 4.14 this issue should get fixed or at least improve. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.