November 14, 201411 yr Hi From beta10a i start getting something like this ... before i never had any problems i just raport this because already i get this 5-6 times - sometime after 3 day, other after 7 days loop: Write error at byte offset 2958520320, length 4096. loop: Write error at byte offset 2931404800, length 4096. loop: Write error at byte offset 2958520320, length 4096. loop: Write error at byte offset 2958647296, length 4096. loop: Write error at byte offset 2958774272, length 4096. loop: Write error at byte offset 2967339008, length 4096. loop: Write error at byte offset 2967465984, length 4096. loop: Write error at byte offset 2967592960, length 4096. loop: Write error at byte offset 2960388096, length 4096. loop: Write error at byte offset 2960388096, length 4096. BTRFS: error (device loop8) in btrfs_commit_transaction:1882: errno=-5 IO failure (Error while writing out transaction) BTRFS info (device loop8): forced readonly BTRFS warning (device loop8): Skipping commit of aborted transaction. ------------[ cut here ]------------ WARNING: CPU: 2 PID: 8064 at fs/btrfs/super.c:259 __btrfs_abort_transaction+0x4b/0xfb() BTRFS: Transaction aborted (error -5) Modules linked in: kvm_intel kvm vhost_net vhost tun veth xt_nat ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 nf_nat iptable_filter ip_tables md_mod w83627ehf hwmon_vid dvbsky_m88ds3103 ftdi_sio usbserial cx25840 dvbsky_m88rs6000 e1000e ptp pps_core cx23885 tveeprom btcx_risc videobuf_dvb i2c_i801 tda18271 cx2341x snd_pcm ahci snd_timer libahci snd soundcore videobuf_dma_sg videobuf_core smipcie i2c_algo_bit dvb_core rc_core v4l2_common videodev mpt2sas raid_class scsi_transport_sas CPU: 2 PID: 8064 Comm: btrfs-transacti Not tainted 3.16.3-unRAID #1 Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Extreme11, BIOS P1.50A 09/03/2014 0000000000000000 ffff8807c9acfcc8 ffffffff815d9dcf ffff8807c9acfd10 ffff8807c9acfd00 ffffffff81040747 ffffffff81294f8e 00000000fffffffb ffff8807d5791800 ffff8805bbcfc1e0 ffffffff816202b0 ffff8807c9acfd60 Call Trace: [<ffffffff815d9dcf>] dump_stack+0x45/0x56 [<ffffffff81040747>] warn_slowpath_common+0x75/0x8e [<ffffffff81294f8e>] ? __btrfs_abort_transaction+0x4b/0xfb [<ffffffff810407a7>] warn_slowpath_fmt+0x47/0x49 [<ffffffff81294f8e>] __btrfs_abort_transaction+0x4b/0xfb [<ffffffff812b8ac6>] cleanup_transaction+0x80/0x21d [<ffffffff8106f0c3>] ? __wake_up_sync+0xd/0xd [<ffffffff812b99d4>] btrfs_commit_transaction+0x857/0x86c [<ffffffff812b5a50>] transaction_kthread+0xf7/0x1c8 [<ffffffff812b5959>] ? btrfs_cleanup_transaction+0x45b/0x45b [<ffffffff8105897e>] kthread+0xd6/0xde [<ffffffff810588a8>] ? kthread_create_on_node+0x168/0x168 [<ffffffff815dfc7c>] ret_from_fork+0x7c/0xb0 [<ffffffff810588a8>] ? kthread_create_on_node+0x168/0x168 ---[ end trace eb91b13af57f7049 ]--- BTRFS: error (device loop8) in cleanup_transaction:1571: errno=-5 IO failure BTRFS info (device loop8): delayed_refs has NO entry
November 14, 201411 yr Did you have either an unclean shutdown of the image or fill up the drive the loopback image sits on?
November 14, 201411 yr Author before was clean shutdown (like always) - im using powerdown plugin about free space Label: none uuid: b85758c1-e2b9-401f-a072-2037ea70ad44 Total devices 1 FS bytes used 3.09GiB devid 1 size 10.00GiB used 7.79GiB path /dev/loop8 Btrfs v3.16.1
November 14, 201411 yr Sorry no not the space within the image file the space on the physical disk the image file lives on. I have found that if the disk fills up (even briefly) regardless if there is space free within the image itself it will corrupt. It may be unrelated but its worth pondering
November 17, 201411 yr i had same issues after upgrading to 10a never had them on 9 had to make 3 new dockr.img files in the last week tried a few times to get this img repaired but btrfs is not easily repairable as reiserfs is .... can we get this loop on XFS ? aka will docker run on XFS? in the hope that xfs is better fixable then btrfs ?
November 24, 201411 yr this morning i woke up again with a BTRFS issue Nov 24 10:24:16 R2D2 kernel: parent transid verify failed on 1015496704 wanted 23427 found 22558 (Minor Issues) Nov 24 10:24:16 R2D2 kernel: parent transid verify failed on 1015496704 wanted 23427 found 22558 (Minor Issues) Nov 24 10:24:16 R2D2 kernel: BTRFS: failed to read tree root on loop8 (Minor Issues) Nov 24 10:24:16 R2D2 php: Not starting Docker: mount error (Errors) tried to fix it but again no luck... root@R2D2:~# /etc/rc.d/rc.docker start mount: wrong fs type, bad option, bad superblock on /dev/loop8, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so Not starting Docker: mount error root@R2D2:~# mount -t btrfs -o ro,recovery /dev/loop8 /var/lib/docker root@R2D2:~# mc root@R2D2:/mnt/cache/dockerfiles# cd / root@R2D2:/# /etc/rc.d/rc.docker stop stopping docker ... unmounting docker loopback root@R2D2:/# btrfs-zero-log /dev/loop8 parent transid verify failed on 1015496704 wanted 23427 found 22558 parent transid verify failed on 1015496704 wanted 23427 found 22558 parent transid verify failed on 1015496704 wanted 23427 found 22558 parent transid verify failed on 1015496704 wanted 23427 found 22558 Ignoring transid failure Couldn't setup extent tree mounting with the ro option worked but only thing you can do is take a backup.... so busy again installing all my dockers on a new image
November 29, 201411 yr here we go again Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) Nov 30 02:08:44 R2D2 kernel: parent transid verify failed on 525189120 wanted 3970 found 2778 (Minor Issues) trying to recover fails .... root@R2D2:/# /etc/rc.d/rc.docker stop stopping docker ... 1e572081f341 74d39161fafd 093c54de1be4 377be3b0d379 aaf687ddde85 8587f20a8101 9034c4ef30d2 unmounting docker loopback root@R2D2:/# btrfs check --repair /dev/loop8 enabling repair mode check_mounted(): Could not open /dev/loop8 Could not check mount status: No such file or directory root@R2D2:/# losetup /dev/loop8 /mnt/cache/dockerimage/docker2.img losetup: failed to setup loop device: No such file or directory root@R2D2:/# mknod -m 660 /dev/loop8 b 7 8 root@R2D2:/# losetup /dev/loop8 /mnt/cache/dockerimage/docker2.img root@R2D2:/# btrfs check --repair /dev/loop8 enabling repair mode parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 Ignoring transid failure Couldn't setup extent tree Couldn't setup device tree Checking filesystem on /dev/loop8 UUID: 7d2b78b1-2b82-4afa-bd6d-919af1ba55f3 Critical roots corrupted, unable to fsck the FS Segmentation fault root@R2D2:/# btrfs-zero-log /dev/loop8 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 Ignoring transid failure Couldn't setup extent tree root@R2D2:/# Couldn't setup extent tree > ^C root@R2D2:/# btrfs check --init-csum-tree /dev/loop8 Creating a new CRC tree parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 Ignoring transid failure Couldn't setup extent tree Couldn't setup device tree Checking filesystem on /dev/loop8 UUID: 7d2b78b1-2b82-4afa-bd6d-919af1ba55f3 Critical roots corrupted, unable to fsck the FS Segmentation fault root@R2D2:/# btrfs check --repair --init-csum-tree /dev/loop8 enabling repair mode Creating a new CRC tree parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 parent transid verify failed on 524779520 wanted 3969 found 2778 Ignoring transid failure Couldn't setup extent tree Couldn't setup device tree Checking filesystem on /dev/loop8 UUID: 7d2b78b1-2b82-4afa-bd6d-919af1ba55f3 Critical roots corrupted, unable to fsck the FS Segmentation fault root@R2D2:/# mount -o recovery,ro /dev/loop8 /var/lib/docker root@R2D2:/# /etc/rc.d/rc.docker start starting docker ... root@R2D2:/# mounting it in recovery mode mounts it read only all other commands fail Lime tech please let us know how we can recover these images i am getting crazy from this ... i have till 132gb free on my cache drive where this image is .... so it is not that the drive fills up i think it is more something with the cache drive being busy and that then the btrfs driver fails...
February 8, 201511 yr Not 100% sure where to post this, but I ran into the docker.img corruption issue today. I'm using v6 beta12. I had rebooted my server after replacing a failed parity drive, and noticed that docker was not started. When I clicked the start docker button on the UI, the following error was seen in the log: Feb 8 08:49:23 Tower php: /etc/rc.d/rc.docker start Feb 8 08:49:23 Tower kernel: BTRFS info (device loop8): disk space caching is enabled Feb 8 08:49:23 Tower kernel: parent transid verify failed on 88735744 wanted 11878 found 8248 Feb 8 08:49:23 Tower kernel: parent transid verify failed on 88735744 wanted 11878 found 8248 Feb 8 08:49:23 Tower php: Not starting Docker: mount error Feb 8 08:49:23 Tower php: Feb 8 08:49:23 Tower kernel: BTRFS: open_ctree failed Based on other posts, I did the following to get back up and running: 1. i renamed my corrupted docker.img file to something else 2. On the docker webUI page, I hit start to let docker recreate the docker.img file 3. Added my apps back using the MY.xxxxx templates (THANK YOU FOR THESE!) The drive on which my original docker.img was located was nowhere near full. I'm not sure what would have caused the corruption. -Scott
February 8, 201511 yr I think we have this issue solved. More details soon!! Is it soon, yet? Almost ;-)
February 16, 201511 yr This issue is resolved in beta13 by setting a nodatacow flag on the btrfs loopback image file itself. However, this will require the recreation of your Docker image file in order for the bug to be resolved. Existing image files created from earlier betas will still be subject to this bug should the parent disk device the image resides on fill up with data. For these reasons, we highly encourage users to delete and recreate their docker image file from beta13 or later. NOTE: THIS ONLY APPLIES IF YOUR DOCKER IMAGE FILE RESIDES ON A DEVICE FORMATTED WITH BTRFS! If your Docker image file is located on a non-btrfs device, you should not be impacted by this issue. However, in the rare event that recreating your image would be too large to ask, and you wish to continue the use of your existing image file as it is today, you can potentially protect yourself from future corruption by moving the image file from it's current location off to a secondary device that is NOT formatted with the btrfs filesystem. If you wish to then return the image to the btrfs-formatted device, those comfortable with command line tools can connect via telnet / ssh and perform the following: 1) Create a new directory on your btrfs-formatted disk device. mkdir tmp 2) Apply the nodatacow flag to the folder. chattr +C -R tmp 3) Move the docker image file from your non-btrfs disk to this tmp folder. mv /mnt/path/to/docker.img /mnt/path/to/tmp/ 4) To confirm that the nodatacow flag is appropriately listed on your file, you can type the following command from within the tmp directory you created: lasttr If you see a C at the end of the attributes for the image file, that means you set the flag correctly. In theory, if your image didn't already have corrupted bits, doing this will prevent further corruption in the future, but this is only theory and not proven.
Archived
This topic is now archived and is closed to further replies.