Dockers won't/don't start

September 20, 201411 yr

Hello,

At some point during today, it seems all my dockers were no longer working properly. I wanted to restart them, but after hitting "stop" and "start" they no longer load. It says the startup attempt is made, but no update ever occurs, and it never starts running, and the "start" button is available again. My log looks like this at the time

Sep 19 22:01:47 Alex-MEDIA php: /usr/bin/docker start PlexMediaServer
Sep 19 22:01:47 Alex-MEDIA php:

I'm also receieving this sin the log -- not sure if its related

Sep 19 22:00:41 Alex-MEDIA kernel: btrfs_readpage_end_io_hook: 10 callbacks suppressed
Sep 19 22:00:41 Alex-MEDIA kernel: BTRFS info (device sdd1): csum failed ino 258 off 537739264 csum 1719360858 expected csum 1713725129

Sep 19 22:00:41 Alex-MEDIA kernel: btrfs_dev_stat_print_on_error: 8 callbacks suppressed

Sep 19 22:00:41 Alex-MEDIA kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 55, flush 0, corrupt 0, gen 0

Sep 19 22:00:41 Alex-MEDIA php:

Sep 19 22:00:41 Alex-MEDIA kernel: BTRFS info (device sdd1): csum failed ino 258 off 1611481088 csum 516816771 expected csum 1713725129

Sep 19 22:00:41 Alex-MEDIA kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 56, flush 0, corrupt 0, gen 0

Does anyone have any trouble shooting ideas?

Thanks for any help!

Quote

September 20, 201411 yr

On the dockerman page under extensions, what does your docker volume info say?

Quote

September 20, 201411 yr

Author

Hello,

It says:

Label: none uuid: 60e76ea4-1d88-4147-bdb8-108db68f85ff
Total devices 1 FS bytes used 1.86GiB

devid 1 size 10.00GiB used 5.04GiB path /dev/loop8

Btrfs v3.16

Quote

September 20, 201411 yr

K. Can you try clicking the log button on the top right of the webGUI and then click the scrub button on that docker page. Keep both windows open and copy / paste them here when the scrub completes if you are still having an issue...

Quote

September 20, 201411 yr

Author

Hello -- got this when clicking scrub from the docker page.

btrfs scrub start /var/lib/docker -B -R -d -r 2>&1
ERROR: scrubbing /var/lib/docker failed for device id 1 (Input/output error)

scrub device /dev/loop8 (id 1) canceled

scrub started at Sat Sep 20 09:06:27 2014 and was aborted after 0 seconds

data_extents_scrubbed: 0

tree_extents_scrubbed: 0

data_bytes_scrubbed: 0

tree_bytes_scrubbed: 0

read_errors: 0

csum_errors: 0

verify_errors: 0

no_csum: 0

csum_discards: 0

super_errors: 0

malloc_errors: 0

uncorrectable_errors: 0

unverified_errors: 0

corrected_errors: 0

last_physical: 0

Nothing shows up in the log

Quote

September 20, 201411 yr

Need some time to ponder this one..

Quote

September 20, 201411 yr

Author

I did a scrub on my cache drive where docker is installed -- I don't know if this will help you

btrfs scrub start /dev/sdd1 -B -R -d -r 2>&1
scrub device /dev/sdd1 (id 1) done

scrub started at Sat Sep 20 09:07:00 2014 and finished after 562 seconds

data_extents_scrubbed: 739844

tree_extents_scrubbed: 42578

data_bytes_scrubbed: 39747026944

tree_bytes_scrubbed: 697597952

read_errors: 0

csum_errors: 258

verify_errors: 0

no_csum: 1418687

csum_discards: 91176

super_errors: 0

malloc_errors: 0

uncorrectable_errors: 0

unverified_errors: 0

corrected_errors: 0

last_physical: 142845411328

Sep 20 09:11:33 Alex-MEDIA kernel: scrub_handle_errored_block: 246 callbacks suppressed
Sep 20 09:11:33 Alex-MEDIA kernel: BTRFS: checksum error at logical 19632574464 on dev /dev/sdd1, sector 40458408, root 5, inode 258, offset 4362137600, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:11:33 Alex-MEDIA kernel: btrfs_dev_stat_print_on_error: 246 callbacks suppressed

Sep 20 09:11:33 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 775, gen 0

Sep 20 09:11:33 Alex-MEDIA kernel: BTRFS: checksum error at logical 19632578560 on dev /dev/sdd1, sector 40458416, root 5, inode 258, offset 4362141696, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:11:33 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 776, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28241952768 on dev /dev/sdd1, sector 57273600, root 5, inode 258, offset 537395200, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 777, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242018304 on dev /dev/sdd1, sector 57273728, root 5, inode 258, offset 537460736, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 778, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28241956864 on dev /dev/sdd1, sector 57273608, root 5, inode 258, offset 537399296, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 779, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242022400 on dev /dev/sdd1, sector 57273736, root 5, inode 258, offset 537464832, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 780, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28241960960 on dev /dev/sdd1, sector 57273616, root 5, inode 258, offset 537403392, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 781, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242026496 on dev /dev/sdd1, sector 57273744, root 5, inode 258, offset 537468928, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 782, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28241965056 on dev /dev/sdd1, sector 57273624, root 5, inode 258, offset 537407488, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 783, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242030592 on dev /dev/sdd1, sector 57273752, root 5, inode 258, offset 537473024, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 784, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28241969152 on dev /dev/sdd1, sector 57273632, root 5, inode 258, offset 537411584, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 785, gen 0

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242034688 on dev /dev/sdd1, sector 57273760, root 5, inode 258, offset 537477120, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:14 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 786, gen 0

Sep 20 09:12:19 Alex-MEDIA kernel: scrub_handle_errored_block: 237 callbacks suppressed

Sep 20 09:12:19 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242247680 on dev /dev/sdd1, sector 57274176, root 5, inode 258, offset 537690112, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:19 Alex-MEDIA kernel: btrfs_dev_stat_print_on_error: 237 callbacks suppressed

Sep 20 09:12:19 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1025, gen 0

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242251776 on dev /dev/sdd1, sector 57274184, root 5, inode 258, offset 537694208, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1026, gen 0

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242255872 on dev /dev/sdd1, sector 57274192, root 5, inode 258, offset 537698304, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1027, gen 0

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242259968 on dev /dev/sdd1, sector 57274200, root 5, inode 258, offset 537702400, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1028, gen 0

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242264064 on dev /dev/sdd1, sector 57274208, root 5, inode 258, offset 537706496, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1029, gen 0

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242268160 on dev /dev/sdd1, sector 57274216, root 5, inode 258, offset 537710592, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1030, gen 0

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242272256 on dev /dev/sdd1, sector 57274224, root 5, inode 258, offset 537714688, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1031, gen 0

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: checksum error at logical 28242276352 on dev /dev/sdd1, sector 57274232, root 5, inode 258, offset 537718784, length 4096, links 1 (path: docker/docker.img)

Sep 20 09:12:20 Alex-MEDIA kernel: BTRFS: bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 1032, gen 0

Quote

September 20, 201411 yr

So after the scrub completed, you still having problems?

Quote

September 20, 201411 yr

Author

yup, same issue

Quote

September 20, 201411 yr

OK, and the drive to which you have your docker.IMG file placed isn't having any other issues or problems?

This looks like a bug potentially. I'm going to move this thread to defect reports.

Quote

September 20, 201411 yr

Now short term you could try stopping docker, changing the path to where you want your docker image installed, and see if the problems repeat.

Quote

September 20, 201411 yr

Author

No other issues that I am aware of. Just did a quick read/write test to it and that seems to be fine.

Will give the new docker path a shot and report back

Quote

September 20, 201411 yr

Author

Set up a new docker and it seems to be working there.

Quote

September 20, 201411 yr

Set up a new docker and it seems to be working there.

So, not a bug?

Quote

September 20, 201411 yr

Well not sure. Maybe let's see if the corruption occurs again.

Quote

October 2, 201411 yr

Has any further corruption occurred since we last discussed this topic? If I don't see any replies to this from the OP or anyone else with this issue, I'll chalk this up as a "fluke" and mark solved. Thanks.

Quote

October 20, 201411 yr

I have experienced the same issue multiple times by now...

I recreated the docker.img file two days ago (redownloaded all docker apps) and now my syslog gets filled with "Oct 20 10:57:54 Tower kernel: BTRFS: bad tree block start 0 961789952" and the same output as above when trying to scrub.

the docker.img is on my cache disk (which is formatted as xfs, after i had similar troubles with the cache disk formatted as btrfs)

currently the apps are running and working, but if I want to install a new app, I get an error about a read only file system.

I'll recreate the docker.img this evening (at work currently) and see, if it happens again.

Docker Volume Info:

Label: none uuid: 3a6c101d-875f-4b30-9bd4-e3efe1f97e0f

Total devices 1 FS bytes used 1.78GiB

devid 1 size 25.00GiB used 4.04GiB path /dev/loop8

Btrfs v3.16.1

Scrub Output:

btrfs scrub start /var/lib/docker -B -R -d -r 2>&1

ERROR: scrubbing /var/lib/docker failed for device id 1 (Input/output error)

scrub device /dev/loop8 (id 1) canceled

scrub started at Mon Oct 20 11:00:46 2014 and was aborted after 0 seconds

data_extents_scrubbed: 0

tree_extents_scrubbed: 0

data_bytes_scrubbed: 0

tree_bytes_scrubbed: 0

read_errors: 0

csum_errors: 0

verify_errors: 0

no_csum: 0

csum_discards: 0

super_errors: 0

malloc_errors: 0

uncorrectable_errors: 0

unverified_errors: 0

corrected_errors: 0

last_physical: 0

Quote

December 14, 201411 yr

I have a similar issue.

Label: none uuid: 770bfc48-e24a-45ed-a9a1-966153371173

Total devices 1 FS bytes used 2.22GiB

devid 1 size 20.00GiB used 12.04GiB path /dev/loop8

Btrfs v3.17.2

Scrub Output:

Dec 14 20:43:00 Tower emhttp: get_filesystem_status: statfs: /mnt/user/backup Stale NFS file handle

Dec 14 20:43:00 Tower emhttp: get_filesystem_status: statfs: /mnt/user/splunk Stale NFS file handle

Dec 14 20:43:00 Tower kernel: btrfs_dev_stat_print_on_error: 80 callbacks suppressed

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2062, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2063, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS info (device sdh1): no csum found for inode 3135 start 683868160

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2064, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2065, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS info (device loop8): no csum found for inode 263 start 0

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2066, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2067, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2068, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS info (device sdh1): no csum found for inode 3135 start 683868160

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2069, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2070, flush 1, corrupt 0, gen 0

Dec 14 20:43:00 Tower kernel: BTRFS info (device loop8): no csum found for inode 263 start 0

Dec 14 20:43:00 Tower kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 2071, flush 1, corrupt 0, gen 0