Docker Service Failed to Start - Acompanied by Errors


Recommended Posts

Docker service is suddenly failing to start. It is set to use the cache. No SMART errors on the cache array and no errors on scub. 131GB free space.

 

I am getting many errors like so when starting the Docker service:

 

Quote

Mar 2 16:42:43 MediaVault kernel: BTRFS info (device loop2): has skinny extents
Mar 2 16:42:43 MediaVault kernel: lo_write_bvec: 74 callbacks suppressed
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 56836096, length 4096.
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 65273856, length 4096.
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 69304320, length 4096.
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 69844992, length 4096.
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 98959360, length 4096.
Mar 2 16:42:43 MediaVault kernel: print_req_error: 74 callbacks suppressed
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 111008
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 100253696, length 4096.
Mar 2 16:42:43 MediaVault kernel: btrfs_dev_stat_print_on_error: 74 callbacks suppressed
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 100696064, length 4096.
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 127488
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 103006208, length 4096.
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 135360
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 136416
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 325271552, length 4096.
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 193280
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 333709312, length 4096.
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 195808
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 196672
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 201184
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 635296
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 651776
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Mar 2 16:42:43 MediaVault kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2259: errno=-5 IO failure (Error while writing out transaction)
Mar 2 16:42:43 MediaVault kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Mar 2 16:42:43 MediaVault kernel: BTRFS: error (device loop2) in cleanup_transaction:1860: errno=-5 IO failure
Mar 2 16:42:43 MediaVault kernel: BTRFS info (device loop2): delayed_refs has NO entry
Mar 2 16:42:43 MediaVault kernel: BTRFS: error (device loop2) in btrfs_replay_log:2281: errno=-5 IO failure (Failed to recover log tree)
Mar 2 16:42:43 MediaVault root: mount: /var/lib/docker: can't read superblock on /dev/loop2.
Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): open_ctree failed
Mar 2 16:42:43 MediaVault root: mount error

I have attached logs. Given that I have no drive errors, and lots of free space, am I probably looking at a corrupted docker image here?

 

Edit: Doing a lot of googling, it seems that these errors are probably related to docker corruption from one thing or another.

 

Also, is there anything I can do in order to get a response and or help from someone? I am kinda new asking for support on the forums but over the last year I have made several posts requesting assistance and I never get so much as a reply. Is there something else I need to be doing in order to get some help?

mediavault-diagnostics-20200302-1425.zip

Edited by Civrock
Link to comment
10 hours ago, Civrock said:

Given that I have no drive errors, and lots of free space,

This is the problem, you're using 2 different size devices on the raid1 cache pool, despite the GUI showing free space you don't have any, since only the size of the smallest device is usable in raid1.

Link to comment
10 hours ago, Civrock said:

Also, is there anything I can do in order to get a response and or help from someone? I am kinda new asking for support on the forums but over the last year I have made several posts requesting assistance and I never get so much as a reply. Is there something else I need to be doing in order to get some help?

You usually just need to wait, the forum support is done mostly by volunteers on their free time, so a little patience is needed, if you don't get a response in 24H you can bump it.

Link to comment
8 hours ago, johnnie.black said:

This is the problem, you're using 2 different size devices on the raid1 cache pool, despite the GUI showing free space you don't have any, since only the size of the smallest device is usable in raid1.

Well, crap. I was going completely off what the GUI said because I thought it was a raid 0 configuration 😕 makes sense that I am having this issue if it is raid1.

 

So, I have a few questions:

 

- I have the appdata and docker shares set to prefer which I thought would allow the docker service to write to the array in the event that the cache was full. Is that not the case here?

- What is the best way to go about replacing the smaller drive with a larger one so that I have 2 drives of the same size? Since it is in a mirror configuration, do I just replace the the smaller drive and let it rebuild or is it more complicated than that?

- Once I get everything sorted, is there any way to turn the cache pool into a raid0 configuration in order to get more performance out of it? I am already backing up dockers and appdata onto the main array regularly so losing cheap cache drives would not be the end of the world for me?

Link to comment
13 hours ago, Civrock said:

I thought it was a raid 0 configuration

Raid0 also wouldn't use the full available space, see here for more info.

 

13 hours ago, Civrock said:

I have the appdata and docker shares set to prefer which I thought would allow the docker service to write to the array in the event that the cache was full. Is that not the case here?

Docker image is a single file, it can't be split to the array, appdata can use the array if the minimum free space settings is correctly set, though not really recommended to split appdata between cache and array.

 

13 hours ago, Civrock said:

- What is the best way to go about replacing the smaller drive with a larger one so that I have 2 drives of the same size? Since it is in a mirror configuration, do I just replace the the smaller drive and let it rebuild or is it more complicated than that?

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=480419

 

13 hours ago, Civrock said:

Once I get everything sorted, is there any way to turn the cache pool into a raid0 configuration in order to get more performance out of it? I am already backing up dockers and appdata onto the main array regularly so losing cheap cache drives would not be the end of the world for me?

Explained on the first link posted above.

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.