Civrock Posted March 3, 2020 Share Posted March 3, 2020 (edited) Docker service is suddenly failing to start. It is set to use the cache. No SMART errors on the cache array and no errors on scub. 131GB free space. I am getting many errors like so when starting the Docker service: Quote Mar 2 16:42:43 MediaVault kernel: BTRFS info (device loop2): has skinny extents Mar 2 16:42:43 MediaVault kernel: lo_write_bvec: 74 callbacks suppressed Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 56836096, length 4096. Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 65273856, length 4096. Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 69304320, length 4096. Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 69844992, length 4096. Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 98959360, length 4096. Mar 2 16:42:43 MediaVault kernel: print_req_error: 74 callbacks suppressed Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 111008 Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 100253696, length 4096. Mar 2 16:42:43 MediaVault kernel: btrfs_dev_stat_print_on_error: 74 callbacks suppressed Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 100696064, length 4096. Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 127488 Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 103006208, length 4096. Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 135360 Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 136416 Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 325271552, length 4096. Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 193280 Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: loop: Write error at byte offset 333709312, length 4096. Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 195808 Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 196672 Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 201184 Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 635296 Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: print_req_error: I/O error, dev loop2, sector 651776 Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0 Mar 2 16:42:43 MediaVault kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2259: errno=-5 IO failure (Error while writing out transaction) Mar 2 16:42:43 MediaVault kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction. Mar 2 16:42:43 MediaVault kernel: BTRFS: error (device loop2) in cleanup_transaction:1860: errno=-5 IO failure Mar 2 16:42:43 MediaVault kernel: BTRFS info (device loop2): delayed_refs has NO entry Mar 2 16:42:43 MediaVault kernel: BTRFS: error (device loop2) in btrfs_replay_log:2281: errno=-5 IO failure (Failed to recover log tree) Mar 2 16:42:43 MediaVault root: mount: /var/lib/docker: can't read superblock on /dev/loop2. Mar 2 16:42:43 MediaVault kernel: BTRFS error (device loop2): open_ctree failed Mar 2 16:42:43 MediaVault root: mount error I have attached logs. Given that I have no drive errors, and lots of free space, am I probably looking at a corrupted docker image here? Edit: Doing a lot of googling, it seems that these errors are probably related to docker corruption from one thing or another. Also, is there anything I can do in order to get a response and or help from someone? I am kinda new asking for support on the forums but over the last year I have made several posts requesting assistance and I never get so much as a reply. Is there something else I need to be doing in order to get some help? mediavault-diagnostics-20200302-1425.zip Edited March 3, 2020 by Civrock Quote Link to comment
JorgeB Posted March 3, 2020 Share Posted March 3, 2020 10 hours ago, Civrock said: Given that I have no drive errors, and lots of free space, This is the problem, you're using 2 different size devices on the raid1 cache pool, despite the GUI showing free space you don't have any, since only the size of the smallest device is usable in raid1. Quote Link to comment
JorgeB Posted March 3, 2020 Share Posted March 3, 2020 10 hours ago, Civrock said: Also, is there anything I can do in order to get a response and or help from someone? I am kinda new asking for support on the forums but over the last year I have made several posts requesting assistance and I never get so much as a reply. Is there something else I need to be doing in order to get some help? You usually just need to wait, the forum support is done mostly by volunteers on their free time, so a little patience is needed, if you don't get a response in 24H you can bump it. Quote Link to comment
Civrock Posted March 3, 2020 Author Share Posted March 3, 2020 8 hours ago, johnnie.black said: This is the problem, you're using 2 different size devices on the raid1 cache pool, despite the GUI showing free space you don't have any, since only the size of the smallest device is usable in raid1. Well, crap. I was going completely off what the GUI said because I thought it was a raid 0 configuration 😕 makes sense that I am having this issue if it is raid1. So, I have a few questions: - I have the appdata and docker shares set to prefer which I thought would allow the docker service to write to the array in the event that the cache was full. Is that not the case here? - What is the best way to go about replacing the smaller drive with a larger one so that I have 2 drives of the same size? Since it is in a mirror configuration, do I just replace the the smaller drive and let it rebuild or is it more complicated than that? - Once I get everything sorted, is there any way to turn the cache pool into a raid0 configuration in order to get more performance out of it? I am already backing up dockers and appdata onto the main array regularly so losing cheap cache drives would not be the end of the world for me? Quote Link to comment
JorgeB Posted March 4, 2020 Share Posted March 4, 2020 13 hours ago, Civrock said: I thought it was a raid 0 configuration Raid0 also wouldn't use the full available space, see here for more info. 13 hours ago, Civrock said: I have the appdata and docker shares set to prefer which I thought would allow the docker service to write to the array in the event that the cache was full. Is that not the case here? Docker image is a single file, it can't be split to the array, appdata can use the array if the minimum free space settings is correctly set, though not really recommended to split appdata between cache and array. 13 hours ago, Civrock said: - What is the best way to go about replacing the smaller drive with a larger one so that I have 2 drives of the same size? Since it is in a mirror configuration, do I just replace the the smaller drive and let it rebuild or is it more complicated than that? https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=480419 13 hours ago, Civrock said: Once I get everything sorted, is there any way to turn the cache pool into a raid0 configuration in order to get more performance out of it? I am already backing up dockers and appdata onto the main array regularly so losing cheap cache drives would not be the end of the world for me? Explained on the first link posted above. 1 Quote Link to comment
Civrock Posted March 4, 2020 Author Share Posted March 4, 2020 Right on. Thanks for the explainations and your patience. I will get to reading! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.