Nah, just the appdata share since it seems to be the template default.

Squid is already on it with the latest version of Community Applications plugin.

Experimental (though it shouldn't cause any issues anywhere) and optional.

I updated my UNRAID version to 6.0.1 in the hope that would eventually fix any Docker issues (even though there is nothing in the release notes about this) but the problem persists.

 

I can reproduce a "crash" of my containers and provide any other logs and diagnostics if this would help?

Hopefully not too off topic as it is related to my cache disk and therefore potentially my docker image.

 

I wanted to replace my cache disk to eliminate this as the source of error, the only files that I have there that I want to keep are my VMs.

One VM copied fine but I'm getting errors during the copy of the other VM (cp: error reading ‘vdisk1.img’: Input/output error) I checked the destination and specified the disk during the copy and there is plenty of room.

 

Therefore, does anyone have am idea of how I can force the copy?

Could this indicate an error with the cache disk? I ran a SMART disk check and this came up clean.

Thanks

 

For those having no docker tab, did you check the following:

Do not have the old Dockerman plugin

Do not have a corrupted or full docker.img file

Do not have a full log system

 

For the last 2 points use the following command

df -h

 

Then look at /var/log and /var/lib/docker (i think thats the locations)

After a reboot the docker tab came back, however the problems with containers crashing persist.

 

I installed v6.0.0 cleanly, formatting the flash drive beforehand, so there were no plugins.

 

I've recreated the docker.img a few times, however I'm sure you've seen my previous posts explaining, docker runs fine for a time upon rebooting then the issues come back.

 

How do I check if the system logs are full?

 

There's plenty of room on the flash drive.

df -h gives this,

 

Filesystem      Size  Used Avail Use% Mounted on

tmpfs          128M  40M  89M  31% /var/log

/dev/sda1      3.8G  290M  3.5G  8% /boot

/dev/md1        1.9T  1.9T  696M 100% /mnt/disk1

/dev/md2        1.9T  1.9T  13G 100% /mnt/disk2

/dev/md3        1.9T  1.5T  407G  79% /mnt/disk3

/dev/md4        3.7T  3.2T  466G  88% /mnt/disk4

/dev/md5        1.9T  1.8T  21G  99% /mnt/disk5

/dev/md6        1.9T  1.9T  14G 100% /mnt/disk6

/dev/md7        1.9T  1.8T  42G  98% /mnt/disk7

/dev/md8        1.9T  1.8T  84G  96% /mnt/disk8

/dev/md9        1.9T  1.8T  54G  98% /mnt/disk9

/dev/md10      1.9T  1.9T  1.7G 100% /mnt/disk10

/dev/md11      1.9T  1.8T  30G  99% /mnt/disk11

/dev/md12      1.9T  1.8T  54G  98% /mnt/disk12

/dev/md13      1.9T  1.4T  463G  76% /mnt/disk13

/dev/md14      1.9T  934G  930G  51% /mnt/disk14

/dev/md15      1.9T  1.1T  754G  60% /mnt/disk15

/dev/sdn1      466G  180G  286G  39% /mnt/cache

shfs            30T  26T  3.3T  89% /mnt/user0

shfs            30T  27T  3.6T  89% /mnt/user

/dev/loop1      1.8M  89K  1.6M  6% /etc/libvirt

/dev/loop0      15G  2.3G  12G  17% /var/lib/docker

 

/var/log/syslog gives this, (the last few lines)

 

Jul  4 15:19:32 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:19:32 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:19:32 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:30:36 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:30:36 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:30:36 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:31:41 msstorage kernel: mdcmd (353): spindown 3

Jul  4 15:31:49 msstorage kernel: mdcmd (354): spindown 9

Jul  4 15:31:57 msstorage kernel: mdcmd (355): spindown 7

Jul  4 15:32:04 msstorage kernel: mdcmd (356): spindown 6

Jul  4 15:32:04 msstorage kernel: mdcmd (357): spindown 10

Jul  4 15:32:22 msstorage kernel: mdcmd (358): spindown 5

Jul  4 15:32:22 msstorage kernel: mdcmd (359): spindown 11

Jul  4 15:34:01 msstorage kernel: mdcmd (360): spindown 1

Jul  4 15:34:01 msstorage kernel: mdcmd (361): spindown 4

Jul  4 15:39:10 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:39:10 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:39:10 msstorage kernel: BTRFS warning (device sdn1): csum failed ino 273 off 7155044352 csum 2214257214 expected csum 3520190875

Jul  4 15:40:55 msstorage kernel: verify_parent_transid: 2 callbacks suppressed

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 127549440 wanted 2290 found 1968

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 127549440 wanted 2290 found 1968

Jul  4 15:40:55 msstorage kernel: BTRFS info (device loop0): no csum found for inode 264 start 0

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 127549440 wanted 2290 found 1968

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 127549440 wanted 2290 found 1968

Jul  4 15:40:55 msstorage kernel: BTRFS info (device loop0): no csum found for inode 264 start 4096

Jul  4 15:40:55 msstorage kernel: BTRFS warning (device loop0): csum failed ino 264 off 0 csum 3551632587 expected csum 0

Jul  4 15:40:55 msstorage kernel: BTRFS warning (device loop0): csum failed ino 264 off 4096 csum 3166042738 expected csum 0

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 126992384 wanted 2290 found 2289

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 126992384 wanted 2290 found 2289

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 126992384 wanted 2290 found 2289

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 126992384 wanted 2290 found 2289

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 127926272 wanted 2290 found 1969

Jul  4 15:40:55 msstorage kernel: BTRFS (device loop0): parent transid verify failed on 127926272 wanted 2290 found 1969

Jul  4 15:40:57 msstorage kernel: mdcmd (362): spindown 2

Jul  4 15:40:58 msstorage kernel: mdcmd (363): spindown 14

Jul  4 16:00:39 msstorage kernel: mdcmd (364): spindown 13

 

/var/lib/docker, hmmm this is interesting, I tried to list the files here and have the same error as when I try to copy from the cache disk.

 

root@msstorage:/var/lib/docker# ls

/bin/ls: reading directory .: Input/output error

 

 

Whatever system you are using BTRFS on is corrupted. One such system everyone has is the docker.img is internally a BTRFS filesystem. You can try running a scrub on it to fix it, but i have never seen anyone have any luck on uncorrupting it other than having to delete it and recreate it.

 

Some say BTRFS is stable and mature, but i can't think it is when there are so many corruptions reported that are unfixable.

Thanks BRiT.

 

I don't care about the docker.img but I would really like my VM back if possible.

 

I'm running the scrub now and post the results once finished.

Does anyone run docker on a non BTRFS file system?

What are your experiences?

 

I believe I remember reading that this file system was better to compression or maybe de-duplication but I'm willing to sacrifice that on a more stable file system for docker.

Does anyone run docker on a non BTRFS file system?

What are your experiences?

 

I believe I remember reading that this file system was better to compression or maybe de-duplication but I'm willing to sacrifice that on a more stable file system for docker.

 

On unRAID it's not possible without great trouble to run docker on non BTRFS since LimeTech's implementation uses BTRFS inside the "docker.img" file.

 

Now if you meant what filesystem is used to store the docker.img, then I have that on a XFS cache drive.

Ah ok, If I understand then the filesystem inside the docker file is BTRFS and the docker image can be stored on any file system?

 

Thanks for the info.

Ah ok, If I understand then the filesystem inside the docker file is BTRFS and the docker image can be stored on any file system?

 

Thanks for the info.

 

Correct.

I'm wondering if all of the host paths are set up correctly?  Could improper host paths where the containers are storing their metadata within docker.img be causing the image to fill up and corrupt?

 

The path I use for container configuration is

/mnt/disk15/Docker-Configs.

 

Is this wrong?

 

Here is the result from the read only scrub (This is from the disk management page and not docker)

 

scrub started at Sat Jul  4 16:57:41 2015, running for 2544 seconds

total bytes scrubbed: 179.39GiB with 13 errors

error details: csum=13

corrected errors: 0, uncorrectable errors: 0, unverified errors: 0

 

Would you advise that I redo this and try to correct errors?

 

Would this in any way create more of a risk to my VM that I can't copy?

 

Thanks

Whatever system you are using BTRFS on is corrupted. One such system everyone has is the docker.img is internally a BTRFS filesystem. You can try running a scrub on it to fix it, but i have never seen anyone have any luck on uncorrupting it other than having to delete it and recreate it.

 

Some say BTRFS is stable and mature, but i can't think it is when there are so many corruptions reported that are unfixable.

 

I guess you were right :-(

 

scrub started at Sat Jul  4 19:17:11 2015 and finished after 2531 seconds

total bytes scrubbed: 179.49GiB with 13 errors

error details: csum=13

corrected errors: 0, uncorrectable errors: 13, unverified errors: 0

So after many hours of trying different methods of copying the VM I wanted off my cache drive I gave up after all the errors.

I could however still run the VM without any visible issues and I backed up the server from within the guest OS.

 

I changed my cache drive to be sure (although I believe the error to have been on the BTRFS file system and not the disk) as I had a spare 500GB disk doing nothing.

 

I formated this to XFS and re-did all my VMs and docker containers, 2 days running and so far so good.

 

Thanks for the advice given here and I won't be using BTRFS on my cache drive any time soon