BLKMGK Posted May 22, 2019 Share Posted May 22, 2019 I'm no Docker expert but I'm attempting to add my unRAID server into a Docker Swarm I've got configured on my network. I'm able to join but I get network errors when I attempt to coordinate client\worker containers on my unRAID server. There's a script for checking the Docker configuration named check-config.sh and when I run this it looks like just a single module is missing and near as I can tell it explains my networking issues. I've seen other threads where people have compiled their own kernels to get around this but since I also run the NVIDIA kernel it's not something I'd like to tackle myself. We've got pretty slick container support now and I'm hoping this module was simply one that wasn't thought of when compilation was done for some reason. The line that gets printed as missing when I check the config is as follows: CONFIG_NETFILTER_XT_MATCH_IPVS: missing This comes out of the "generally needed" section and is the only module there not installed. Thanks! 1 Quote Link to comment
BLKMGK Posted May 22, 2019 Author Share Posted May 22, 2019 The script to check the Docker configuration can be found here-> https://github.com/moby/moby/blob/master/contrib/check-config.sh 1 Quote Link to comment
seer_tenedos Posted May 25, 2019 Share Posted May 25, 2019 I would love this to be added as well. Maybe the team building the NVIDIA kernel could add it if unraid is unwilling to add it though it would be great to know why they don't want to add it. 1 Quote Link to comment
CHBMB Posted May 25, 2019 Share Posted May 25, 2019 @bonienl This topic has been bumped in three different places this morning, here, here, and here. I would suggest the topic here is the most appropriate one as it actually defines the kernel requirements to run swarm, although I haven't tested (and am not going to as I have zero interest in swarm on Unraid.) From an Unraid-Nvidia plugin I can state that my position is we won't add it. The project was never designed to modify Unraid any more than add drivers and enable the ability to utilise them in containers, and I'm not modifying .config every time I build a new release as I have to do that with the DVB kernel releases and it's a PITA. If @limetech wish to include the module in the kernel then Unraid-Nvidia will naturally follow being downstream. 1 Quote Link to comment
bonienl Posted May 25, 2019 Share Posted May 25, 2019 @CHBMB LOL that's asking for a lot attention... but indeed no reason to make different requests for the same thing. 1 Quote Link to comment
seer_tenedos Posted May 25, 2019 Share Posted May 25, 2019 @CHBMB while I would love @limetech to add this I honestly doubt they ever will. This has been requested by different people for the last 2 years atleast with no movement or response. I was just hoping that the NVIDIA plugin may have a way to work around these sort of issues with kernel modules but sadly it sounds like that is not the case. In my case I am not looking to create a single node swarm to use some of the serverless and other technology that will not run if the docker engine is not in swarm mode or on kubernetes. Guess the only way is to do this all on a VM that runs full features docker so I can run as a single node swarm. Quote Link to comment
CHBMB Posted May 25, 2019 Share Posted May 25, 2019 (edited) In my experience, @limetech are far more likely to respond to a request which clearly states the requirement, which, in this case is a kernel module, than just a plain "Can we get docker swarm?" Problem with the second question is it's an unknown quantity of what needs to be done and why this is important? Just remember Unraid can't be everything to everyone, and whilst some decisions might seem arbitrary, there may be good reasons why stuff isn't changed upstream. For instance if adding that kernel module breaks something else, it's a no go. As some useful information, here's the output of the above script. root@server:~# wget https://raw.githubusercontent.com/moby/moby/master/contrib/check-config.sh --2019-05-25 12:00:31-- https://raw.githubusercontent.com/moby/moby/master/contrib/check-config.sh Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.16.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.16.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10314 (10K) [text/plain] Saving to: ‘check-config.sh’ check-config.sh 100%[===================>] 10.07K --.-KB/s in 0s 2019-05-25 12:00:31 (208 MB/s) - ‘check-config.sh’ saved [10314/10314] root@server:~# chmod +x check-config.sh root@server:~# ./check-config.sh warning: /proc/config.gz does not exist, searching other paths for kernel config ... info: reading kernel config from /usr/src/linux-4.19.41-Unraid/.config ... Generally Necessary: - cgroup hierarchy: properly mounted [/sys/fs/cgroup] - CONFIG_NAMESPACES: enabled - CONFIG_NET_NS: enabled - CONFIG_PID_NS: enabled - CONFIG_IPC_NS: enabled - CONFIG_UTS_NS: enabled - CONFIG_CGROUPS: enabled - CONFIG_CGROUP_CPUACCT: enabled - CONFIG_CGROUP_DEVICE: enabled - CONFIG_CGROUP_FREEZER: enabled - CONFIG_CGROUP_SCHED: enabled - CONFIG_CPUSETS: enabled - CONFIG_MEMCG: enabled - CONFIG_KEYS: enabled - CONFIG_VETH: enabled (as module) - CONFIG_BRIDGE: enabled - CONFIG_BRIDGE_NETFILTER: enabled - CONFIG_NF_NAT_IPV4: enabled (as module) - CONFIG_IP_NF_FILTER: enabled (as module) - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled - CONFIG_NETFILTER_XT_MATCH_IPVS: missing - CONFIG_IP_NF_NAT: enabled (as module) - CONFIG_NF_NAT: enabled (as module) - CONFIG_NF_NAT_NEEDED: enabled - CONFIG_POSIX_MQUEUE: enabled Optional Features: - CONFIG_USER_NS: enabled - CONFIG_SECCOMP: enabled - CONFIG_CGROUP_PIDS: enabled - CONFIG_MEMCG_SWAP: enabled - CONFIG_MEMCG_SWAP_ENABLED: enabled (cgroup swap accounting is currently enabled) - CONFIG_LEGACY_VSYSCALL_EMULATE: enabled - CONFIG_BLK_CGROUP: enabled - CONFIG_BLK_DEV_THROTTLING: enabled - CONFIG_IOSCHED_CFQ: enabled - CONFIG_CFQ_GROUP_IOSCHED: enabled - CONFIG_CGROUP_PERF: enabled - CONFIG_CGROUP_HUGETLB: missing - CONFIG_NET_CLS_CGROUP: missing - CONFIG_CGROUP_NET_PRIO: missing - CONFIG_CFS_BANDWIDTH: enabled - CONFIG_FAIR_GROUP_SCHED: enabled - CONFIG_RT_GROUP_SCHED: enabled - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module) - CONFIG_IP_VS: missing - CONFIG_IP_VS_NFCT: missing - CONFIG_IP_VS_PROTO_TCP: missing - CONFIG_IP_VS_PROTO_UDP: missing - CONFIG_IP_VS_RR: missing - CONFIG_EXT4_FS: enabled (as module) - CONFIG_EXT4_FS_POSIX_ACL: enabled - CONFIG_EXT4_FS_SECURITY: missing enable these ext4 configs if you are using ext3 or ext4 as backing filesystem - Network Drivers: - "overlay": - CONFIG_VXLAN: enabled (as module) Optional (for encrypted networks): - CONFIG_CRYPTO: enabled - CONFIG_CRYPTO_AEAD: enabled - CONFIG_CRYPTO_GCM: enabled (as module) - CONFIG_CRYPTO_SEQIV: enabled - CONFIG_CRYPTO_GHASH: enabled (as module) - CONFIG_XFRM: enabled - CONFIG_XFRM_USER: enabled - CONFIG_XFRM_ALGO: enabled - CONFIG_INET_ESP: enabled - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled - "ipvlan": - CONFIG_IPVLAN: missing - "macvlan": - CONFIG_MACVLAN: enabled (as module) - CONFIG_DUMMY: missing - "ftp,tftp client in container": - CONFIG_NF_NAT_FTP: missing - CONFIG_NF_CONNTRACK_FTP: missing - CONFIG_NF_NAT_TFTP: enabled (as module) - CONFIG_NF_CONNTRACK_TFTP: enabled - Storage Drivers: - "aufs": - CONFIG_AUFS_FS: missing - "btrfs": - CONFIG_BTRFS_FS: enabled - CONFIG_BTRFS_FS_POSIX_ACL: enabled - "devicemapper": - CONFIG_BLK_DEV_DM: enabled (as module) - CONFIG_DM_THIN_PROVISIONING: enabled (as module) - "overlay": - CONFIG_OVERLAY_FS: enabled - "zfs": - /dev/zfs: missing - zfs command: missing - zpool command: missing Limits: - /proc/sys/kernel/keys/root_maxkeys: 1000000 root@server:~# Edited May 25, 2019 by CHBMB 1 Quote Link to comment
CHBMB Posted May 25, 2019 Share Posted May 25, 2019 Also examining the current v6.7.1rc1 kernel Show that the module is enabled. I'll have a bit more of a poke around now as I'm just a bit curious. Quote Link to comment
seer_tenedos Posted May 25, 2019 Share Posted May 25, 2019 Last time I tried to test swarm was 6.6.x I have not tried on the 6.7 version or the 6.7.1. Maybe I should. I will find my notes as it is a single command to turn it on it was just most of my docker networking died when I did before. Apart from that I did not notice other issues but only played a little with no networking. Everything came up fine but packets never made it. Quote Link to comment
CHBMB Posted May 25, 2019 Share Posted May 25, 2019 root@matrix:/mnt/disk1/dvb# patch /mnt/disk1/dvb/kernel/.config /mnt/disk1/dvb/swarm.patch patching file /mnt/disk1/dvb/kernel/.config root@matrix:/mnt/disk1/dvb# ./check-config.sh /mnt/disk1/dvb/kernel/.config info: reading kernel config from /mnt/disk1/dvb/kernel/.config ... Generally Necessary: - cgroup hierarchy: properly mounted [/sys/fs/cgroup] - CONFIG_NAMESPACES: enabled - CONFIG_NET_NS: enabled - CONFIG_PID_NS: enabled - CONFIG_IPC_NS: enabled - CONFIG_UTS_NS: enabled - CONFIG_CGROUPS: enabled - CONFIG_CGROUP_CPUACCT: enabled - CONFIG_CGROUP_DEVICE: enabled - CONFIG_CGROUP_FREEZER: enabled - CONFIG_CGROUP_SCHED: enabled - CONFIG_CPUSETS: enabled - CONFIG_MEMCG: enabled - CONFIG_KEYS: enabled - CONFIG_VETH: enabled (as module) - CONFIG_BRIDGE: enabled - CONFIG_BRIDGE_NETFILTER: enabled - CONFIG_NF_NAT_IPV4: enabled (as module) - CONFIG_IP_NF_FILTER: enabled (as module) - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module) - CONFIG_IP_NF_NAT: enabled (as module) - CONFIG_NF_NAT: enabled (as module) - CONFIG_NF_NAT_NEEDED: enabled - CONFIG_POSIX_MQUEUE: enabled Optional Features: - CONFIG_USER_NS: enabled - CONFIG_SECCOMP: enabled - CONFIG_CGROUP_PIDS: enabled - CONFIG_MEMCG_SWAP: enabled - CONFIG_MEMCG_SWAP_ENABLED: enabled (cgroup swap accounting is currently enabled) - CONFIG_LEGACY_VSYSCALL_EMULATE: enabled - CONFIG_BLK_CGROUP: enabled - CONFIG_BLK_DEV_THROTTLING: enabled - CONFIG_IOSCHED_CFQ: enabled - CONFIG_CFQ_GROUP_IOSCHED: enabled - CONFIG_CGROUP_PERF: enabled - CONFIG_CGROUP_HUGETLB: missing - CONFIG_NET_CLS_CGROUP: missing - CONFIG_CGROUP_NET_PRIO: missing - CONFIG_CFS_BANDWIDTH: enabled - CONFIG_FAIR_GROUP_SCHED: enabled - CONFIG_RT_GROUP_SCHED: enabled - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module) - CONFIG_IP_VS: enabled (as module) - CONFIG_IP_VS_NFCT: missing - CONFIG_IP_VS_PROTO_TCP: missing - CONFIG_IP_VS_PROTO_UDP: missing - CONFIG_IP_VS_RR: missing - CONFIG_EXT4_FS: enabled (as module) - CONFIG_EXT4_FS_POSIX_ACL: enabled - CONFIG_EXT4_FS_SECURITY: missing enable these ext4 configs if you are using ext3 or ext4 as backing filesystem - Network Drivers: - "overlay": - CONFIG_VXLAN: enabled (as module) Optional (for encrypted networks): - CONFIG_CRYPTO: enabled - CONFIG_CRYPTO_AEAD: enabled - CONFIG_CRYPTO_GCM: enabled (as module) - CONFIG_CRYPTO_SEQIV: enabled - CONFIG_CRYPTO_GHASH: enabled (as module) - CONFIG_XFRM: enabled - CONFIG_XFRM_USER: enabled - CONFIG_XFRM_ALGO: enabled - CONFIG_INET_ESP: enabled - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled - "ipvlan": - CONFIG_IPVLAN: missing - "macvlan": - CONFIG_MACVLAN: enabled (as module) - CONFIG_DUMMY: missing - "ftp,tftp client in container": - CONFIG_NF_NAT_FTP: missing - CONFIG_NF_CONNTRACK_FTP: missing - CONFIG_NF_NAT_TFTP: enabled (as module) - CONFIG_NF_CONNTRACK_TFTP: enabled - Storage Drivers: - "aufs": - CONFIG_AUFS_FS: missing - "btrfs": - CONFIG_BTRFS_FS: enabled - CONFIG_BTRFS_FS_POSIX_ACL: enabled - "devicemapper": - CONFIG_BLK_DEV_DM: enabled (as module) - CONFIG_DM_THIN_PROVISIONING: enabled (as module) - "overlay": - CONFIG_OVERLAY_FS: enabled - "zfs": - /dev/zfs: missing - zfs command: missing - zpool command: missing Limits: - /proc/sys/kernel/keys/root_maxkeys: 1000000 And here's a patch file for v6.7.0rc1 1139a1140 > CONFIG_NETFILTER_XT_MATCH_IPVS=m 1165c1166,1210 < # CONFIG_IP_VS is not set --- > CONFIG_IP_VS=m > # CONFIG_IP_VS_IPV6 is not set > # CONFIG_IP_VS_DEBUG is not set > CONFIG_IP_VS_TAB_BITS=12 > > # > # IPVS transport protocol load balancing support > # > # CONFIG_IP_VS_PROTO_TCP is not set > # CONFIG_IP_VS_PROTO_UDP is not set > # CONFIG_IP_VS_PROTO_ESP is not set > # CONFIG_IP_VS_PROTO_AH is not set > # CONFIG_IP_VS_PROTO_SCTP is not set > > # > # IPVS scheduler > # > # CONFIG_IP_VS_RR is not set > # CONFIG_IP_VS_WRR is not set > # CONFIG_IP_VS_LC is not set > # CONFIG_IP_VS_WLC is not set > # CONFIG_IP_VS_FO is not set > # CONFIG_IP_VS_OVF is not set > # CONFIG_IP_VS_LBLC is not set > # CONFIG_IP_VS_LBLCR is not set > # CONFIG_IP_VS_DH is not set > # CONFIG_IP_VS_SH is not set > # CONFIG_IP_VS_MH is not set > # CONFIG_IP_VS_SED is not set > # CONFIG_IP_VS_NQ is not set > > # > # IPVS SH scheduler > # > CONFIG_IP_VS_SH_TAB_BITS=8 > > # > # IPVS MH scheduler > # > CONFIG_IP_VS_MH_TAB_INDEX=12 > > # > # IPVS application helper > # > # CONFIG_IP_VS_NFCT is not set I haven't tested as I have no interest in swarm, but it can now be put to @limetech if they wish to update things upstreamswarm.patch Quote Link to comment
CHBMB Posted May 25, 2019 Share Posted May 25, 2019 30 minutes ago, seer_tenedos said: Last time I tried to test swarm was 6.6.x I have not tried on the 6.7 version or the 6.7.1. Maybe I should. I will find my notes as it is a single command to turn it on it was just most of my docker networking died when I did before. Apart from that I did not notice other issues but only played a little with no networking. Everything came up fine but packets never made it. There are two parameters that need changing IP_VS NETFILTER_XT_MATCH_IPVS Quote Link to comment
CHBMB Posted May 25, 2019 Share Posted May 25, 2019 v6.7.0rc1-swarm.zip Here's a version of v6.7.0rc1 with the swarm patch applied. The check config script will still show it as missing the above two modules but they are included, but the script checks the contents of /usr/src/linux-4.19.43-Unraid/.config which I haven't changed as that would require unpacking bzroot. Instead I applied the patch above and just recreated bzmodules and bzfirmware The proof will be if it works or not. v6.7.0rc1-swarm.zip 1 Quote Link to comment
BLKMGK Posted May 25, 2019 Author Share Posted May 25, 2019 Wow, this blew up overnight! As I stated above, I'm no Docker expert but I'm attempting to learn more and I DO have a use case for this. I've been relying on the check-config script and making an educated guess that this "missing" module was what was causing me issues. I understand however that it may not be properly checking our kernels and it's possible there's a different module that's needed for what I'm doing but this seemed a good place to start as it's networking that breaks for my use case. I would agree that having this done in an upstream fashion by @limetech is the way to go. I'm an avid user of the NVIDIA enhanced kernels but the last thing I'd like to see is more work placed upon @CHBMB as he's already got his hands full with crazy support questions and to add to it would be insane! He has the ability to add this but that's asking for too much IMO. I did briefly consider trying to compile and test a custom kernel myself as documentation to do it exists (sans NVIDIA) but I figured I'd ask first as that's a learning curve I'm not yet willing to climb if possible. I'm presently running 6.7.1 RC1 (NVIDIA) but I will attempt to test the v6.7.0RC1 kernel posted above and report back. It may take a day or so as I've got a lengthy list of jobs running right now that I'd like to complete first so anyone else able to help out I'd appreciate it. Either way I need to test to ensure my issue is solved with this addition or figure out what else might be blocking me. I've seen it asked elsewhere what containers would benefit from Swarm. One general thing that might be useful by enabling Swarm is DockerDNS which I don't *think* works right now. DockerDNS as I understand it is an internal Docker DNS system that gets turned on when Swarm is activated and it could prove generally useful as you can reference other containers by name internally. That's not why I want this however so I'll try to explain better what I'm up to in hopes that folks might be a little more eager 😎 ============begin long explanation you can skip=========== Like many I use my unRAID to store video. As such I try to be judicious in how I store it. I can rip a BD and end up with a 30GB file, I can then compress it down with x.265 and store it for a third of that or less while maintaining excellent video quality and not compromising on sound quality. The catch is that this requires some fairly hefty amounts of processing* and time investment. Most of us are familiar with HandBrake, some of us may be also be familiar with RipBot264. RipBot264 allows you to "cluster" Windows computers in order to encode individual videos more quickly. Each computer encodes portions of the video and it's joined back together at the end. RipBot264 is pretty well supported and mostly works well too - I use it on a few Windows machines. However I have a fairly decent amount of Linux hardware in my home and the developer of RipBot is unwilling to support Linux, he advised me to run VMs of Windows instead when I asked, I'm not willing to do so. I sat down and analyzed how the RipBot264 program does it's work, I studied ffmpeg and x.265 functionality, I asked ignorant questions, and I realized that duplicating what he was doing on Linux was completely possible with the exception of AVISynth**. I also realized that it could be done more efficiently without some of the intermediate steps that program takes. I built a proof of concept script, tested it, and it worked WELL! I happen to work with some talented guys and Docker is one of the technologies that we're beginning to use, Swarm is an area of interest too. I managed to interest some of my coworkers (one in particular) in my little project as I'm not a well versed programmer. Working together we've built a system that can cluster encode videos VERY well on a home lab I have setup that includes a 48thread machine***, a 24 thread ESX server, and a small PC tucked in the corner for Kodi use. What I've been unable to join (properly) into this cluster is my 32thread unRAID box 😢 End-state I'd like to end up with a container that can live in the unRAID "app" repo or be easily side-loaded that could allow users to join unRAID servers into a cluster for added compute power with our new toy. I'd tried this by hand from the commandline and was able to load a worker container and join our swarm but unable to get the container to receive any tasking or comms. This is when I began digging into what might be missing from the kernel etc. Thus I've arrived here! So YES, I have some need of unRAID more fully enabling Swarm functionality for our little project. As a side note - yes we intend to release our code to others, ALL of it, and yeah we will need some help in the future. My selfish hope is that when released to others with more skill than we possess features of interest can be built that we're interested in. We still have a few things to add and test before making it public. Yes, there are products that do this commercially - at eye watering prices. As yet I've found no built and supported OpenSource program duplicating this so we've scratched our own itch so to speak as is the OpenSource way! There, sorry for the lengthy dissertation lol *at present time I don't leverage GPU hardware for this but may begin leveraging it in the future, I'd want it just to speedup the math not run through the GPU's onboard encoder as I want as much control over quality as possible. For some reason RipBot refuses to use any of my current desktop GPU hardware - ugh. **see Vapoursynth for a Linux solution that we've yet to touch. AVISynth is pretty powerful though so yes eventually we'll want filtering ability. ***this is all older XEON hardware, when Ryzen 3000 is released I'll be upgrading ALL of it including my unRAID server 😈 Quote Link to comment
BLKMGK Posted May 25, 2019 Author Share Posted May 25, 2019 For reference here's what I get when I run the check-config script on a machine that Swarms just fine blkmgk@smaug:~$ ./check-config.sh warning: /proc/config.gz does not exist, searching other paths for kernel config ... info: reading kernel config from /boot/config-4.15.0-50-generic ... Generally Necessary: - cgroup hierarchy: properly mounted [/sys/fs/cgroup] - apparmor: enabled and tools installed - CONFIG_NAMESPACES: enabled - CONFIG_NET_NS: enabled - CONFIG_PID_NS: enabled - CONFIG_IPC_NS: enabled - CONFIG_UTS_NS: enabled - CONFIG_CGROUPS: enabled - CONFIG_CGROUP_CPUACCT: enabled - CONFIG_CGROUP_DEVICE: enabled - CONFIG_CGROUP_FREEZER: enabled - CONFIG_CGROUP_SCHED: enabled - CONFIG_CPUSETS: enabled - CONFIG_MEMCG: enabled - CONFIG_KEYS: enabled - CONFIG_VETH: enabled (as module) - CONFIG_BRIDGE: enabled (as module) - CONFIG_BRIDGE_NETFILTER: enabled (as module) - CONFIG_NF_NAT_IPV4: enabled (as module) - CONFIG_IP_NF_FILTER: enabled (as module) - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module) - CONFIG_IP_NF_NAT: enabled (as module) - CONFIG_NF_NAT: enabled (as module) - CONFIG_NF_NAT_NEEDED: enabled - CONFIG_POSIX_MQUEUE: enabled Optional Features: - CONFIG_USER_NS: enabled - CONFIG_SECCOMP: enabled - CONFIG_CGROUP_PIDS: enabled - CONFIG_MEMCG_SWAP: enabled - CONFIG_MEMCG_SWAP_ENABLED: missing (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1") - CONFIG_LEGACY_VSYSCALL_EMULATE: enabled - CONFIG_BLK_CGROUP: enabled - CONFIG_BLK_DEV_THROTTLING: enabled - CONFIG_IOSCHED_CFQ: enabled - CONFIG_CFQ_GROUP_IOSCHED: enabled - CONFIG_CGROUP_PERF: enabled - CONFIG_CGROUP_HUGETLB: enabled - CONFIG_NET_CLS_CGROUP: enabled (as module) - CONFIG_CGROUP_NET_PRIO: enabled - CONFIG_CFS_BANDWIDTH: enabled - CONFIG_FAIR_GROUP_SCHED: enabled - CONFIG_RT_GROUP_SCHED: missing - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module) - CONFIG_IP_VS: enabled (as module) - CONFIG_IP_VS_NFCT: enabled - CONFIG_IP_VS_PROTO_TCP: enabled - CONFIG_IP_VS_PROTO_UDP: enabled - CONFIG_IP_VS_RR: enabled (as module) - CONFIG_EXT4_FS: enabled - CONFIG_EXT4_FS_POSIX_ACL: enabled - CONFIG_EXT4_FS_SECURITY: enabled - Network Drivers: - "overlay": - CONFIG_VXLAN: enabled (as module) Optional (for encrypted networks): - CONFIG_CRYPTO: enabled - CONFIG_CRYPTO_AEAD: enabled - CONFIG_CRYPTO_GCM: enabled - CONFIG_CRYPTO_SEQIV: enabled - CONFIG_CRYPTO_GHASH: enabled - CONFIG_XFRM: enabled - CONFIG_XFRM_USER: enabled (as module) - CONFIG_XFRM_ALGO: enabled (as module) - CONFIG_INET_ESP: enabled (as module) - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module) - "ipvlan": - CONFIG_IPVLAN: enabled (as module) - "macvlan": - CONFIG_MACVLAN: enabled (as module) - CONFIG_DUMMY: enabled (as module) - "ftp,tftp client in container": - CONFIG_NF_NAT_FTP: enabled (as module) - CONFIG_NF_CONNTRACK_FTP: enabled (as module) - CONFIG_NF_NAT_TFTP: enabled (as module) - CONFIG_NF_CONNTRACK_TFTP: enabled (as module) - Storage Drivers: - "aufs": - CONFIG_AUFS_FS: enabled (as module) - "btrfs": - CONFIG_BTRFS_FS: enabled (as module) - CONFIG_BTRFS_FS_POSIX_ACL: enabled - "devicemapper": - CONFIG_BLK_DEV_DM: enabled - CONFIG_DM_THIN_PROVISIONING: enabled (as module) - "overlay": - CONFIG_OVERLAY_FS: enabled (as module) - "zfs": - /dev/zfs: missing - zfs command: missing - zpool command: missing Limits: - /proc/sys/kernel/keys/root_maxkeys: 1000000 Quote Link to comment
itimpi Posted May 25, 2019 Share Posted May 25, 2019 Just thought that it might be worth mentioning that if they are all running on the same machine you are not going to gaming anything by splitting up the video encoding tasks. Ffmpeg is already multi-processr aware and capable of using all available core to speed up encoding. Quote Link to comment
CHBMB Posted May 25, 2019 Share Posted May 25, 2019 44 minutes ago, BLKMGK said: One general thing that might be useful by enabling Swarm is DockerDNS which I don't *think* works right now. DockerDNS as I understand it is an internal Docker DNS system that gets turned on when Swarm is activated and it could prove generally useful as you can reference other containers by name internally. That's not why I want this however so I'll try to explain better what I'm up to in hopes that folks might be a little more eager 😎 Docker DNS as far as I know works fine as long as you create a custom docker network and run all the containers on that. It's what we use in the letsencrypt reverse proxy configs. Did a diff on that working config versus the default Unraid one. Essentially these are the missing components on Unraid CONFIG_NETFILTER_XT_MATCH_IPVS: missing CONFIG_CGROUP_HUGETLB: missing CONFIG_NET_CLS_CGROUP: missing CONFIG_CGROUP_NET_PRIO: missing CONFIG_IP_VS: missing CONFIG_IP_VS_NFCT: missing CONFIG_IP_VS_PROTO_TCP: missing CONFIG_IP_VS_PROTO_UDP: missing CONFIG_IP_VS_RR: missing CONFIG_EXT4_FS_SECURITY: missing CONFIG_IPVLAN: missing CONFIG_DUMMY: missing CONFIG_NF_NAT_FTP: missing CONFIG_NF_CONNTRACK_FTP: missing CONFIG_AUFS_FS: missing Which ones of these are important I couldn't say. But it does illustrate nicely that things aren't just as simple as flicking a switch to enable a single kernel module. I activated the top one in that build I made earlier in the this thread. How essential the others are to your requirements I don't know, and I'm reluctant to f**k with stuff I don't understand at a kernel level on the Nvidia build. Quote Link to comment
BLKMGK Posted May 25, 2019 Author Share Posted May 25, 2019 (edited) 34 minutes ago, itimpi said: Just thought that it might be worth mentioning that if they are all running on the same machine you are not going to gaming anything by splitting up the video encoding tasks. Ffmpeg is already multi-processr aware and capable of using all available core to speed up encoding. Sadly this isn't completely true. Even RipBot allows you to run multiple encoding workers on a single machine. Ffmpeg and x.265 are indeed multithreaded however they don't scale past about 6threads very well. The machine I have with 48threads was one I tested on and it was a failure to say the least. I had a few threads working but quite a few threads sat completely idle. That machine with multiple worker containers pegs all cores and the fans become quite noisy In any case, I currently have three machines in the cluster and would like to add my unRAID machine as a fourth! Edit: Also, Windows is going to be adding Docker support soon and we will make sure our workers can live on those. That would allow me to add at least two more machines including one that's a competitor to my 48thread system 😮 Edited May 25, 2019 by BLKMGK Moar Info! Quote Link to comment
BLKMGK Posted May 25, 2019 Author Share Posted May 25, 2019 19 minutes ago, CHBMB said: Docker DNS as far as I know works fine as long as you create a custom docker network and run all the containers on that. It's what we use in the letsencrypt reverse proxy configs. Did a diff on that working config versus the default Unraid one. Essentially these are the missing components on Unraid CONFIG_NETFILTER_XT_MATCH_IPVS: missing CONFIG_CGROUP_HUGETLB: missing CONFIG_NET_CLS_CGROUP: missing CONFIG_CGROUP_NET_PRIO: missing CONFIG_IP_VS: missing CONFIG_IP_VS_NFCT: missing CONFIG_IP_VS_PROTO_TCP: missing CONFIG_IP_VS_PROTO_UDP: missing CONFIG_IP_VS_RR: missing CONFIG_EXT4_FS_SECURITY: missing CONFIG_IPVLAN: missing CONFIG_DUMMY: missing CONFIG_NF_NAT_FTP: missing CONFIG_NF_CONNTRACK_FTP: missing CONFIG_AUFS_FS: missing Which ones of these are important I couldn't say. But it does illustrate nicely that things aren't just as simple as flicking a switch to enable a single kernel module. I activated the top one in that build I made earlier in the this thread. How essential the others are to your requirements I don't know, and I'm reluctant to f**k with stuff I don't understand at a kernel level on the Nvidia build. I appreciate you taking the time to diff these results! I had to run out the door after posting but I'm not surprised there's other differences. The module I highlighted is one of the modules listed as "common", the research I've done seemed to indicate it was a good place to start! I don't blame you for not wanting to tinker with the NVIDIA builds for sure. As it happens I've given a few unRAID systems to friends over the years as gifts, I've reached out to one friend who's not utilizing theirs heavily right now and will be bringing it home to test with using the build you posted earlier My fingers are crossed that I'll find success or at least answers as to what's needed and I can load test with it to see if weirdness occurs. I'm hoping this is enough, I recognize that enabling modules at the kernel level willy nilly isnt a great idea and am hoping that what I need proves benig to what Tom has in place and doesn't harm the KVM support either. I hope to test tonight if someone doesn't beat me to it! I really appreciate the support and responses guys! Quote Link to comment
BLKMGK Posted May 26, 2019 Author Share Posted May 26, 2019 I've got the test system up and running on the custom kernel, it's joined into the swarm (I could to that before tho), but I'm still having some issues that could very easily be my ignorance. A friend of mine who's been working with me on this is going to need to take a look at the error I see. From what I can tell it's a node labeling issue but I had thought I'd built it correctly in the yaml file. Anyway, the kernel appears stable and I'm hopeful that once I figure out my configuration issue (and no doubt learn something) that this will WORK. Fingers crossed and a big THANK YOU to @CHBMB for giving me something to test with! I'm hopeful and truly appreciative of the investigation he did on this as frankly it was beyond me. Stay tuned, as I know more I'll be sure and share it with everyone. If others have need of Swarm and could test their use cases that would be helpful too since I may not be pushing many boundaries here Quote Link to comment
BLKMGK Posted May 26, 2019 Author Share Posted May 26, 2019 So yup, I needed to label the node properly and then it began being available to the swarm, so that was good. However, after some troubleshooting inside the worker container we're still getting errors as it attempts to connect back to it's tasking engine. An error 113 - "no route found to host". Inside the container we can ping by hostname but the IP we get is crazy and seems to change depending upon the instantiation of the container. We are suspecting that Docker's weird DNS mechanisms may be confusing the troubleshooting. In any case we are unable to get a route back to the tasking database so the app fails, the worker cannot get work and never checks in. Not sure what's causing this. I believe tonight I may drop one of the other (not unRAID) nodes and go through the steps of re-adding it to ensure that something hasn't occurred with our container somehow. If it works perfectly with a standard Linux install I'm not sure what the next steps will be. I may try loading up a VM on unRAID, installing Docker, and trying that - this rings incredibly backwards to me however. Will try to troubleshoot this more tonight, it's driving me cxrazy but I've got to get out of the house now lol so just an update Quote Link to comment
seer_tenedos Posted May 26, 2019 Share Posted May 26, 2019 9@BLKMGK I have not had a chance to test yet but I can tell you how I initially a2 my tests that were easy. Don't do a multi node swarm. Just turn the unraid into a single node swarm of itself. That means you don't need to worry about networking between boxes etc. When I did that then created multi containers with just a basic test webpage on them on a docker network I could I could not talk between them of I docker exec into them or from my host to them for memory. Sadly I just have my main unraid server and no spare but I may still try the kernel on it in a day or 2 when I get some time. Also just for reference I want swarm for Openfaas. 1 Quote Link to comment
CHBMB Posted May 26, 2019 Share Posted May 26, 2019 Try this one..... root@matrix:/mnt/disk1/dvb# check-config.sh .config-swarm info: reading kernel config from .config-swarm ... Generally Necessary: - cgroup hierarchy: properly mounted [/sys/fs/cgroup] - CONFIG_NAMESPACES: enabled - CONFIG_NET_NS: enabled - CONFIG_PID_NS: enabled - CONFIG_IPC_NS: enabled - CONFIG_UTS_NS: enabled - CONFIG_CGROUPS: enabled - CONFIG_CGROUP_CPUACCT: enabled - CONFIG_CGROUP_DEVICE: enabled - CONFIG_CGROUP_FREEZER: enabled - CONFIG_CGROUP_SCHED: enabled - CONFIG_CPUSETS: enabled - CONFIG_MEMCG: enabled - CONFIG_KEYS: enabled - CONFIG_VETH: enabled (as module) - CONFIG_BRIDGE: enabled - CONFIG_BRIDGE_NETFILTER: enabled - CONFIG_NF_NAT_IPV4: enabled (as module) - CONFIG_IP_NF_FILTER: enabled (as module) - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module) - CONFIG_IP_NF_NAT: enabled (as module) - CONFIG_NF_NAT: enabled (as module) - CONFIG_NF_NAT_NEEDED: enabled - CONFIG_POSIX_MQUEUE: enabled Optional Features: - CONFIG_USER_NS: enabled - CONFIG_SECCOMP: enabled - CONFIG_CGROUP_PIDS: enabled - CONFIG_MEMCG_SWAP: enabled - CONFIG_MEMCG_SWAP_ENABLED: enabled (cgroup swap accounting is currently enabled) - CONFIG_LEGACY_VSYSCALL_EMULATE: enabled - CONFIG_BLK_CGROUP: enabled - CONFIG_BLK_DEV_THROTTLING: enabled - CONFIG_IOSCHED_CFQ: enabled - CONFIG_CFQ_GROUP_IOSCHED: enabled - CONFIG_CGROUP_PERF: enabled - CONFIG_CGROUP_HUGETLB: missing - CONFIG_NET_CLS_CGROUP: enabled (as module) - CONFIG_CGROUP_NET_PRIO: enabled - CONFIG_CFS_BANDWIDTH: enabled - CONFIG_FAIR_GROUP_SCHED: enabled - CONFIG_RT_GROUP_SCHED: enabled - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module) - CONFIG_IP_VS: enabled (as module) - CONFIG_IP_VS_NFCT: enabled - CONFIG_IP_VS_PROTO_TCP: enabled - CONFIG_IP_VS_PROTO_UDP: enabled - CONFIG_IP_VS_RR: enabled (as module) - CONFIG_EXT4_FS: enabled (as module) - CONFIG_EXT4_FS_POSIX_ACL: enabled - CONFIG_EXT4_FS_SECURITY: missing enable these ext4 configs if you are using ext3 or ext4 as backing filesystem - Network Drivers: - "overlay": - CONFIG_VXLAN: enabled (as module) Optional (for encrypted networks): - CONFIG_CRYPTO: enabled - CONFIG_CRYPTO_AEAD: enabled - CONFIG_CRYPTO_GCM: enabled (as module) - CONFIG_CRYPTO_SEQIV: enabled - CONFIG_CRYPTO_GHASH: enabled (as module) - CONFIG_XFRM: enabled - CONFIG_XFRM_USER: enabled - CONFIG_XFRM_ALGO: enabled - CONFIG_INET_ESP: enabled - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled - "ipvlan": - CONFIG_IPVLAN: enabled (as module) - "macvlan": - CONFIG_MACVLAN: enabled (as module) - CONFIG_DUMMY: enabled (as module) - "ftp,tftp client in container": - CONFIG_NF_NAT_FTP: enabled (as module) - CONFIG_NF_CONNTRACK_FTP: enabled (as module) - CONFIG_NF_NAT_TFTP: enabled (as module) - CONFIG_NF_CONNTRACK_TFTP: enabled - Storage Drivers: - "aufs": - CONFIG_AUFS_FS: missing - "btrfs": - CONFIG_BTRFS_FS: enabled - CONFIG_BTRFS_FS_POSIX_ACL: enabled - "devicemapper": - CONFIG_BLK_DEV_DM: enabled (as module) - CONFIG_DM_THIN_PROVISIONING: enabled (as module) - "overlay": - CONFIG_OVERLAY_FS: enabled - "zfs": - /dev/zfs: missing - zfs command: missing - zpool command: missing Limits: - /proc/sys/kernel/keys/root_maxkeys: 1000000 Unraid-v6.7.1rc1-swarm-v2.zipswarm-v2.patch 1 Quote Link to comment
BLKMGK Posted May 27, 2019 Author Share Posted May 27, 2019 Another big thanks! Loaded it up and it's running stable as an unRAID server. I'm still seeing Swarm issues with our project though and at this point I'm starting to suspect something else could be amiss so it would be helpful if someone else could test as well. Same error about a route to host not being found.😣 I have spun up another standard Linux host as a VM and am waiting for my more knowledgeable friend to lend a hand adding it to our swarm and testing in case I screwed something up previously. Docker DNS seems pretty weird so diddling with it is confusing for me trying to troubleshoot it. The suggestion of a single host swarm seems like a good one and maybe we can try that on this test system. Unfortunately I'm about to take a trip away from home for a few weeks. I'll have a laptop and VPN access at least and my partner in crime will be coming along too so hopefully there will be some downtime to better troubleshoot this together. I'll update as I figure things out, I suspect my server will be getting a 12core Ryzen soon so I'd love to be able to utilize it fully BTW it's pretty weird seeing the container appear and disappear as the swarm comes up and down let me tell you! unRAID currently doesn't have an XML file for it so it's being loaded at the CLI. I'll figure out how to more normally load it in the future once the silly thing works. A big THANK YOU to @CHBMB! Quote Link to comment
CHBMB Posted May 27, 2019 Share Posted May 27, 2019 2 hours ago, BLKMGK said: Another big thanks! Loaded it up and it's running stable as an unRAID server. I'm still seeing Swarm issues with our project though and at this point I'm starting to suspect something else could be amiss so it would be helpful if someone else could test as well. Same error about a route to host not being found.😣 I have spun up another standard Linux host as a VM and am waiting for my more knowledgeable friend to lend a hand adding it to our swarm and testing in case I screwed something up previously. Docker DNS seems pretty weird so diddling with it is confusing for me trying to troubleshoot it. The suggestion of a single host swarm seems like a good one and maybe we can try that on this test system. Unfortunately I'm about to take a trip away from home for a few weeks. I'll have a laptop and VPN access at least and my partner in crime will be coming along too so hopefully there will be some downtime to better troubleshoot this together. I'll update as I figure things out, I suspect my server will be getting a 12core Ryzen soon so I'd love to be able to utilize it fully BTW it's pretty weird seeing the container appear and disappear as the swarm comes up and down let me tell you! unRAID currently doesn't have an XML file for it so it's being loaded at the CLI. I'll figure out how to more normally load it in the future once the silly thing works. A big THANK YOU to @CHBMB! The only thing missing from that last build was filesystem stuff that isn't relevant to Unraid and CGROUP_HUGETLB Which I tried to activate, but couldn't..... Disclaimer: I have no real idea what HUGETLB is and only a rudimentary understanding of C Groups. Might take another look tonight. Quote Link to comment
seer_tenedos Posted May 29, 2019 Share Posted May 29, 2019 @CHBMB thanks for all the help on this and sorry to take so long to get a chance to test it. Firstly i will point out how i am testing it incase other interested people want to try or can spot my mistake. 1. create a single node docker swarm (docker swarm init) 2. follow connectivity tests from https://gist.github.com/alexellis/8e15f2ea1af7281268ec7274686985ba Sadly the latest patch above did not quite work for me. The network create and service create looks like they work $ docker network create --driver=overlay --attachable=true testnet $ docker service create --network=testnet --name web --publish 80 --replicas=5 nginx:latest but when i used curl below to test it could not connect docker run --name alpine --net=testnet -ti alpine:latest sh / # apk add --no-cache curl fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz (1/5) Installing ca-certificates (20190108-r0) (2/5) Installing nghttp2-libs (1.35.1-r0) (3/5) Installing libssh2 (1.8.2-r0) (4/5) Installing libcurl (7.64.0-r1) (5/5) Installing curl (7.64.0-r1) Executing busybox-1.29.3-r10.trigger Executing ca-certificates-20190108-r0.trigger OK: 7 MiB in 19 packages / # curl web curl: (7) Failed to connect to web port 80: Host is unreachable / # ping web PING web (10.0.0.2): 56 data bytes either something is still missing or i have set something up wrong. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.