KptnKMan

I've been on a mission to get UEFI and rBar working this last few weeks. I did everything above before reading all of this, and confirmed everything in a bare-metal install (Also so I could use NVFlash to backup the old and new BIOS). After all of this, I still get a black screen when booting the VM that works perfectly fine with CSM enabled. Interestingly, I also find that I am unable to boot into "Unraid GUI mode" when booting Unraid UEFI enabled. When I boot the "Unraid GUI mode", everthing appears to boot correctly, and at the last moment when the screen goes black to show the GUI all I see is a flashing terminal cursor in the top left corner of the screen. Normally the GUI would boot and display the graphical login prompt. Has anyone else experienced this or understand why this is? This happens before booting any VM, and VMs boot to a black screen also, when they would normally boot ok. I'm also seeing this in my logs with CSM disabled. I was going to start a new thread about this summarising my experience, but I don't know if maybe I/we should try to work together about it here?

@LexBergif you're interested in doing a lot of reading around 10Gb cards, I'd recommend checking out the thread I started about this. Lots of info in there, details, links and recommendations of what I did and eventually purchased. I'm still happy with my 10Gb setup to this day.

Please, I'm trying to seek some help. I have got this container running, but after setup when I got back to the login page it flashes up for a second then disappears. I've reinstalled multiple times, cleared postgres and redis, reinstalled all dependencies. I can login initially and setup Photonix, then next login the login screen flashes up for a second then disappears again. This happens from then on each time the login is accessed. The logs don't seem to show any errors, shows postgres connected. I've tried accessing from different browsers, InPrivate mode, different client workstations, same login prompt flashes up for a second then is gone. Its quite irritating and I cannot find what is the matter. Can anyone help?

Sorry no, I strongly disagree with the assertion of "going bleeding edge hurt nothing". We all know clearly enough, that updating can break thing just as much as fix things... and often does introduce issues. If you need your unraid for anything more than a test/play box, then it's something we all consider when upgrading especially to dev/RC releases. The prospect of encountering issues on a working server that you need for daily functions, for the sake of "bleeding edge" is (to put it lightly) discouraging. This thread is about temperature monitors in kernel versions of unraid, so that being said, the idea of upgrading to an unraid version to maybe get a feature that you know is not in the kernel version of that release is risking a lot and frankly a bit stupid.

Thanks, I appreciate you clarifying some of the key points here. I'm going to think a bit more about exactly my use case and figure out how this can be better worked into my backup plan process. I may use another tool together with this, to get 2-way syncing for some selective workflows.

Hi, using the settings RCLONE_OPERATION=copy and RCLONE_DIRECTION=both I can see traffic in both directions like so: 1) new local file, copied to remote, yes. renamed file produces duplicate. 2) new remote file, copied to local, yes. renamed file produces duplicate. 3) change local file (edit txt), changes are reflected on remote, yes. 4) change remote file (edit txt), changes are NOT reflected on local, no. Changes are overwritten. This is single changes, single change at a time.

Ok, I understand the risks of what you've described, but I think maybe I'm not communicating this well enough. I think I'm using the word/term "sync" loosely, so maybe I can clarify. My bad. I set the variables to RCLONE_OPERATION=copy and RCLONE_DIRECTION=both, and made 2 changes on the remote cloud side; I created a new file "dont_delete_me_please.txt" and renamed a file "rclone_test2.txt" to "rclone_test_2.txt" (added the 2nd underscore to the filename). With that logic, I expected the operation to delete the local copy of "rclone_test2.txt", then copy down "rclone_test_2.txt" and "dont_delete_me_please.txt". But I had the duplicate behaviour as shown in the screenshot. It duplicated the renamed file, and copied down the new file. But I take it that the RCLONE_OPERATION=copy behaviour will never delete anything, but only copy, which is fair enough (Is that correct?). The original issue however, was that when setting RCLONE_OPERATION=sync, it would only sync in 1 direction. So am I correct in thinking that (Assuming RCLONE_DIRECTION=both) an RCLONE_OPERATION=sync operation will copy everything in 1 direction, then copy everything (including the previous copy) in the other direction (including new duplicates)? Am I missing something?

Ok, so I changed the variable RCLONE_OPERATION=sync to RCLONE_OPERATION=copy and now files are copied, but and changes are not recognised. As a result a changed file is duplicated, where the "old" (un-renamed) file is replaced and the "new" (renamed) file is synced. Any ideas?

Hi, After some messing around I've got docker to connect to my OneDrive, and I'm testing syncing. However, I can only seem to sync files UP, but not DOWN, and if I add files to the folder from the OneDrive side, they are deleted. If I change something on the OneDrive side, the "new" (changed) file gets deleted, and the "old" (unchanged) file is synced back up in its place. Seems like a working 1-way sync, I'm trying to get 2-way. I've resolved all errors in the logs/reports, and the link seems to be working. Anyone know why this is happening? I have the variables set as following: RCLONE_OPERATION=sync RCLONE_DIRECTION=both

Yeah that's fair enough, I didn't know if there was something else behind your comment. I suppose it is fair enough, I often think out loud as well. The script error does seem to have gone after the recreation, so I think there was a script with an "assumption" somewhere that didn't check if the bridge device existed first. it is a bit strange though. I'm not sure if I'll get to the bottom of that one, but hopefully someone can find this and it might help them in the future. I usually use notepad++ from another machine, which allows for whitespace and non-printable character detection, as I've had that annoyance before. Its a good thing to keep vigilant for, and I will do a deeper look to see if something is occurring. For now, it seems to have confirmed my suspicions that the files were created when the network configuration was modified in the GUI, but that caused a bunch of things to go wrong at the same time. It's quite sad that this doesn't want to behave with a relatively simple configuration.

If the temp improvements are confirmed to be in kernel 5.16, in my opinion it doesn't seem like a good idea to upgrade just to see if it will work if 6.10rc2 is confirmed to have kernel 5.14.15. Seems like a good way to possibly create more issues (Like the share configuration being disabled/modified) where there might not be right now. I've been waiting and considering 6.10rc2 for a while, especially with the first inclusion of swTPM for Windows 11, but still I wouldn't advise upgrade without solid reasoning. Right now, I've been watching and reading to see what the reported experiences are. I know I can potentially roll back, but that's not a great plan for me personally. I need my servers, and I'm still struggling with some other unraid issues around 10Gbit networking and bonding. Honestly, I wish that kernel 5.16 would be used if that includes temp monitor updates for X570 motherboards, but that will come later for other reasons I suppose. Also, my X570-PLUS seems to work quite well. Saying that, I'm not keen to change anything, as I've done that before and lost all of my sensors. ¯\_(ツ)_/¯

These are the sensors I see on the X570-PLUS: These are the sensors I see on the X570-PRO:

Hi, I upgrade both my systems last week to BIOS version 4021 and I've been having the same issues. I still have working temps on the X570-PLUS and not on the X570-PRO. Seeing as they are both using the same chipset, I've been trying to copy the temp configuration from Unraid1 over to Unraid2, but I've not managed to make it work yet. I did this by copying the file /boot/config/plugins/dynamix.system.temp.plg and the contents from /boot/config/plugins/dynamix.system.temp/ to the other server. After a reboot, that didn't seem to work. If anyone has any idea how I can copy a working temp config to another server, I would be grateful.

You mean beside the netboot bios? I haven't seen any delays other than that with the cards coming up, they seem to be recognised and active when Unraid boots on both servers. Yep, I'm using the dual-port SFP+ cards, and only using a single SFP+ on each card, just as you say. I'm not sure why more "ghost" ports would be registered, as I can see all 3 (Including the onboard 1Gb) listed when the system comes online. Is a ghost port something that has been observed in other uses of the Mellanox X-3 cards? Is this known behaviour with dual-port cards? The BIOS on these cards is the latest installed from the Mellanox site, installed using the Mellanox Tools for Unraid, exactly as instructed for the tools (As detailed in the GUI). Also, I have this issue on both servers where there is seemingly a skipped interface. I noticed in the network.cfg that initially there is the line for 'BONDNICS[0]'. Originally, this read as 'BONDNICS[0]="eth0 eth1 eth2 eth3"' But now it reads as 'BONDNICS[0]="eth0 eth1 eth3"' on Unraid1, and 'BONDNICS[0]="eth0 eth2 eth3"' on Unraid2. Unraid1 is where I'm having the issue where it seems to not be able to decide which interface id is being used, and lists 4 in the network-rules.cfg file. Unraid2 has not shown this particular issue. Also, this bootup error seems to have disappeared after I jumped through hoops to remove-then-re-add the br0 interface, that I mentioned above. I suspected that was related to the bridge getting messed up somehow, and that seems to have reset it. I understand that the bridge itself is also listed as a device, as basically everything is in linux. So with it being broken the startup scripts were probably either looking for it or assuming it was there? to setup the bridge and/or bond? Interestingly, this s***show began after I simply set the Interface Description. Changed nothing else. Before that I had wiped the network.cfg and network-rules.cfg files, all settings were running default, and I noted that the network.cfg file was not recreated until I set the Interface Description. As soon as I did that, everything seemed to get weird. Happened on both Unraid systems. ¯\_(ツ)_/¯

So I got the br0 network interface back to a "working" state by essentially: - in settings, disabling Docker - in settings, disabling VMManager - network settings, disable bonding and bridging - reboot - fix "interface rules" duplicates manually again - confirm that everything is in correct order - reboot - fix "interface rules" again, and confirm - network settings, enable bonding and bridging - reboot - confirm settings stuck, fix "interface rules" again - confirm br0 is present and working - enable docker - enable VMManager - verify VMs work again Really not sure why that got broken. Also on Unraid1, the physical interfaces now seem to show up as eth0, eth1 and eth3. I still need to manually modify the network-rules.cfg file after each bootup, otherwise 4 interfaces show up in the "interface rules" GUI. I just encountered the same br0 missing issue on Unraid2, and resolved it following the same process above, but the interfaces are labelled the same afterward. Would really appreciate if anyone has any idea what might be happening?

I started a new thread to investigate some issues I encountered today regarding more networking trouble. I can troubleshoot networking issues well enough, but Unraid configuration I'm not an expert at. Something strange is going on, and I'm really not sure why. I'd really just like to get the bond networking working stable and well so I can not have to mess with it.

Hi all, I've been getting very strange behaviour on both my Unraid systems since upgrading my NICs to 10Gb Mellanox ConnectX-3 cards. I've chronicled my journey to using 10Gb in a separate thread over here. Started this thread to try and figure out if there is a way to resolve the issues I'm encountering. My intention is to have the bonded network (active backup mode 1) working with the first port of the Mellanox 10Gb card as primary, and the onboard 1Gb as secondary failover. My understanding of how this should work is that the "primary" interface should be prioritised and fail-over to the secondary, then further interfaces (if configured) if primary interface is interrupted. I understand the bond should also fail-over back to the primary interface, should it reconnect. That doesn't seem to be happening, or I'm misunderstood. So far I've not been able to make that work as expected, and I'm struggling to determine why. When either system boots up, the onboard 1Gb is set as primary bonding interface until I manually unplug the 1Gb and it fails-over to the 10Gb. It does not seem to fail-over back to the 1Gb, unless I unplug the 10Gb. I'm quite sure its not the cards themselves that are the issue, as they seem to operate within expectations, but some software configuration or settings in Unraid. At this point I'm running everything pretty standard, and still experiencing issues. Is there some way to manually set the bonding I'm also seeing some other strange behaviours that I can't determine the reason for, and I continue to troubleshoot: On both servers, I have (and are using) the 10Gb Mellanox card, and the 1Gb onboard for the respective motherboard. On both servers, I have tried wiping the network.cfg and network-rules.cfg but the same issues return. On both servers, bonding (active-backup mode 1) and bridging is enabled by default. If anyone has any advice or can help troubleshoot I'd be thankful. A few issues are present, being: - network interface assignments changed. On Unraid1 I have eth0, eth1, eth2 and also eth3(?) yet I only should have 3 physical interfaces. On Unraid2 I'm getting eth0, eth2, and eth3. I'm not sure where eth1 has been skipped. - duplicate entries in the network-rules.cfg file. On Unraid1, I'm seeing 4 entries in config for the rules (eth0 to eth3) but still show 3 physical interfaces. This causes an apparently known issue where I am unable to edit the "interface rules" in the network setting GUI. I have to go back to the network-rules.cfg file, delete one of the entries and back to the GUI. This reoccurs after each reboot. On Unraid2, this occured once but has not seemed to be an issue since, the system has apparently settled on eth0, eth2 and eth3. - br0 has disappeared. On Unraid1, as of today, the br0 interface apparently seems to have disappeared. I'm not sure why this is, I haven't modified anything in networking as yet, apart from modifying the "interface rules" order and setting an interface description. I can access the webgui but I now get an error when trying to start any VMs: - boot up log errors (I think) related to interface assignments. On Unraid1, I have started seeing errors in the bootup log stating: 'Not enough information "dev argument is required. Error: Argument "down" is wrong: Device does not exist' There doesn't seem to show an IP on the main bootup screen either, not sure if this is related:

Yeah, this is what I'm wondering about as well. I'm going to have to do some tests and see if there is a timeout I'm missing. I remember seeing in the CRS309 that there were a few timeout settings, so I assumed there would be something similar for unraid. That's something I could do. I imagine the passive side would just be regular links? I guess that would that mean disabling the bond0, then configuring the separate switch interfaces as a bond interface as before? Would that mess with the DHCP or do I need to spoof both nics as the same MAC? 🤔 I need to read the mikrotik wiki to find out their recommendations, or try their forum as you advised. 🤔 I need to go find some answers, I appreciate you taking the time to run through this with me.

Ah gotcha. Thanks, that makes total sense. Absolutely, this seems to match what I read and understand. I'm not sure how to set the priority in unraid apart from the NIC order in the "interface rules". Is there a config I can set where I can assign some kind of weight to the ethX interfaces? I made sure to rearrange the interfaces to prioritise both the 10Gb links (1 being disconnected of course, but ok) and then the 1Gb link. This way I figured it would just go down the totem until it reaches the 1Gb link, and refer back to the 10Gb when it returns. Doesn't seem to follow that behaviour, unless there is a timeout I'm not aware of? I figured that 30 seconds was the norm, even though it failed-over immediately, didn't seem to skip a beat. Hmm, I assumed this was the order in the network-rules.cfg file or the interface order in the "bonding members of bond0". Interestingly, there is no network.cfg file since I deleted it to reset the config. I assumed this is because I have not modified anything, and the active-backup bond0 was configured by default. If I set an interface description or something, I expect it to return. I initially set the "interface rules" in the order I want, but I should look further into configuring linux UDEV rules, if there is something else I can set for weight or timeout. I think there's something I'm missing, but I can't see anything else obvious.

Ok, so I pulled the 1Gb cable on unraid2 (the unraid end) and it also failed over to the 10Gb. I can also see it on the CRS309: iperf3 confirms the (inconsistent as before with MTU@1500) 10Gb link: root@blaster:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 32828 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 822 MBytes 6.89 Gbits/sec 0 291 KBytes [ 5] 1.00-2.00 sec 894 MBytes 7.50 Gbits/sec 0 297 KBytes [ 5] 2.00-3.00 sec 876 MBytes 7.35 Gbits/sec 0 300 KBytes [ 5] 3.00-4.00 sec 930 MBytes 7.80 Gbits/sec 0 294 KBytes [ 5] 4.00-5.00 sec 721 MBytes 6.05 Gbits/sec 0 300 KBytes [ 5] 5.00-6.00 sec 964 MBytes 8.08 Gbits/sec 0 288 KBytes [ 5] 6.00-7.00 sec 876 MBytes 7.35 Gbits/sec 0 303 KBytes [ 5] 7.00-8.00 sec 916 MBytes 7.69 Gbits/sec 0 291 KBytes [ 5] 8.00-9.00 sec 775 MBytes 6.50 Gbits/sec 0 280 KBytes [ 5] 9.00-10.00 sec 880 MBytes 7.38 Gbits/sec 0 283 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.45 GBytes 7.26 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.45 GBytes 7.26 Gbits/sec receiver iperf Done. Honestly, I'm less concerned about the speed and MTU settings at this point, it seems to work so that's good enough. I'd rather like to figure out how to set a priority/preference on the eth0 10Gb link.

Pulled the 1Gb cable (the unraid end) on unraid1, and looks like it failed over to the 10Gb. After plugging it back in, it stayed on the 10Gb as primary: I need to figure out how to configure unraid to prioritise the 10Gb always, unless its disconnected. Any idea on that?

iperf3 confirms the 1Gb is primary: root@blaster:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 52652 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 106 MBytes 889 Mbits/sec 0 235 KBytes [ 5] 1.00-2.00 sec 105 MBytes 883 Mbits/sec 0 235 KBytes [ 5] 2.00-3.00 sec 105 MBytes 880 Mbits/sec 0 235 KBytes [ 5] 3.00-4.00 sec 107 MBytes 898 Mbits/sec 0 238 KBytes [ 5] 4.00-5.00 sec 106 MBytes 893 Mbits/sec 0 238 KBytes [ 5] 5.00-6.00 sec 105 MBytes 877 Mbits/sec 0 235 KBytes [ 5] 6.00-7.00 sec 107 MBytes 894 Mbits/sec 0 235 KBytes [ 5] 7.00-8.00 sec 105 MBytes 882 Mbits/sec 0 235 KBytes [ 5] 8.00-9.00 sec 106 MBytes 891 Mbits/sec 0 235 KBytes [ 5] 9.00-10.00 sec 106 MBytes 891 Mbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.03 GBytes 888 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.03 GBytes 886 Mbits/sec receiver iperf Done.

That makes sense, and I should probably do some more reading on the subject. I will say that the active-backup bond worked just fine without the CRS309 being setup with a special bonding config... So maybe that could be it. I've just nuked the CRS309, and both unraids, all came up without issues first time, with the active-backup bond setup by default (Only with the onboard primary). I've just switched them around and rebooted. Yeah, no worries. I appreciate all you've done to help me out. I'm not really sure what to make of this, but I was also on the long-term until yesterday. Well, I'm going ahead right now with some testing. Everything seems to be behaving, reboots come up without fuss. Also took this opportunity to move the unraid SFP+ DACs along, to give that S+RJ10 a little space. In the new config, it looks like the 1Gb is working as primary, so I will pull some cables later and see if it fails over to the 10Gb. From my observations, the standby always has no Rx transmissions, and the Tx is only ever a few kbps. The CRS309 reset config (only set interface names): unraid with default config (only reordered the nics):

Yeah, I did some reading and came to the same conclusion before I posted that the 10Gb ones NEED to be separated, but not so much for the 1Gb ones. I'll keep an eye on them regardless, but the 1Gb transceivers are barely warm to the touch. Well, I'm quite a daring guy so that sounds harmless. When I have time I'll do some research and see if anyone has done a teardown of the CRS309, and if not then I'll take a look. Its not reached 90c but its been in the mid 80s at load, so I'll investigate the cooling possibilities just the same. Yeah super weird. Pulled it at the CRS309 side, should that make a difference? This whole goose chase has kinda confused me, as it doesn't seem to have a good reason for it. I might just wipe the network config by deleting the network.cfg and network-rules.cfg files and start again, not sure what else I can do on the unraid side to be honest. Yeah sorry this is my bad, I miscommunicated here. I didn't add an extra bridge, I know that would be a bad idea. There's a default configured bridge already present, and that's what I used. I meant to say: "I started by removing the interfaces from the default bridge (As that's what the error I experienced earlier about this referred to), and configuring a new bonding interface (aka bonding-unraid2), as you advised in your earlier post. Then I added that bonding-unraid2 interface to the bridge." Reading your instructions, that's exactly what I did so that's good to know at least. From everything I've read and videos I've seen, setting up an active-backup bond should be simple and easy. Story of my life. 🙄 In the unraid interface it even states: "Mode 1 (active-backup) is the recommended setting. Other modes allow you to set up a specific environment, but may require proper switch support. Choosing a unsupported mode can result in a disrupted communication." I can set the link L2-MTU and L3-MTU but on the bond interface, I'm not sure why, I cant set the L3-MTU (Labelled 'MTU'). You can see in the 2nd screenshot I posted earlier, the L2-MTU is greyed out, and inaccessible: Yeah I thought that the active-backup bond would be simple enough, especially as its "default" bond in unraid, and would be nice to fail back to the onboard controller... But its not proven so simple. Maybe I should have just gotten a 2nd DAC for each server and tried bond across the dual 10Gb instead. Then I could have just used more simple 802.3ad link-aggregation, balance-rr or another method that requires same-speed links, like I did before when I had 1Gb cards. I guess you live and learn, still looking to nuke the network config on both systems and start again, maybe should do on the CRS309 as well, in case I (more than likely) messed something up. Yeah, thanks I appreciate that. Honestly I'm hesitant to start from the beginning seeking help in yet another forum just yet, so I'm gonna go with the scorched earth approach first, just nuke all the configs, start again and see if that might fix things. I think I might just go step by step and see where it all falls apart.

Ah, yeah, a couple things there. I bought 2x S+RJ10 10Gb tranceivers and 2x S-RJ01 1Gb (I mistakenly stated them as S-RJ10), which are rated for much lower temperature. Currently they are next to each other, as in the diagram, but as of now they are very cool and not even nearly as lava hot as the single S+RJ10 hooked up to the media room CSS610. I'm planning to relocate the S-RJ01s soon, just haven't yet as they seem good right now. Yeah, I combed over the specs of the S+RJ10 and S-RJ01 to make sure I understood their tolerances as well. That 10Gb transceiver is very hot, burns to the touch. I might have to put a small fan near it or something. Not yet, I temporarily relocated cards I don't need for a while out and they are both running in the 2nd full PCIEx16 slot of each system. I'd like to get a good idea of full performance before I eventually move them, if at all. At some point I will have to, I think. Proper VLANs are something I have my network pre-segmented for already, so that I can do that. My IOT stuff all uses separate WLANs from my domestic household stuff, and the Home Automation cabinet is separated for this exact reason as well. I played with VLANs at home a while ago, but now I have all widespread VLAN-capable hardware so the time is getting close... very close. I try to be take things "one at a time" so I know things are stable before moving onto the next radical change. I guess thats the Ops Engineer in me. Haha, I should probably be spending less money not more but this 10Gbit setup has been a pipe dream of mine for more than 8 years. I'm finally doing it, and very happy its working as well as it has. Saying that... something weird is happening... Yeah, the bonds were working well... or so I thought... but I've highlighted something very frustratingly confusing by trying to properly configure the active-backup bond for both systems. First of all, I did some testing by unplugging/disabling the 10Gb link for unraid2, which seemed to fail the link over as expected, but then it refused to fail back. It refused like at all, causing the server to become unavailable, until I eventually rebooted the system and it reverted to the 10Gb as primary (Without touching/changing anything else). This doesn't seem correct, as I shouldn't have to reboot the server to fix a failover. I could see the link was up, and registering on the CRS309 but I could only see a couple Kb of Tx and nothing Rx. So I decided that I should properly configure the active-backup on the CRS309, especially as I can now do it on a single box. I did some reading, and watched some YouTube vids about it, then dived in. Simple, I thought... I started by removing the interfaces from the default bridge (As that's what the error I experienced earlier about this referred to), and configuring a new bridge, as you advised in your earlier post. Then I added that "bonding-unraid2" interface to the bridge. Now there's a few things I have run into that are frustrating me, I've been round and round it for hours now and I'm not sure what to do. I've setup both unraid systems identically the same now and have had varying results. 1) If I so much as touch the bonding configuration of either bond on the CRS309, they stop working, and the entire server goes dark. I can't reach it, I have to gracefully shut it down or reboot it at the box and upon reboot, it comes up available again. Even so much as looking at the interface and clicking Ok or Apply (Without changing anything) causes it to freak out and give up. 2) I cannot set the MTU of the bonding-unraidX interface, as much as I've tried. It refuses, and throws an error that the "MTU could be set", then the server goes dark again until reboot. So at this point I've set everything back to 1500 on both unraid systems, just to try and have consistency to test and get working. 3) My second system unraid2, like refuses to play ball 90% of the time, and boots up with a link-local address of 169.254.129.35, indicating that it cannot find the network. I double/triple/quadruple checked the unraid network config and rebooted more than a dozen times to figure out what's going on. I found that a cold boot works more often and picks up an Ip of 192.168.178.12, but a reboot usually stalls for a bit right before login, and then uses 169.254.129.35. I use DHCP on my network, from my WAN router, never had any DHCP issues. I think something on the CRS309 is very wrong, and I'm missing something. I've tried using "arp" link monitoring with an IP, as well as Mii, both seem to have the same behaviour. At this point I've shut down unraid2, as the alternative would be to delete the bonding interface and go back to what was "working". 4) My primary system unraid1, doesn't seem to be too bothered about any of this, and seems to be working with the bonding-unraid1 configuration on the CRS309. I haven't tested the bond failover by pulling the cable yet, as I've been busy trying to figure out why unraid2 is not picking up an IP with an identical config. 5) I updated the CRS309 to latest RouterOS stable 6.49, and nothing seems to have helped. Same issues. I saw some mentions of link stability in the changelogs and thought it might be worth a shot. At this point I'm not sure what to do. Some config screenshots (both systems networking configured identically) if that helps: