BurntOC

I've tried downgrading the MariaDB release to several of the prior versions. It shows successful on install, but they all fail with this same error.

So 6.12.x was giving me fits. I downgraded back to 6.11.5, but I also had to use mover and dynamix file manager to move my files off cache and reformat back to xfs, then put the files back on (since 6.11 didn't natively support zfs). My other containers are starting fine, but I have 3 Mariadb dockers that are throwing errors like this: Caught SIGTERM signal! UMASK corrected from 022 to 0640 ... 230628 16:41:31 mysqld_safe Logging to '/config/databases/c6cc2824f233.err'. 230628 16:41:31 mysqld_safe Starting mariadbd daemon with databases from /config/databases I tried changing the UMASK to 0640 in the template but it seems to revert. I tried copying the directory from an extracted backup and it fails with a different error. Thought maybe permissions needed to be reset to nobody:users and 744 like the others, but the error remains the same: ─────────────────────────────────────── GID/UID ─────────────────────────────────────── User UID: 99 User GID: 100 ─────────────────────────────────────── Setting Up Initial Databases Couldn't set an owner to '/auth_pam_tool_dir/auth_pam_tool'. It must be root, the PAM authentication plugin doesn't work otherwise.. Cannot change ownership of the '/auth_pam_tool_dir' directory to the 'abc' user. Check that you have the necessary permissions and try again. Installing MariaDB/MySQL system tables in '/config/databases' ... Installation of system tables failed! Examine the logs in /config/databases for more information. I also have one PostgreSQL database that is erroring with this type of message, but it seems it might be continuing to launch properly after that and chmod and chowning as above didn't seem to help here either. 2023-06-27 22:54:22.998 PDT [30] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Permission denied 2023-06-27 22:54:23.008 PDT [30] LOG: using stale statistics instead of current ones because stats collector is not responding 2023-06-27 22:54:23.008 PDT [30] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Permission denied 2023-06-27 22:54:40.470 PDT [243] FATAL: could not open file "global/pg_filenode.map": Permission denied 2023-06-27 22:55:10.539 PDT [1] LOG: received fast shutdown request 2023-06-27 22:55:10.539 PDT [1] LOG: could not open file "postmaster.pid": Permission denied 2023-06-27 22:55:10.540 PDT [1] LOG: aborting any active transactions 2023-06-27 22:55:10.541 PDT [1] LOG: background worker "logical replication launcher" (PID 32) exited with exit code 1 2023-06-27 22:55:10.541 PDT [27] LOG: shutting down 2023-06-27 22:55:10.542 PDT [27] PANIC: could not open file "/var/lib/postgresql/data/global/pg_control": Permission denied 2023-06-27 22:55:10.542 PDT [1] LOG: checkpointer process (PID 27) was terminated by signal 6: Aborted 2023-06-27 22:55:10.542 PDT [1] LOG: terminating any other active server processes 2023-06-27 22:55:10.542 PDT [31] LOG: could not open temporary statistics file "pg_stat/global.tmp": Permission denied Any ideas why just my databases seem to be jacked and how to fix them? UPDATE - yes, the PostgreSQL db seems to be working okay for Immich. This just seems to be a problem with my MariaDB containers.

Thanks for the response. In reading that on the Downgrade instruction page it had confused me. I presumed I misread it as I didn't recognize them as compressed files without an extension. I ended up copying the config files from my backup last week, reformatting cache to xfs, and using previous apps to reinstall all of my containers. Other than the fact that most didn't originally start and showed "not available" despite having installed the Docker Update Patch plugin, they seem to come up after a forced update. I'm going to reboot when done with that, but I'm optimistic that I'm back on 6.11.5 ok. Marking as solved for now. UPDATE - Welp, that's depressing. Seeing all sorts of oddness now. Can't SSH in on some of the interfaces I could before. None of my containers autostarted as configured. They can be manually started, but I can't access any of them. Guess I'll have to start back with the bz file extract and restore again. UPDATE 2 - I ended up wiping my Unraid usb, copying all the files from the backup archive I made several days ago, and then copying the bz* files from the 6.11.5 as @JorgeB suggested and it seems to be working again.

So the 6.12.x firmwares have caused serious instability and performance issues with one of my two servers. I finally decided I'd had enough and this morning I ran mover, reformatted my cache drive from zfs back to xfs, and I attempted to downgrade my firmware to 6.11.5 My understanding was that you download the zip file, extract it, and copy those files and folders over the ones on your usb. I did that, and when I booted I realized I had a default config when I saw the server name "Tower" and it wouldn't accept my credentials. It's been probably a week since I did a server flash drive backup, in a moment of laziness this morning instead of doing that I did at least copy my existing config directory off in case I needed it. Anyway, I copied that config directory over the one on the usb and tried to boot, but now I'm not even seeing a gui. Can someone tell me how I fix this?

I figured out that removing macvlan via modprobe and clearing out the syslog entries for it stopped the bogus message (credit to another post here for that). I'll probably do the diag submission, but honestly I'm tempted to wait a month or two to see if kernel updates fix it as there seems to be a number of these oddities with 11th gen systems right now.

The server hangs finally drove me to abandon macvlan for now and I flipped the switch back to ipvlan again. I've rebooted a couple of times since for various reasons, then in reviewing my FCP scan this morning it is still reporting DOZENS of macvlan call trace warnings - and I'm not running macvlan. Meanwhile my very similar Unraid server with the same model quad nic and pretty similar network setup (on a 10th gen Intel vs the 11th gen on this one) hums along on macvlan very nicely. Frustration peaking.

As I mentioned in my previous post, I have a quad port Intel NIC and each port has it's own /24. Each goes via a managed switch to the firewall where again, each has a specific VLAN assigned (so not in Unraid interface settings). MOST all my containers have an IP assigned, and host access is disabled in Docker settings. NO containers are assigned to br0. While I ran pre 6.11 with no issues, unfortunately I swapped the NIC and drives from an AMD 5900X/X570 build to an 11th gen Intel and mobo combo around the time 6.11 was out and at some point FCP started reporting macvlan call trace warnings and I also started having these server hangs. By my understanding, I've follow all the steps, but I continue to have this problem unless I switch to ipvlan. Frustrating.

@boniel Hate asking dumb questions, but I have to in this case. Does a multi-port NIC count as separate adapters for this solution? I'd assume so, but I've moved all my containers off the br0 interface (on an Intel quad-port NIC port0) and I'm still seeing reports of macvlan call traces. I prefer macvlan to ipvlan because some of my containers require it, and it seems like it would potentially offer better security as all 4 NIC ports are separated by subnet and VLANs into my managed switch and then into similarly separate NIC ports on my firewall. Just trying to figure out how to get rid of these call trace errors and improve stability.

Okay. I checked the filesystem so I'll look at safe mode testing.

I have two unraid servers, both have Intel quad NICs with lan and guest interfaces ("inner", and one also has an iot interface "outer". After originally running one on a 5900X (which ran great after some initial periodic hanging until I moved a few remaining items off br0), both now run on Intel 10/11th gen systems. Anyway, those worked pretty well, also, but after migrating many of the containers from the inner server to the outer one, the outer one has started becoming non-responsive periodically. I finally changed the Docker network type from ipvlan to macvlan yesterday because I hoped it would help with stability, though prior to the migration they had both worked really well on ipvlan and I think that option offers better security - so I'm hoping to revert back at some point. The only thought I had is that I still have a unifi-controller container that is running on a br0 sub interface and maybe the system doesn't like that any more, but it had worked fine that way on the very similar inner server. When it hangs, basically the gui becomes non-responsive first. Usually most/all the containers continue to work, at least for a while, but some recent network weirdness suggests to me that maybe things like swag stop fully passing traffic. Sometimes I can console in, but even trying to intitiate a reboot there doesn't seem to work. I have to do a rapid init shutdown or it never actually reboots. unraid-outer-diagnostics-20230503-0923.zip Anyway, hoping someone can help me figure out the cause. Thanks in advance for any help.

Got it. I thought it was just checking events since the last run. Good to know.

Okay, so I decided to just switch to ipvlan on one of my Unraid servers as a test. Today I wake up to find I am still getting macvlan call trace warnings from FCP. Any idea what's up with that? To be clear, it's not a timing of the message - I ran a check again manually and it is reporting the same thing, but I also checked my Docker settings and yup, ipvlan.

@bonienl I have been using macvlan since like day 1, and while I did have to move some things off br0 a couple of years ago due to crashes. I have br0, br2, and br3 now. I don't have any containers running on br0, just a couple on bridge and the others on br2 and br3, but I am getting the macvlan call traces error from FCP. Is that just something that can be ignored, or do I gain any sort of network benefits from ipvlan if things aren't crashing (e.g. fewer broadcasts or such)?

Looks like a few others have seen this lately as well.

Just stopping in to say I encountered this issue out of the blue today as well. About to reboot to get back in.

Guess you have to add a lot of drive space for virtual memory or make your VM memory a lot larger than 8GB....

I have a 32GB 5900X system with a RTX 3070 passed through and 8GB for the Windows VM. Things were working fine for a long time but I rebuilt my server and VM and now I'm running into this and it is crashing Fortnite and other games shortly after I start playing them. Can anyone help me figure out why this would possibly be the case?

Did you have any luck figuring this out? I have a RTX 3070 and things were working fine for a long time but I rebuilt my server and VM and now I'm running into this. Grrr....

Cutting through a whole bunch of stuff, I ran into BIOS flickering and other issues recently so I switched from Legacy to UEFI on my tried and true Unraid server. Booted fine, but starting the VM now results in a black screen and filling my logs within seconds with device not ready errors. Had to add video=efifb:off to my boot options to fix that......anyway, everything else seemed fine, but my Windows guest can no longer find and pair with bluetooth devices though everything looks okay. I created another Windows VM from scratch - same thing. I've tried going back to legacy and just dealing with the flickering at boot but that didn't fix this strange issue. Can someone help me find the solution to why Unraid sees the bluetooth adapter just fine and Windows shows the wizards and devices okay, but it won't let me find and add a bluetooth device in the VM? Thanks.

I’ve been at this a long time, but in an attempt to cut down this long thread even a bit I’m going to try to get to the what I have and what I want to see if one of you experts can solve this. Here’s what I have: · Unraid server running 6.10.3 · I350-T4V2 for Unraid (eth0,eth1,eth2,eth3) · Realtek 2.5GB NIC – passed through to VM on GUEST VLAN · 3 VLANS: o MGMT (corresponds to 192.168.20.x/24) o GUEST (corresponds to 192.168.60.x/24) o IOT (corresponds to 192.168.70.x/24) · Unifi switches and APs with non-Unifi firewall Here are my goals: · Maximize security (especially keeping container traffic mostly in GUEST and IOT networks and segregated from management traffic whenever possible), and keep VMs in GUEST network · Single VLAN per interface I originally had eth0 with an address in the 192.168.10.x range, with the MGMT VLAN on it. · Eth1 was unused · Eth2 had no interface IP, but was on VLAN GUEST with an IP assigned to the VLAN interface. · Eth3 had no interface IP, but was on VLAN IOT with an IP assigned to the VLAN interface. That seemed to work well until recently, but I swapped some servers around and though it seems to be set this way my Unraid gui and SSH access is seeing all sort of slowness and drops. I spoke about it here, but I got no response: https://forums.unraid.net/topic/126512-recent-constant-disconnects-and-web-gui-extremely-slow-diagnostics-zip-attached/ As of right now I can ping from Unraid console and get DNS resolution, but it fails to get a response and things like Community Apps fail to populate. I’ve tried dozens of options including moving the MGMT VLAN to eth1, but I’m running into issues ranging from DNS resolution to continued slowness. Also, maybe I’m just being an idiot because I feel like my brain is broken, but with one VLAN per interface I’d think I could set the appropriate native VLANs on my Unifi switch ports with nothing else tagged and it would work, but it doesn’t seem to work unless I set those as tagged ports on those VLANs. I’ve read literally dozens of threads and web pages about this and I can’t see anyone who has clearly addressed this. I was hoping @bonienl@Vr2Io or one of the other gurus here could tell me how I SHOULD have this set up so that my switch ports are tagged or untagged as they should be, my interfaces are set properly, and maybe even if I have to change my interface rules to re-prioritize interfaces as the slow GUI and disconnects makes dealing with the whole setup unbearable. THANK YOU for any assistance.

Also adding that if I ssh, for example, into other devices off those same switches, I get zero reconnects, adding additional fuel to my theory this is an Unraid issue of some sort....

Just wanted to add that I'd seen the DNS hostname error in FCP and I'd fixed that but the problem remains.

I've been happily running Unraid on a couple of servers for about 2 years or so. Love it. Recently I "swapped" my two servers (one "private" and the other more "media server". Both have I350-T4 quad NICs with a VLAN on each interface, so in practice, what this basically meant was disabling/deleting the IOT interface on the media server, moving sensitive data and containers from one to the other, adding an IOT interface on the "new" media server, and firing things up. It all seems to work, but after doing all of this I am having lots of GUI page timeouts and slowness requiring refreshes to see updates and changes, SSH terminal windows and SFTP windows time out in seconds, and the entire SSH session seems to reconnect every 40 seconds or so. It's happening on both servers, and it did not happen before the move 10 days ago. I've tried deleting and rebuilding network.cfg, changing VLAN interfaces between auto and static, changing default gateways and metrics, and I'm just stumped. Unfortunately, I did also update my Unifi switch firmware around the same time period so I can't absolutely rule that out, but changing back to the previous version didn't seem to help. And I can say that my online gaming, Netflix streaming, Channels DVR container streams, etc. seem to play without any major issues so if there are hiccups being encountered there they are brief enough the buffers are handling it. Most games are FPS with low ping times showing so all of this leads me to believe the culprit is something with Unraid. I was hoping someone might see something screwed up - maybe it was all along and wasn't biting me before, or it could be new. I appreciate any help. unraid2-diagnostics-20220727-1036.zip

I started down this path but what's odd is that I got swag and unifi-controller working fine this way, but of course I'm having to go to Previous Apps to have it actually reinstall the containers and it's showing the apps from BEFORE the transfer, even though I'd deleted all that out of the appdata share, rebooted the server, etc. If I manually browse there it shows all the correct container appdata folders I want it to have, but it doesn't reflect that in the server....

That was going to be my next question when I realized at least part of the issue is I didn't move or swap the docker.img files first. I did just complete another backup of each with the plugin, and everything is shut down at the moment, including Docker services. Can just use the appdata backup plugin and just have each restore from the other's backups? Do I have to do any other prep, like delete docker.img and delete existing appdata folders so they don't "merge"?