Ryonez
Members
-
Joined
-
Last visited
-
Mmm, and the job files say the smart move is being used, even though that setting is off. I've uninstalled the plugin for now, and the mover works without screwing my ram over. My suggestions for plugin development with my experiences so far would be: Make sure the settings are being read and used. I'm not confidant they are currently. Offer storing the files in tmp on alternative storage, such as the cache or array. While ram is a great place for active files, it's able to starve out other processes and trigger the OOM. I'll add some screenshots from the last run (which I had to cancel), but as is it's not safe to run the plugin with the OOM risk.
-
Were you responding to another message other than my own? Mine was referring to files created that were filling up the ram.
-
It should be affecting your ram, unless you have tmp pointed to your cache.
-
I'm finding even with "Log Mover Tuning plugin actions" set to no, that it's logging to `/tmp/ca.mover.tuning`. The problem file is `Filtered_files_`*, which grows and grows, using up a large amount of ram, eventually triggering OOM.
-
When working with docker, I've been having issues for the past month, though not without finding what exactly the trigger is. The first time this happened was when I update all containers, around the 1st of May for me. Forced an unclean reboot after oom-killer was triggered, and the containers loaded up all right after the OS came back up, albeit with high CPU usage, though I can't remember if the memory usage spiked at the time. The second was a few days later, May the 4th for me. Again triggered during an update, forced another unclean reboot after oom-killer was triggered, this time the containers did not come back up, docker chocked the system up both on CPU and memory. Symptoms: Changes to the docker image seem to illicit high CPU usage from docker. Is also causes high spikes in memory usage. The spike in memory usage can memory to get completely used, triggering the oom-killer. The more containers in use, the more likely it seems to cause this every time. The amount required to trigger oom-killer is unknown currently. Things tried to resolve this: Checking the docker image filesystem (Result: No problems). Deleting and rebuilding the docker image, several times. Did a cache filesystem check (BTRFS, Result: Errors found.) Replaced the cache drives completely. Re-flashing the the boot USB and only copying configs back over. Starting shifting almost all my docker container configs to portainer, as docker template duplicate fields and restores unwanted fields from template updates. "Currently" (I'm having to rebuild the docker image again atm), only 3-4 containers are unRaid managed. Currently it seems if I only load containers up in small batches, I can get every container up. But only from a clean docker image, or when editing a small number of containers at a time (this I'm feeling iffy on). For the first diagnostic log attached (atlantis-diagnostics-20230507-1710), this seems to have logs from the 4th of May mentioned above. It'll have the logs from when docker was no longer able to be started. I ended up leaving docker off when it triggered the oom-killer. the log on the 7th shows that zombie process from the docker containers where preventing unmounting the cache. and unclean reboot was done. For the second diagnostic log attached (atlantis-diagnostics-20230521-1029), I was working on getting a small container stack of 3 images going. But I was configuring them, meaning they were reloaded several times in a row, and seemly creating a situation where the CPU and Memory spikes got stacked up enough for docker to trigger the oom-killer. Marking this as urgent as docker killing itself via oom-killer is leaving a big risk of data corruption, along with leaving docker pretty much useless currently. atlantis-diagnostics-20230507-1710.zip atlantis-diagnostics-20230521-1029.zip
-
I've been experiencing what appears to be this issue. I've been shifting almost all of my docker management over to Portainer. My best guess so far is that the templates added fields back into templates that started screwing with the docker image and broke things somehow. I've been trying pretty much everything, deleting the docker img, even replaced the cache drives, reformatting the os drive. The only things kept that entire time was the config files. There's still some dockers to shift over, but most are on Portainer, and things seem to be steady. This issue first appeared when updating some images. Otherwise, I'd been running what I had for over a year.
-
Hi there. I had to reboot my server recently, and I received a `Possible Hack Attempt` notification from a plugin called `Fix Common Problems'. Looking into this, I think I've found a bug/security issue. The server's SSH port should be configured to port `1122`. However while the system is booting up, SSH is started before the set configuration is read, leading to port 22 being exposed. Now, I have port 22 forwarded to the server for a service running `Forgejo`, a code repository service. The reason port 22 is being used is to lessen client configuration, and to make it simpler for users. The bug here is unRaid is loading SSH with default values for a time, and later restarting with the correct config. This creates a window for attackers to attempt to access the system through something that isn't meant to be exposed. Steps to reproduce: 1. Set a different port for SSH through unjRaid's web UI. 2. Reboot the server. 3. There is now a windows of opportunity to attack the server via the default SSH port as it starts. Plugins extend this window. Some of the Logs: SSH being started with port 22: SSH login attempts being made during this window of opportunity: There's more than just those. SSH being restarted with the correct configuration: I'll also make a quick note it looks like smbd is having the same treatment, it's loaded with defaults, then restarted with the proper configs later. So it appears there might be several system services being started like this when they shouldn't. Will make this as urgent, though I'll admit I'm not 100% if SSH will accept the correct password. This server runs a chat service, so uptime is import for me, and testing would interrupt that. Please adjust the urgency as needed if it's not as bad as I think it might be. atlantis-diagnostics-20230227-0254.zip
-
Been working with a VaultHunters3 server using your image, and I've noticed the the minecraft servers aren't being cleanly shutdown when the container is shutdown. Is there a way to fix this? Clean shutdowns are very much preferred.
-
Second week in a row: Apr 25 02:10:37 Atlantis CA Backup/Restore: docker stop -t 60 atlantis-yoko Apr 25 02:10:37 Atlantis CA Backup/Restore: Backing up USB Flash drive config folder to Apr 25 02:10:37 Atlantis CA Backup/Restore: Using command: /usr/bin/rsync -avXHq --delete --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" /boot/ "/mnt/user/Backups/machine_specific_backups/atlantis_flash/" > /dev/null 2>&1 Apr 25 02:10:42 Atlantis CA Backup/Restore: Changing permissions on backup Apr 25 02:10:42 Atlantis CA Backup/Restore: Backing up libvirt.img to /mnt/user/Backups/machine_specific_backups/atlantis_libvirt/ Apr 25 02:10:42 Atlantis CA Backup/Restore: Using Command: /usr/bin/rsync -avXHq --delete --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" "/mnt/user/system/libvirt/libvirt.img" "/mnt/user/Backups/machine_specific_backups/atlantis_libvirt/" > /dev/null 2>&1 Apr 25 02:10:42 Atlantis CA Backup/Restore: Changing permissions on backup Apr 25 02:10:42 Atlantis CA Backup/Restore: Backing Up appData from /mnt/user/appdata/ to /mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected] Apr 25 02:10:42 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar -cvaf '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar' --exclude "alteria-plex/Library/Application Support/Plex Media Server/Metadata" --exclude "0-pihole" --exclude "alteria-postgres" --exclude "alteria-hydrus-server" * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress Apr 25 02:10:42 Atlantis CA Backup/Restore: Backup Complete Apr 25 02:10:42 Atlantis CA Backup/Restore: Verifying backup Apr 25 02:10:42 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar --diff -C '/mnt/user/appdata/' -af '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress Apr 25 02:10:42 Atlantis kernel: br-55ef392db062: port 1(veth391173c) entered blocking state Apr 25 02:11:43 Atlantis kernel: br-55ef392db062: port 70(vetha9a6916) entered forwarding state Apr 25 02:11:44 Atlantis CA Backup/Restore: ####################### Apr 25 02:11:44 Atlantis CA Backup/Restore: appData Backup complete Apr 25 02:11:44 Atlantis CA Backup/Restore: ####################### Apr 25 02:11:44 Atlantis CA Backup/Restore: Deleting /mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected] Apr 25 02:11:45 Atlantis CA Backup/Restore: Backup / Restore Completed That's two weeks of missed backups, this is a big issue.
-
Just had the plugin fail to backup. Looking at the logs, everything worked fine, verification was good, but the backup file just does not exist. Apr 18 02:10:41 Atlantis CA Backup/Restore: docker stop -t 60 atlantis-yoko Apr 18 02:10:41 Atlantis CA Backup/Restore: Backing up USB Flash drive config folder to Apr 18 02:10:41 Atlantis CA Backup/Restore: Using command: /usr/bin/rsync -avXHq --delete --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" /boot/ "/mnt/user/Backups/machine_specific_backups/atlantis_flash/" > /dev/null 2>&1 Apr 18 02:10:44 Atlantis CA Backup/Restore: Changing permissions on backup Apr 18 02:10:44 Atlantis CA Backup/Restore: Backing up libvirt.img to /mnt/user/Backups/machine_specific_backups/atlantis_libvirt/ Apr 18 02:10:44 Atlantis CA Backup/Restore: Using Command: /usr/bin/rsync -avXHq --delete --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" "/mnt/user/system/libvirt/libvirt.img" "/mnt/user/Backups/machine_specific_backups/atlantis_libvirt/" > /dev/null 2>&1 Apr 18 02:10:44 Atlantis CA Backup/Restore: Changing permissions on backup Apr 18 02:10:44 Atlantis CA Backup/Restore: Backing Up appData from /mnt/user/appdata/ to /mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected] Apr 18 02:10:45 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar -cvaf '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar' --exclude "alteria-plex/Library/Application Support/Plex Media Server/Metadata" --exclude "0-pihole" --exclude "alteria-postgres" --exclude "alteria-hydrus-server" * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress Apr 18 02:10:45 Atlantis CA Backup/Restore: Backup Complete Apr 18 02:10:45 Atlantis CA Backup/Restore: Verifying backup Apr 18 02:10:45 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar --diff -C '/mnt/user/appdata/' -af '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress Apr 18 02:10:45 Atlantis kernel: br-55ef392db062: port 1(veth2d7642e) entered blocking state Apr 18 02:11:46 Atlantis kernel: br-55ef392db062: port 70(veth7449b16) entered forwarding state Apr 18 02:11:46 Atlantis CA Backup/Restore: ####################### Apr 18 02:11:46 Atlantis CA Backup/Restore: appData Backup complete Apr 18 02:11:46 Atlantis CA Backup/Restore: ####################### Apr 18 02:11:46 Atlantis CA Backup/Restore: Backup / Restore Completed Looking at the folder the backup should be in, it's empty. Any idea what happened?
-
I'm unable to fill a var via a command via scripts currently. Going back, I can see it stopped working correctly on the 14/12/21 (NZT). The command is this: #!/bin/bash -x DBLIST="$(docker exec -t alteria-postgres psql -U postgres -d postgres -q -t -A -c 'SELECT datname from pg_database')" echo "${DBLIST}" I'm at a loss atm, I can't think what would stop this, and the command can be run manually in the terminal, returning the list of databases correctly. The failure was very wonky as well. It saved all the databases fully, then I got zero bit files, then some had data but not all, then the script started running multiple times before outright just stopping. Overview of the base folder: Last Good Backup: When it started messing up Would anyone have any ideas what on earth happened and how to get this working?
-
To be fair, this container can run without mysql. It's just with the template enforcing yanking the `DB_HOST` field back in it'll keep breaking the container. Honestly that's something that needs to be exposed to users, a way to tell CA to not change/update the template/s. As it stands the container is working fine with sqlite, showing mysql is not needed. While it has been a annoying experience for the user, seeing a requirement for something that's not actually needed would likely turn them away completely. If the `requires` tag is to be used, it should be used only for things that are actually required, otherwise it losses it's purpose.
-
The detection doesn't have anything to do with the template. The template is basically a form someone has stuck together, telling unRaid where to pull the container, what env vars the user is expected to provide, and some other information, like support links. I found out the `DB_HOST` was causing the issue for looking a mysql server as I looked at the files used to make the container image, finding the relevant check here. Here is where you can find information on how templates are made and used: https://wiki.unraid.net/DockerTemplateSchema Also, templates gotten from CA will replace removed template values. `DB_HOST` will return at some point. I have this issue with templates exposing ports that just should not be exposed and it drive me freaking crazy. This is something managed by the CA plugin. The way to "fix" this is to remove the `TemplateURL` value from the template. You won't get updates for the template anymore, but it will stop the template from changing to what can be undesired or even unsafe values for your environment. You'll have to manually edit the template file on the usb (via the unraid console or ssh). Open the unRaid Console Navigate to `/boot/config/plugins/dockerMan/templates-user` Look for the xml file with the name matching the name of the template you're wanting to edit (This is the "Name" field at the top of the update container page for the container). `nano <name>.xml` should work. Ctrl+x and confirm save to close once the change are made. Find the template url field, it'll look like ` <TemplateURL>https://raw.githubusercontent.com/selfhosters/unRAID-CA-templates/master/templates/openldap.xml</TemplateURL>` Chance it to be just ` </TemplateURL>` Save the file. It might help to think of the CA Templates as just shortcuts. They are great for quickly setting things up, but they are bad for forcing values you might not want around on you. And don't worry to much about my time, I have plenty to spare >.<
-
Right, hammered at this a little and got it kinda working. Delete the `config.json` file. It's setup incorrectly anyway (example config is here), but we'll have the container remake it. Remove `DB_HOST` from the template. This is what is triggering the search for a mysql database. Add a new variable to the template. Call it something like "DB_URL". The key: "CMD_DB_URL". The Value: "sqlite:///config/hedgedoc.sqlite". Don't include the quotes. Start the container, navigate to the exposed port. It might look kinda broken, that's because the config isn't fully setup. Google from this point should be able to help you fill out the config as needed, like why the page looks broken. I won't be to much help here, my set-up uses a database and a domain with a reverse proxy. It's also been a while since I've done it. Hope this helps.
-
I should clarify, I am using ssd cache pools. I used hdd incorrectly in this instance. However, I'd still prefer to not have several database systems having a go at them, it feels much better having one system managing the read/writes and queuing.
