May 13, 20251 yr Hey everyone, todayI had a memory leak I guess, 75% of 64GB RAM used. I got many errors navigating the UI (especially on the docker and apps tabs). After a restart, all my docker containers have lost their data and can't start anymore. Seems like the data in disk1/appdata has been affected, just all configured data is gone, the rest is fine. Vaultwarden works fine, for Plex I had to regenerate all thumbnails, everything else seems to be working fine. Gitlab does not start anymore and it looks like all data has been lost Any tips? Edited May 13, 20251 yr by Maertuerer
May 13, 20251 yr Author 8 minutes ago, MAM59 said: diagnostics? (hopefully syslog "save to usb" was turned on before) Don't think I enabled that, is it under the syslog server settings? *Local Syslog Server is disabled "Mirror to flash" was turned off. What now? Edited May 13, 20251 yr by Maertuerer
May 13, 20251 yr turn it on and wait for the next crash to happen. Without it, all messages are gone after the reboot you did.
May 13, 20251 yr Author 3 minutes ago, JorgeB said: Please post the diagnostics. nas-diagnostics-20250513-1633.zip
May 13, 20251 yr Author 11 minutes ago, MAM59 said: turn it on and wait for the next crash to happen. Without it, all messages are gone after the reboot you did. The "crash" happened after multiple years of no problems...
May 13, 20251 yr Filesystem Size Used Avail Use% Mounted on /dev/md1p1 1.9T 1.2T 717G 63% /mnt/disk1 There's only one disk, and it's mounting and has data, so it's not a filesystem issue, if there is data missing, it may have been deleted/moved by someone/something.
May 13, 20251 yr Author Just now, JorgeB said: Filesystem Size Used Avail Use% Mounted on /dev/md1p1 1.9T 1.2T 717G 63% /mnt/disk1 There's only one disk, and it's mounting and has data, so it's not a filesystem issue, if there is data missing, it may have been deleted/moved by someone/something. Like I said, it looks like all changes in all of my containers (some running over 2 years) I did have been reverted after the high RAM problem and a subsequent restart.
May 13, 20251 yr Why do you have 200G docker.img? Have you had problems filling it? That is many times larger than should be necessary if things are configured correctly.
May 13, 20251 yr Author 3 minutes ago, trurl said: Why do you have 200G docker.img? Have you had problems filling it? That is many times larger than should be necessary if things are configured correctly. No idea, where would I configure the size? Maybe just precaution years ago?
May 13, 20251 yr 7 minutes ago, Maertuerer said: No idea, where would I configure the size? Maybe just precaution years ago? According to those diagnostics, it was only using 15G of the 200G. If usage isn't growing, we can take a look at that later. How many containers do you have?
May 13, 20251 yr Author 9 minutes ago, trurl said: According to those diagnostics, it was only using 15G of the 200G. If usage isn't growing, we can take a look at that later. How many containers do you have? 18, of which only 6 were running all the time.
May 13, 20251 yr Author Is this someting useful? root@NAS:~# dmesg | grep -i error [ 0.422417] ACPI BIOS Error (bug): AE_AML_BUFFER_LIMIT, Field [CAP1] at bit offset/length 64/32 exceeds size of target Buffer (64 bits) (20220331/dsopcode-198) [ 0.422428] ACPI Error: Aborting method \_SB._OSC due to previous error (AE_AML_BUFFER_LIMIT) (20220331/psparse-529)
May 13, 20251 yr 4 minutes ago, Maertuerer said: 18, of which only 6 were running all the time. Do any of them have a host path that isn't to a user share?
May 13, 20251 yr Author 10 minutes ago, trurl said: Do any of them have a host path that isn't to a user share? I am not completely understanding that question (noob), but is this what you are inquiring about? Both homarr settings, as well as gitlab data are gone Vaultwarden is still completely fine: /data/mnt/user/appdata/vaultwarden Data in /mnt/disk1/JDownloaderDownloads is fine, Data in /mnt/disk1/appdata/jdownloader2/ has been reset So it looks like only docker data in /mnt/disk1/appdata/ has been reset to the default values? Unfortunately, for homarr that would not explain the problem... Edited May 13, 20251 yr by Maertuerer
May 13, 20251 yr Looks like you have the appdata backup plugin installed. So do you have backups of your appdata?
May 13, 20251 yr Author 2 minutes ago, trurl said: Looks like you have the appdata backup plugin installed. So do you have backups of your appdata? YES!!!! In there I got vaultwarden (didn't break) but also homarr and plex (which broke). Gitlab was not configured to backup external Data .... downloader was actually set to be skipped during backup .... Just gotta look into restoring the backup and reconfigure the missing backups correctly. No real data loss in gitlab, since I also have everything locally, but still sucks. THANK YOU VERY MUCH FOR YOUR SUPPORT, Ricky!
May 13, 20251 yr Author So, I tried to delete unused docker images and I am getting execution errors and the UI is Almost unusable now, CPU at 100% To me it feels like the SSD is dying right now nas-diagnostics-20250513-1755.zip Edited May 13, 20251 yr by Maertuerer
May 13, 20251 yr Just noticed you have nvme as disk1. SSDs in the array cannot be trimmed, and can only be written at parity speed (you don't have parity). The best place for SSDs is in pools outside the array. But, since you are on an old version of Unraid, you are required to have an array. Not clear there is anything wrong with disk1 though. You might try deleting and recreating docker.img (at only 30G this time) https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file then reinstalling your containers from Previous Apps. https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also note https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks
May 13, 20251 yr Author 28 minutes ago, trurl said: Just noticed you have nvme as disk1. SSDs in the array cannot be trimmed, and can only be written at parity speed (you don't have parity). The best place for SSDs is in pools outside the array. But, since you are on an old version of Unraid, you are required to have an array. Not clear there is anything wrong with disk1 though. You might try deleting and recreating docker.img (at only 30G this time) https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file then reinstalling your containers from Previous Apps. https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also note https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks Yeah, I had to use the SSD in the array... Might as well try to update to the most recent version, if I then can move the nvme from array to pool and keep using it. Is there a manual way to call trim on the ssd?
May 13, 20251 yr 3 minutes ago, Maertuerer said: Is there a manual way to call trim on the ssd? I think you would have to remove it from the array first
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.