Michael_P

Whatever container was running that java instance ran the host OOM, if you were doing something at the time, then you can narrow it down - else you can run ps auxf from the CLI to figure out which container is kicking it off Mar 4 19:12:18 UnraidZFS kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Mar 4 19:12:18 UnraidZFS kernel: [1130142] 1001 1130142 1843 898 53248 0 0 bash Mar 4 19:12:18 UnraidZFS kernel: [1130845] 1001 1130845 723 389 40960 0 0 sh Mar 4 19:12:18 UnraidZFS kernel: [1131059] 1001 1131059 5665517 4073503 33357824 0 0 java Mar 4 19:12:18 UnraidZFS kernel: [1131387] 1001 1131387 1451 449 45056 0 0 tail Mar 4 19:12:18 UnraidZFS kernel: [1157589] 1001 1157589 723 389 40960 0 0 sh Mar 4 19:12:18 UnraidZFS kernel: [1270535] 1001 1270535 723 390 40960 0 0 sh Mar 4 19:12:18 UnraidZFS kernel: [1270541] 1001 1270541 723 389 40960 0 0 ldapsearch Mar 4 19:12:18 UnraidZFS kernel: [1270543] 1001 1270543 723 387 45056 0 0 _client-script. Mar 4 19:12:18 UnraidZFS kernel: [1270576] 1001 1270576 4994726 122374 1310720 0 0 java Mar 4 19:12:18 UnraidZFS kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=a36642b942ed2e5ad6e110c86d428bebeaaf36997e6a55d45fab3878a001182e,mems_allowed=0,oom_memcg=/docker/a36642b942ed2e5ad6e110c86d428bebeaaf36997e6a55d45fab3878a001182e,task_memcg=/docker/a36642b942ed2e5ad6e110c86d428bebeaaf36997e6a55d45fab3878a001182e,task=java,pid=1131059,uid=1001 Mar 4 19:12:18 UnraidZFS kernel: Memory cgroup out of memory: Killed process 1131059 (java) total-vm:22662068kB, anon-rss:16269568kB, file-rss:24444kB, shmem-rss:0kB, UID:1001 pgtables:32576kB oom_score_adj:0

...OK, but it's killed it 3 times starting on back on the 1st with the last being at 12:29 yesterday so... Mar 2 12:29:06 Zotarios kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Mar 2 12:29:06 Zotarios kernel: [ 20873] 0 20873 109 0 28672 0 0 s6-svscan Mar 2 12:29:06 Zotarios kernel: [ 21161] 0 21161 55 0 24576 0 0 s6-supervise Mar 2 12:29:06 Zotarios kernel: [ 21164] 0 21164 52 0 24576 0 0 s6-linux-init-s Mar 2 12:29:06 Zotarios kernel: [ 21198] 0 21198 55 0 24576 0 0 s6-supervise Mar 2 12:29:06 Zotarios kernel: [ 21200] 0 21200 55 0 24576 0 0 s6-supervise Mar 2 12:29:06 Zotarios kernel: [ 21201] 0 21201 55 0 24576 0 0 s6-supervise Mar 2 12:29:06 Zotarios kernel: [ 21202] 0 21202 55 0 24576 0 0 s6-supervise Mar 2 12:29:06 Zotarios kernel: [ 21222] 0 21222 52 0 24576 0 0 s6-ipcserverd Mar 2 12:29:06 Zotarios kernel: [ 21477] 0 21477 1835 160 53248 0 0 bash Mar 2 12:29:06 Zotarios kernel: [ 21496] 0 21496 1421 32 49152 0 0 sleep Mar 2 12:29:06 Zotarios kernel: [3644454] 99 3644454 7343318 6169236 49897472 0 0 Plex Media Serv Mar 2 12:29:06 Zotarios kernel: [3644504] 99 3644504 16382 8377 143360 0 0 Plex Script Hos Mar 2 12:29:06 Zotarios kernel: [3644560] 99 3644560 16035 8098 143360 0 0 Plex Script Hos Mar 2 12:29:06 Zotarios kernel: [3644719] 99 3644719 10725 480 94208 0 0 Plex Tuner Serv Mar 2 12:29:06 Zotarios kernel: [3645595] 99 3645595 2243091 45438 581632 0 0 Plex Transcoder Mar 2 12:29:06 Zotarios kernel: [3646274] 99 3646274 2244279 48306 598016 0 0 Plex Transcoder Mar 2 12:29:06 Zotarios kernel: [3646796] 99 3646796 650 277 45056 0 0 EasyAudioEncode Mar 2 12:29:06 Zotarios kernel: [3646901] 99 3646901 2243991 46613 573440 0 0 Plex Transcoder Mar 2 12:29:06 Zotarios kernel: [3647491] 99 3647491 12825 2091 135168 0 0 Plex Transcoder Mar 2 12:29:06 Zotarios kernel: [3652223] 99 3652223 14089 1715 147456 0 0 Plex Transcoder Mar 2 12:29:06 Zotarios kernel: [3653741] 99 3653741 14687 2246 155648 0 0 Plex Transcoder Mar 2 12:29:06 Zotarios kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/docker/9aa73a6ac4ac029a8a5b2fe5e8319b94cc5623094e9cd30d992f3765e29ee251,task_memcg=/docker/9aa73a6ac4ac029a8a5b2fe5e8319b94cc5623094e9cd30d992f3765e29ee251,task=Plex Media Serv,pid=3644454,uid=99 Mar 2 12:29:06 Zotarios kernel: Memory cgroup out of memory: Killed process 3644454 (Plex Media Serv) total-vm:29373272kB, anon-rss:24667224kB, file-rss:244kB, shmem-rss:9476kB, UID:99 pgtables:48728kB oom_score_adj:0 Mar 2 12:29:09 Zotarios kernel: oom_reaper: reaped process 3644454 (Plex Media Serv), now anon-rss:136kB, file-rss:244kB, shmem-rss:36kB

Looks like you have Plex transcoding to RAM, and multiple streams are running your system OOm

Can't answer the icloud bit, but deleting it from device storage should move it to the trash in nextcloud - I've never actually seen it delete anything server side tho, not sure if there's a new sync setting but everything I delete on the phone stays in the correct folder on the server.

No idea about the error, but disabling the trash bin makes nextcloud hold on to the file if it's deleted on the device indefinitely instead of 30 days or whatever. It's not a true backup solution in the sense that it just mirrors the data on whatever device you're syncing - delete it on the device, it deletes it on nextcloud. Having the trash bin gives you the opportunity to un-delete it or store it there until you delete it yourself manually.

Seems more of a Windows issue than an Unraid issue - have you run memtest on your Windows box

You can change the behavior, just google 'unraid power button behavior'

Doesn't have to be an animal, a bug can short it. I had a media center machine in a closet that kept turning itself off randomly during the night, and Windows reported a power button push as the source of the shutdown. Turns out the button was sticking juuuuuust enough to let the machine run but with enough thermal expansion/vibration/farts it would make contact and shut down

Could be a button going bad (I've had it happen more than once), or a critter ran across the terminals

Suggested it because If the batteries are on their way out and the power blinks it wouldn't alert it'd just power off/reset

A lot of call traces, so it's hardware related - try running it without your GPU

It would That's up to you

For images and video from the phones, I use Nextcloud - just disable trashbin_retention_obligation and it will just move deleted stuff to the trash and store it there forever

When it's running, try this from the command prompt then post the ps.txt file (sanitize any usernames or passwords) ps auxf > ps.txt

Only troubleshooting I can suggest is to narrow down which container is a problem, by turning them off and seeing what happens - and/or narrowing down what they're doing when the system starts to crawl

You have a call trace in there which might have something to do with it, and some of the, um, files, you're downloading has file names that are too long. How close do you normally run to your max system RAM?

ps auxf might be able to tell you what's kicking it off

Is your backup script writing to syslog, because it's what's filling it up

FWIW- if it was refurbished properly and stamped on the drive 'refurbished' or whatever, then the factory will have wiped SMART data, and FARM data as well, to reset the drive to zero

The OOM reaper is killing your immich process and also your VM - it's likely whatever immich is doing running the host OOM Feb 13 00:00:23 Tower kernel: Out of memory: Killed process 30380 (immich) total-vm:23196948kB, anon-rss:6242364kB, file-rss:52kB, shmem-rss:0kB, UID:99 pgtables:25348kB oom_score_adj:0 Feb 13 00:01:47 Tower kernel: Out of memory: Killed process 688879 (immich) total-vm:23163516kB, anon-rss:6202436kB, file-rss:4080kB, shmem-rss:0kB, UID:99 pgtables:23980kB oom_score_adj:0 Feb 13 04:30:47 Tower kernel: Out of memory: Killed process 12416 (qemu-system-x86) total-vm:6920960kB, anon-rss:2190524kB, file-rss:512kB, shmem-rss:7168kB, UID:0 pgtables:5576kB oom_score_adj:

And every time you run the UPS on the battery, it degrades the bejeezus out of them - so you want to use them as little as possible.

Whatever cope blanket helps you sleep at night, you do you.

Check your logs for Unraid shutdown timeout exceeded or whatever phrase UR uses to make sure it's actually not timing out. Also, you should set it to initiate a shutdown when switched to UPS power as quickly as possible instead of by percentage of battery remaining, 30 to 60 seconds max

Ya, like I said - it's bad. Your drive has reallocated sectors? Guess what, it's bad too - just hasn't bit you (pun!). It failed SMART, it's bad. Are there shades of bad, sure but like @Frank1940 said, what's your tolerance for risk. 2 parity drives are nice, but you'll be holding your breath if you need that bad drive to restore. BTW, I'm not some rich dude and don't do any of this for a living, but I have been doing this long enough (my first HDD was 200MB) to know that for 200 bux it's not worth the hassle. Reallocated sectors are a critical parameter in SMART for a reason.

Could be the UPS batteries, too if they haven't been replaced in a while