SurprisingEdge

I found help elsewhere and figured this out. My BIOS has two options for SecureBoot: Microsoft and Other OS. It's been set to Microsoft forever. Always worked fine. For some reason, it stopped working fine. Changing it to Other OS fixed things. Unraid boots fine now. Notably, the official troubleshooting doc for boot issues doesn't once mention Secure Boot. 😒

I have a computer that has suddenly started refusing to boot Unraid. An Ubuntu install image flashed onto a new USB stick works, but the Unraid installer (7.2.3 or 7.1.4) won't boot. The computer detects the USB stick, and I can prompt it to try UEFI or legacy booting the stick, but it just kicks me back to the bios with no indication why it's failing. I'm reluctant to tamper with the original USB stick, but I've tried two new sticks, a 128GB and a 16GB, with various combinations of things: Using Rufus to format the 128GB as Fat32 before manually coping the Unraid files Starting with flashing the Ubuntu image and then deleting the files and copying Unraid on. Copying those Unraid files from a fresh 7.1.4 install zip, 7.2.3 install zip, as well as from the old USB stick. Copying the "previous" directory from the old USB stick over the root to "downgrade" from 7.2.3 to 7.1.4 I've also tried a bunch of BIOS options. "Fast boot" has been off since long before this. I've tried disabling the CSM. Nothing has helped. The motherboard is a Asus Z87-PRO, for whatever that's worth. I'm at a loss. I've got TB of data on this server, so I'm a bit panicked about not being able to get Unraid to boot at all. It was running fine on 7.1.4 until I hit upgrade, and then it wouldn't boot.

Na, these errors happened before the upgrade. This isn't the first time I've had them either. I have not tried editing the nginx config, that's a can of worms I would really rather not open. So far the only thing I seem to have found that stops them is going to every one of my devices that might have a browser tab open to my Unraid server, and closing all the tabs. It's like there's way too much resource consumption on the server for every browser tab open. I've been limiting how many tabs I have open and haven't had the error since, but I only had like 5 tabs open before so it really doesn't seem like it should be a problem. The server itself has 32GB and is only ever showing like max 30% memory usage. I'm not running VMs, just a bunch of docker containers.

I'm not sure why I would need to downgrade. This error was happening on 6.12.13, before I upgraded.

I'm not running the v7 beta. I just updated from 6.12.13 to 6.12.14. I've updated and rebooted since the problem happened, so there's not much in the logs right now, but I'll attach diagnostics anyway. unraid-diagnostics-20241211-1126.zip

If I tried to modify the nginx config to fix this, wouldn't it just get overwritten at the next reboot? I haven't modified my Unraid nginx config. Is it possible that some App/Plugin I installed modified it? How would I tell? Skimming through all the files in /etc/nginx (including conf.d) I don't see any mentions of any of the Apps/Plugins I have installed. I don't think they've modified it. It's clear that both a) something is running out of memory, and b) the (ram)disk is out of space. The latter is caused by all the logs from the former, that's no mystery. Why is nginx running out of memory is the question.

My /var/log/syslog (and .1, .2) are full of 70MB+ of this error starting about about 22 hours ago: Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [error] 14669#14669: *9624309 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost" Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [crit] 14669#14669: ngx_slab_alloc() failed: no memory Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [error] 14669#14669: shpool alloc failed Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [error] 14669#14669: nchan: Out of shared memory while allocating channel /cpuload. Increase nchan_max_reserved_memory. I've seen this come up a lot on the forums without any real resolution. Prior to this, my logs were also filling up with: Dec 8 20:05:39 unraid nginx: 2024/12/08 20:05:39 [alert] 13011#13011: worker process 10895 exited on signal 6 Dec 8 20:05:41 unraid nginx: 2024/12/08 20:05:41 [alert] 13011#13011: worker process 10896 exited on signal 6 Dec 8 20:05:41 unraid nginx: 2024/12/08 20:05:41 [alert] 13011#13011: worker process 10917 exited on signal 6 Dec 8 20:05:41 unraid nginx: 2024/12/08 20:05:41 [alert] 13011#13011: worker process 10918 exited on signal 6 I can delete the logs to free up space so I can see new logs, but that won't change the fact that this is filling up my logs. Has there been any "official" answer about what causes either of these and how to resolve them?

For future reference: Stopping all containers and disabling docker hasn't unstuck the `move` process. Disabling and re-enabling docker retained all the config as far as I can tell, including my custom containers. I'll do a physical reboot to un-stick the mover and get the logging enablement to kick in. root@unraid:~# ps ax | grep move 1549 pts/0 S+ 0:00 grep move 3130 ? S 0:00 inotifywait -q /boot/changes.txt -e move_self,delete_self,modify 12754 ? D 0:01 /usr/local/bin/move root@unraid:~# lsof -p 12754 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME move 12754 root cwd DIR 0,2 220 4 /root move 12754 root rtd DIR 0,2 460 1 / move 12754 root txt REG 0,27 35304 3937 /usr/local/bin/move move 12754 root mem REG 7,1 3937 /usr/local/bin/move (path dev=0,27) move 12754 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so move 12754 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so move 12754 root 0r FIFO 0,12 0t0 123509987 pipe move 12754 root 1w CHR 1,3 0t0 4 /dev/null move 12754 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root 3r DIR 0,42 97 648799821318062208 /mnt/user

I've tried `mover stop`, and it's ended up doing the same thing as last time: There's a `move` process still running with a file handle on `/mnt/user` that will prevent me from stopping the array. root@unraid:~# ps ax | grep move 3130 ? S 0:00 inotifywait -q /boot/changes.txt -e move_self,delete_self,modify 11637 ? S 0:00 /bin/sh -c /usr/local/sbin/mover &> /dev/null 11640 ? S 0:00 /bin/bash /usr/local/sbin/mover 11650 ? S 0:00 /bin/bash /usr/local/sbin/mover 12754 ? S 0:01 /usr/local/bin/move 22528 pts/0 S+ 0:00 grep move root@unraid:~# mover stop mover: stopped root@unraid:~# ps ax | grep move 3130 ? S 0:00 inotifywait -q /boot/changes.txt -e move_self,delete_self,modify 12754 ? D 0:01 /usr/local/bin/move 23174 pts/0 S+ 0:00 grep move root@unraid:~# lsof -p 12754 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME move 12754 root cwd DIR 0,2 220 4 /root move 12754 root rtd DIR 0,2 460 1 / move 12754 root txt REG 0,27 35304 3937 /usr/local/bin/move move 12754 root mem REG 7,1 3937 /usr/local/bin/move (path dev=0,27) move 12754 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so move 12754 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so move 12754 root 0r FIFO 0,12 0t0 123509987 pipe move 12754 root 1w CHR 1,3 0t0 4 /dev/null move 12754 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root 3r DIR 0,42 97 648799821318062208 /mnt/user That `move` process is still there 20 minutes later. I haven't tried stopping the array because I know it won't be able to due to that `/mnt/user` file handle. This was where I had to initiate a physical reboot, last time. Before trying to reboot, is it safe to disable and re-enable docker, or will I lose any config (custom containers?) if I do that?

I already re-enabled it, but my understanding is that until the mover starts again, the change won't take effect? And I don't think it'll start again until this gets un-stuck? Are there any options for investigating this current stuck mover, or is "press the power button" my only path forward?

Hello. I'm having some problems with the mover. This is the 2nd time in a week or so that it's seemingly gotten "stuck", preventing me from stopping the array to add a disk. The last time I used the physical power button on the computer to trigger a shutdown, which took a while but worked. I'm quite new to Unraid, but fairly familiar with Linux, so I've tried to diagnose it, but I'm stumped. Unfortunately, I had just turned mover logging off yesterday, since I hadn't had any issues with it for a while. There are a number of mover related processes running: 11637 ? S 0:00 /bin/sh -c /usr/local/sbin/mover &> /dev/null 11640 ? S 0:00 /bin/bash /usr/local/sbin/mover 11650 ? S 0:00 /bin/bash /usr/local/sbin/mover 12754 ? S 0:01 /usr/local/bin/move All of which started at 3:40am, when the mover was scheduled for. root@unraid:~# ps -p 11637,11640,11650,12754 -o pid,lstart PID STARTED 11637 Mon Nov 25 03:40:01 2024 11640 Mon Nov 25 03:40:01 2024 11650 Mon Nov 25 03:40:01 2024 12754 Mon Nov 25 03:40:19 2024 It's around 9:30am now, and the cache has ~3g on it, which hasn't been doing up or down any significant amount. root@unraid:~# du -sc /mnt/cache/* 3197044 /mnt/cache/appdata 3197044 total Checking what files the mover processes are accessing only really tells me that `move` is accessing `/mnt/user`, but nothing more. root@unraid:~# lsof -p 11640,11650,12754 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME mover 11640 root cwd DIR 0,2 200 4 /root mover 11640 root rtd DIR 0,2 460 1 / mover 11640 root txt REG 0,2 1251024 485 /bin/bash mover 11640 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so mover 11640 root mem REG 0,2 201128 2118 /lib64/libtinfo.so.6.4 mover 11640 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so mover 11640 root 0u CHR 1,3 0t0 4 /dev/null mover 11640 root 1w CHR 1,3 0t0 4 /dev/null mover 11640 root 2w CHR 1,3 0t0 4 /dev/null mover 11640 root 255r REG 0,27 4028 4731 /usr/local/sbin/mover mover 11650 root cwd DIR 0,2 200 4 /root mover 11650 root rtd DIR 0,2 460 1 / mover 11650 root txt REG 0,2 1251024 485 /bin/bash mover 11650 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so mover 11650 root mem REG 0,2 201128 2118 /lib64/libtinfo.so.6.4 mover 11650 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so mover 11650 root 0r FIFO 0,12 0t0 123375900 pipe mover 11650 root 1w CHR 1,3 0t0 4 /dev/null mover 11650 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root cwd DIR 0,2 200 4 /root move 12754 root rtd DIR 0,2 460 1 / move 12754 root txt REG 0,27 35304 3937 /usr/local/bin/move move 12754 root mem REG 7,1 3937 /usr/local/bin/move (path dev=0,27) move 12754 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so move 12754 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so move 12754 root 0r FIFO 0,12 0t0 123509987 pipe move 12754 root 1w CHR 1,3 0t0 4 /dev/null move 12754 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root 3r DIR 0,42 97 648799821318062208 /mnt/user As far as shares, I have: `appdata` set to primary cache, secondary array, move array -> cache `system` (only the standard docker files) set to primary cache, secondary array, move cache -> array. `domains` (empty) set to primary cache, secondary array, move cache -> array. None of the other shares specify cache. The array has 1 parity and 3 storage disks (xfs), and the cache pool has 2 SSDs (btrfs) for redundancy. I have quite a few docker containers (some from apps, some my own custom ones) running. My next step in debugging would be to stop the docker service and see if the mover seems to do anything, but I'm not sure if disabling and re-enabling docker through the Unraid GUI will retain all my app and container configs, even custom containers I added? Is that safe? Is there any other way to see what activity the mover is doing? Other suggesting for getting it unstuck, short of rebooting again via the power button?