November 25, 20241 yr Hello. I'm having some problems with the mover. This is the 2nd time in a week or so that it's seemingly gotten "stuck", preventing me from stopping the array to add a disk. The last time I used the physical power button on the computer to trigger a shutdown, which took a while but worked. I'm quite new to Unraid, but fairly familiar with Linux, so I've tried to diagnose it, but I'm stumped. Unfortunately, I had just turned mover logging off yesterday, since I hadn't had any issues with it for a while. There are a number of mover related processes running: 11637 ? S 0:00 /bin/sh -c /usr/local/sbin/mover &> /dev/null 11640 ? S 0:00 /bin/bash /usr/local/sbin/mover 11650 ? S 0:00 /bin/bash /usr/local/sbin/mover 12754 ? S 0:01 /usr/local/bin/move All of which started at 3:40am, when the mover was scheduled for. root@unraid:~# ps -p 11637,11640,11650,12754 -o pid,lstart PID STARTED 11637 Mon Nov 25 03:40:01 2024 11640 Mon Nov 25 03:40:01 2024 11650 Mon Nov 25 03:40:01 2024 12754 Mon Nov 25 03:40:19 2024 It's around 9:30am now, and the cache has ~3g on it, which hasn't been doing up or down any significant amount. root@unraid:~# du -sc /mnt/cache/* 3197044 /mnt/cache/appdata 3197044 total Checking what files the mover processes are accessing only really tells me that `move` is accessing `/mnt/user`, but nothing more. root@unraid:~# lsof -p 11640,11650,12754 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME mover 11640 root cwd DIR 0,2 200 4 /root mover 11640 root rtd DIR 0,2 460 1 / mover 11640 root txt REG 0,2 1251024 485 /bin/bash mover 11640 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so mover 11640 root mem REG 0,2 201128 2118 /lib64/libtinfo.so.6.4 mover 11640 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so mover 11640 root 0u CHR 1,3 0t0 4 /dev/null mover 11640 root 1w CHR 1,3 0t0 4 /dev/null mover 11640 root 2w CHR 1,3 0t0 4 /dev/null mover 11640 root 255r REG 0,27 4028 4731 /usr/local/sbin/mover mover 11650 root cwd DIR 0,2 200 4 /root mover 11650 root rtd DIR 0,2 460 1 / mover 11650 root txt REG 0,2 1251024 485 /bin/bash mover 11650 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so mover 11650 root mem REG 0,2 201128 2118 /lib64/libtinfo.so.6.4 mover 11650 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so mover 11650 root 0r FIFO 0,12 0t0 123375900 pipe mover 11650 root 1w CHR 1,3 0t0 4 /dev/null mover 11650 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root cwd DIR 0,2 200 4 /root move 12754 root rtd DIR 0,2 460 1 / move 12754 root txt REG 0,27 35304 3937 /usr/local/bin/move move 12754 root mem REG 7,1 3937 /usr/local/bin/move (path dev=0,27) move 12754 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so move 12754 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so move 12754 root 0r FIFO 0,12 0t0 123509987 pipe move 12754 root 1w CHR 1,3 0t0 4 /dev/null move 12754 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root 3r DIR 0,42 97 648799821318062208 /mnt/user As far as shares, I have: `appdata` set to primary cache, secondary array, move array -> cache `system` (only the standard docker files) set to primary cache, secondary array, move cache -> array. `domains` (empty) set to primary cache, secondary array, move cache -> array. None of the other shares specify cache. The array has 1 parity and 3 storage disks (xfs), and the cache pool has 2 SSDs (btrfs) for redundancy. I have quite a few docker containers (some from apps, some my own custom ones) running. My next step in debugging would be to stop the docker service and see if the mover seems to do anything, but I'm not sure if disabling and re-enabling docker through the Unraid GUI will retain all my app and container configs, even custom containers I added? Is that safe? Is there any other way to see what activity the mover is doing? Other suggesting for getting it unstuck, short of rebooting again via the power button?
November 25, 20241 yr Community Expert 3 minutes ago, SurprisingEdge said: I had just turned mover logging off yesterday Enable it again and post the diags after it happens, difficult to say what the issue could be without that.
November 25, 20241 yr Author I already re-enabled it, but my understanding is that until the mover starts again, the change won't take effect? And I don't think it'll start again until this gets un-stuck? Are there any options for investigating this current stuck mover, or is "press the power button" my only path forward?
November 25, 20241 yr Author I've tried `mover stop`, and it's ended up doing the same thing as last time: There's a `move` process still running with a file handle on `/mnt/user` that will prevent me from stopping the array. root@unraid:~# ps ax | grep move 3130 ? S 0:00 inotifywait -q /boot/changes.txt -e move_self,delete_self,modify 11637 ? S 0:00 /bin/sh -c /usr/local/sbin/mover &> /dev/null 11640 ? S 0:00 /bin/bash /usr/local/sbin/mover 11650 ? S 0:00 /bin/bash /usr/local/sbin/mover 12754 ? S 0:01 /usr/local/bin/move 22528 pts/0 S+ 0:00 grep move root@unraid:~# mover stop mover: stopped root@unraid:~# ps ax | grep move 3130 ? S 0:00 inotifywait -q /boot/changes.txt -e move_self,delete_self,modify 12754 ? D 0:01 /usr/local/bin/move 23174 pts/0 S+ 0:00 grep move root@unraid:~# lsof -p 12754 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME move 12754 root cwd DIR 0,2 220 4 /root move 12754 root rtd DIR 0,2 460 1 / move 12754 root txt REG 0,27 35304 3937 /usr/local/bin/move move 12754 root mem REG 7,1 3937 /usr/local/bin/move (path dev=0,27) move 12754 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so move 12754 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so move 12754 root 0r FIFO 0,12 0t0 123509987 pipe move 12754 root 1w CHR 1,3 0t0 4 /dev/null move 12754 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root 3r DIR 0,42 97 648799821318062208 /mnt/user That `move` process is still there 20 minutes later. I haven't tried stopping the array because I know it won't be able to due to that `/mnt/user` file handle. This was where I had to initiate a physical reboot, last time. Before trying to reboot, is it safe to disable and re-enable docker, or will I lose any config (custom containers?) if I do that?
November 25, 20241 yr Author For future reference: Stopping all containers and disabling docker hasn't unstuck the `move` process. Disabling and re-enabling docker retained all the config as far as I can tell, including my custom containers. I'll do a physical reboot to un-stick the mover and get the logging enablement to kick in. root@unraid:~# ps ax | grep move 1549 pts/0 S+ 0:00 grep move 3130 ? S 0:00 inotifywait -q /boot/changes.txt -e move_self,delete_self,modify 12754 ? D 0:01 /usr/local/bin/move root@unraid:~# lsof -p 12754 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME move 12754 root cwd DIR 0,2 220 4 /root move 12754 root rtd DIR 0,2 460 1 / move 12754 root txt REG 0,27 35304 3937 /usr/local/bin/move move 12754 root mem REG 7,1 3937 /usr/local/bin/move (path dev=0,27) move 12754 root mem REG 0,2 2314520 2055 /lib64/libc-2.37.so move 12754 root mem REG 0,2 235976 2049 /lib64/ld-2.37.so move 12754 root 0r FIFO 0,12 0t0 123509987 pipe move 12754 root 1w CHR 1,3 0t0 4 /dev/null move 12754 root 2w CHR 1,3 0t0 4 /dev/null move 12754 root 3r DIR 0,42 97 648799821318062208 /mnt/user
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.