May 2, 201412 yr I'm not sure where to post this ... because I'm not clear whether the fault lies. I feel that unRAID itself is to blame, but then Xen/ArchVM and the powerdown package are all involved too. If unRAID is shutdown under the control of apcupsd/powerdown, then it all works as expected. Similarly, if I shutdown by a short press of the power button, everything shuts down cleanly and the machine powers off within around 40-60 seconds (is this also under the control of the powerdown script?). However, if I attempt to stop the array from the emhttp interface, or with the shutdown command, then the procedure hangs for ever, waiting for one of my Arch VMs to shutdown. The relevant part of syslog: Apr 30 08:27:34 Tower emhttp: shcmd (74): /usr/local/sbin/emhttp_event stopping_svcs Apr 30 08:27:34 Tower kernel: mdcmd (53): nocheck Apr 30 08:27:34 Tower kernel: md: nocheck_array: check not active Apr 30 08:27:34 Tower emhttp_event: stopping_svcs Apr 30 08:27:34 Tower emhttp: Stop AVAHI... Apr 30 08:27:34 Tower emhttp: shcmd (75): /etc/rc.d/rc.avahidaemon stop |& logger Apr 30 08:27:34 Tower logger: Stopping Avahi mDNS/DNS-SD Daemon: stopped Apr 30 08:27:34 Tower avahi-daemon[2733]: Got SIGTERM, quitting. Apr 30 08:27:34 Tower avahi-dnsconfd[2741]: read(): EOF Apr 30 08:27:34 Tower avahi-daemon[2733]: Leaving mDNS multicast group on interface br0.IPv4 with address 10.2.0.100. Apr 30 08:27:34 Tower avahi-daemon[2733]: avahi-daemon 0.6.31 exiting. Apr 30 08:27:34 Tower emhttp: shcmd (76): /etc/rc.d/rc.avahidnsconfd stop |& logger Apr 30 08:27:34 Tower logger: Stopping Avahi mDNS/DNS-SD DNS Server Configuration Daemon: stopped Apr 30 08:27:34 Tower emhttp: shcmd (77): ps axc | grep -q rpc.mountd Apr 30 08:27:34 Tower emhttp: Stop NFS... Apr 30 08:27:34 Tower emhttp: shcmd (78): /etc/rc.d/rc.nfsd stop |& logger Apr 30 08:27:34 Tower rpc.mountd[3228]: Caught signal 15, un-registering and exiting. Apr 30 08:27:35 Tower emhttp: Stop SMB... Apr 30 08:27:35 Tower emhttp: shcmd (79): /etc/rc.d/rc.samba stop |& logger Apr 30 08:27:35 Tower kernel: lockd: couldn't shutdown host module for net ffffffff81686ec0! Apr 30 08:27:35 Tower kernel: nfsd: last server has exited, flushing export cache Apr 30 08:27:35 Tower emhttp: shcmd (80): rm /etc/avahi/services/smb.service &> /dev/null Apr 30 08:27:35 Tower emhttp: Spinning up all drives... Apr 30 08:27:35 Tower emhttp: shcmd (81): /usr/sbin/hdparm -S0 /dev/sdd &> /dev/null Apr 30 08:27:35 Tower kernel: mdcmd (54): spinup 0 Apr 30 08:27:35 Tower kernel: mdcmd (55): spinup 1 Apr 30 08:27:35 Tower kernel: mdcmd (56): spinup 2 Apr 30 08:27:35 Tower kernel: mdcmd (57): spinup 3 Apr 30 08:27:35 Tower kernel: mdcmd (58): spinup 4 Apr 30 08:27:35 Tower kernel: mdcmd (59): spinup 5 Apr 30 08:27:36 Tower emhttp: Sync filesystems... Apr 30 08:27:36 Tower emhttp: shcmd (82): sync Apr 30 08:27:45 Tower emhttp: shcmd (83): /usr/local/sbin/emhttp_event unmounting_disks Apr 30 08:27:45 Tower emhttp_event: unmounting_disks Apr 30 08:27:45 Tower rc.fan_speed: WARNING: fan_speed called to stop with SERVICE not = disabled Apr 30 08:27:45 Tower rc.unRAID[5425][5426]: Processing /etc/rc.d/rc.unRAID.d/ kill scripts. Apr 30 08:27:45 Tower rc.unRAID[5425][5430]: Running: "/etc/rc.d/rc.unRAID.d/K00.sh" Apr 30 08:27:45 Tower rc.unRAID[5425][5433]: Shutting down domain 1 Apr 30 08:27:45 Tower rc.unRAID[5425][5433]: Shutting down domain 2 Apr 30 08:27:45 Tower rc.unRAID[5425][5433]: Waiting for 2 domains Apr 30 08:28:05 Tower kernel: br0: port 2(vif1.0) entered disabled state Apr 30 08:28:05 Tower rc.unRAID[5425][5433]: Domain 1 has been shut down, reason code 0 Apr 30 08:28:05 Tower kernel: br0: port 2(vif1.0) entered disabled state Apr 30 08:28:05 Tower kernel: device vif1.0 left promiscuous mode Apr 30 08:28:05 Tower kernel: br0: port 2(vif1.0) entered disabled state Apr 30 08:28:05 Tower logger: /etc/xen/scripts/vif-bridge: offline type_if=vif XENBUS_PATH=backend/vif/1/0 Apr 30 08:28:06 Tower logger: /etc/xen/scripts/vif-bridge: brctl delif br0 vif1.0 failed Apr 30 08:28:06 Tower logger: /etc/xen/scripts/vif-bridge: ifconfig vif1.0 down failed Apr 30 08:28:06 Tower logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge offline for vif1.0, bridge br0. As you can see, Domain 1 shuts down, but Domain 2 doesn't. Both domains are running IronicBadger's ArchVM If I 'xl console 2', then I see the following output on the terminal: root@Tower:~# xl console 2 [ 0.174846] drivers/rtc/hctosys.c: unable to open rtc device (rtc0) :: running early hook [udev] :: running hook [udev] :: Triggering uevents... :: performing fsck on '/dev/xvda1' /dev/xvda1: clean, 65133/915712 files, 622065/3661853 blocks ********************** WARNING ********************** * * * The root device is not configured to be mounted * * read-write! It may be fsck'd again later. * * * ***************************************************** :: mounting '/dev/xvda1' on real root :: running cleanup hook [udev] Welcome to Arch Linux! Expecting device dev-hvc0.device... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Reached target Network is Online. [ OK ] Reached target Paths. [ OK ] Reached target Encrypted Volumes. [ OK ] Set up automount Arbitrary Executable File Formats F...utomount Point. [ OK ] Reached target Swap. [ OK ] Created slice Root Slice. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Created slice System Slice. Starting File System Check on Root Device... Mounting POSIX Message Queue File System... Mounting Debug File System... Starting Apply Kernel Variables... Starting Create list of required static device nodes...rrent kernel... Starting Load Kernel Modules... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Setup Virtual Console... Starting udev Coldplug all Devices... Starting Journal Service... [ OK ] Started Journal Service. [ OK ] Reached target Slices. Mounting Temporary Directory... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required static device nodes ...current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Temporary Directory. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Load Kernel Modules. Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ 4.199866] systemd-fsck[83]: /dev/xvda1: clean, 65133/915712 files, 622065/3661853 blocks [ OK ] Started File System Check on Root Device. Starting Remount Root and Kernel File Systems... [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Reached target Local File Systems (Pre). Mounting RPC Pipe File System... [ OK ] Reached target Local File Systems. Starting Trigger Flushing of Journal to Persistent Storage... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Mounted RPC Pipe File System. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Trigger Flushing of Journal to Persistent Storage. [ OK ] Found device /dev/hvc0. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Arch Linux 3.14.1-1-ARCH (hvc0) ArchVM login: [ OK ] Stopped target RPC Port Mapper. [ OK ] Stopped target Graphical Interface. [ OK ] Stopped target Multi-User System. Stopping Deluge Daemon... Stopping NFS Mount Server Daemon... Stopping NFS v2/3 Network Status Monitor Daemon... [ OK ] Stopped NFS v2/3 Network Status Monitor Daemon. Stopping MariaDB database server... Stopping Automounts filesystems on demand... Stopping An automatic NZB and torrent movie downloader... [ OK ] Stopped An automatic NZB and torrent movie downloader. Stopping NFS v4 ID-Name Mapping Daemon... Stopping OpenSSH Daemon... Stopping Logitech Media Server Daemon... Stopping Login Service... [ OK ] Stopped target Login Prompts. Stopping Getty on tty1... Stopping Serial Getty on hvc0... Stopping D-Bus System Message Bus... Starting Generate shutdown-ramfs... [ OK ] Stopped Login Service. [ OK ] Stopped OpenSSH Daemon. [ OK ] Stopped Getty on tty1. [ OK ] Stopped Serial Getty on hvc0. [ OK ] Stopped NFS v4 ID-Name Mapping Daemon. Unmounting RPC Pipe File System... [ OK ] Removed slice system-serial\x2dgetty.slice. [ OK ] Removed slice system-getty.slice. Stopping Permit User Sessions... [ OK ] Unmounted RPC Pipe File System. [ OK ] Stopped Permit User Sessions. [ OK ] Stopped target Remote File Systems. Unmounting /net/tower/mnt/user/Music... Unmounting /net/tower/mnt/user/ReadyTorrents... Unmounting /net/tower/mnt/user/mysql... Unmounting /net/tower/mnt/user/Torrents... [ OK ] Failed unmounting /net/tower/mnt/user/ReadyTorrents. [ OK ] Failed unmounting /net/tower/mnt/user/Torrents. [ OK ] Stopped NFS Mount Server Daemon. Stopping NFS Server Daemon... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped NFS Server Daemon. Stopping RPC Bind... [ OK ] Stopped RPC Bind. [ OK ] Stopped Automounts filesystems on demand. [ OK ] Started Generate shutdown-ramfs. [ OK ] Stopped Logitech Media Server Daemon. [ OK ] Unmounted /net/tower/mnt/user/Music. [ OK ] Unmounted /net/tower/mnt/user/mysql. [ OK ] Stopped target Network is Online. [ OK ] Stopped target Network. Stopping dhcpcd on all interfaces... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dhcpcd on all interfaces. [ *** ] (1 of 2) A stop job is running for Deluge Daemon (3min 17s / 4min 30s)[61680.103948] INFO: task systemd:11392 blocked for more than 120 seconds. [61680.103962] Not tainted 3.14.1-1-ARCH #1 [61680.103966] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61680.104079] INFO: task mysqld:11420 blocked for more than 120 seconds. [61680.104086] Not tainted 3.14.1-1-ARCH #1 [61680.104090] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ **] (1 of 2) A stop job is running for Deluge Daemon (5min 17s / 6min)[61800.103941] INFO: task systemd:11392 blocked for more than 120 seconds. [61800.103954] Not tainted 3.14.1-1-ARCH #1 [61800.103958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61800.104071] INFO: task deluged:920 blocked for more than 120 seconds. [61800.104077] Not tainted 3.14.1-1-ARCH #1 [61800.104081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61800.104240] INFO: task mysqld:11420 blocked for more than 120 seconds. [61800.104246] Not tainted 3.14.1-1-ARCH #1 [61800.104250] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ OK ] Stopped MariaDB database server. [ OK ] Stopped Deluge Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Paths. [ OK ] Stopped target Timers. [ OK ] Stopped target Sockets. [ OK ] Closed Device-mapper event daemon FIFOs. [ OK ] Closed LVM2 metadata daemon socket. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. Stopping Update UTMP about System Boot/Shutdown... Stopping Load/Save Random Seed... Stopping Apply Kernel Variables... [ OK ] Stopped Apply Kernel Variables. Stopping Load Kernel Modules... [ OK ] Stopped Load Kernel Modules. [ OK ] Stopped target Encrypted Volumes. Stopping Setup Virtual Console... [ OK ] Stopped Setup Virtual Console. [ OK ] Stopped target Swap. [ OK ] Stopped Update UTMP about System Boot/Shutdown. [ OK ] Stopped Load/Save Random Seed. Stopping Create Volatile Files and Directories... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Unmounting NFS Configuration File System... Unmounting Temporary Directory... [ OK ] Stopped target Local File Systems (Pre). Stopping Remount Root and Kernel File Systems... [ OK ] Stopped Remount Root and Kernel File Systems. [ OK ] Reached target Shutdown. [61920.103910] INFO: task systemd:11392 blocked for more than 120 seconds. [61920.103925] Not tainted 3.14.1-1-ARCH #1 [61920.103929] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61920.104044] INFO: task deluged:920 blocked for more than 120 seconds. [61920.104052] Not tainted 3.14.1-1-ARCH #1 [61920.104056] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61920.104217] INFO: task mysqld:11420 blocked for more than 120 seconds. [61920.104223] Not tainted 3.14.1-1-ARCH #1 [61920.104227] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [62040.103918] INFO: task systemd:11392 blocked for more than 120 seconds. [62040.103932] Not tainted 3.14.1-1-ARCH #1 [62040.103937] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [62040.104054] INFO: task deluged:920 blocked for more than 120 seconds. [62040.104060] Not tainted 3.14.1-1-ARCH #1 [62040.104064] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. but it is not possible to enter any commands. It would appear, from the console output, that three processes are not shutting down: systemd, deluged and mysqld (MariaDB). 'xl shutdown 2', with the array running, works perfectly well, with VM Id 2 disappearing from the 'xl list'. All VMs are registered with the emhttp Xen Domains Extension, The questions in my mind are: 1) Why are these processes not stopping? 2) What do I need to do to get them to stop? 3) Why does powerdown achieve what appears to be a clean shutdown, while the emhttp 'Stop Array' fails? The emhttp 'Stop Array' appears to attempt to shutdown the VMs (through powerdown?), but it seems that this happens after the nfs and smb servers have already been stopped. The VM is accessing files via nfs - is it the fact that the nfs server has gone awol that causes the ArchVM to hang?
Archived
This topic is now archived and is closed to further replies.