unRAID 6.3.0 updating to 6.3.1 stuck on Sync Filesystems

[JohnnyT]

I was shutting my array down so I could reboot for the update and it is stuck at "Sync filesystems". Any ideas? I looked around the forum and did not find any definitive answers.

[John_M]

I've had that once. It turned out that I had a telnet session open and had issued a command similar to

 

cd /mnt/user/Share

 

and that was keeping the file system from unmounting. So best to make sure you've logged out of any remote sessions.

[JohnnyT]

As far as I know I do not have any active remote sessions. Is there a way to check this? I attached my diagnostics.

davidflix-diagnostics-20170209-1005.zip

[John_M]

I use this little plugin. It can be quite helpful.

 

I'll have a look at your diagnostics and if I see anything I'll get back.

 

[JohnnyT]

I do have that program but I can't use it as I am in a array shut down limbo and can't click on my plugins. Thanks for taking a look.

[John_M]

Try

 

poweroff

 

from the console. That's the official way to do as clean a shutdown as is possible. You have an unhappy server. Call traces, kernel oopses, out of memory.

 

[Squid]

A sync will fail / never complete if you have a program doing very heavy I/O (ie: a preclear in progress)

 

An open session, etc has nothing to do with this

 

Your best bet is to

powerdown -r

[JohnnyT]

I issued powerdown -r and am waiting to see what it does. Hoping I do not have to do another parity check, it takes 36 hours with 8tb drives :-(

 

 

root@davidFlix:~# powerdown -r

 

Broadcast message from root@davidFlix (pts/1) (Thu Feb  9 10:29:59 2017):

 

The system is going down for reboot NOW!

 

[John_M]

Is it doing heavy I/O? It seems to be repeatedly OOMing and killing off processes.

 

[Squid]

Is it doing heavy I/O? It seems to be repeatedly OOMing and killing off processes.

Who looks at diagnostics? 

[JohnnyT]

It should not be doing any heavy IO. I don't have anything special running. No preclears file moves etc. My cache is btrfs raid 10 with 4 drives, not sure it that matters. I issued the restart command and it just ignored it.

[John_M]

Regarding powerdown vs poweroff, straight from the horse's mouth:

 

Note that /usr/local/sbin/powerdown is a script that just invokes either /sbin/reboot or /sbin/poweroff and has been deprecated.

 

'powerdown' is a script formerly used by webGui/emhttp to initiate a gracefull power off or reboot.  The problem is that in it's original form, 'powerdown' relied on emhttp process to sequence the operation, but there were cases where this could not happen or proceeded very slowly.  Ultimately the system commands 'poweroff' or 'reboot' were finally invoked to complete the operation.

 

Anyway, the whole shutdown/poweroff/reboot operation was re-coded a couple releases ago so that now the "stock" linux reboot and poweroff commands work properly to execute a "clean" reboot or poweroff, or at least time-out in a reasonable amount of time (before battery dies in UPS hopefully).

 

The point is you should use /sbin/poweroff instead of 'powerdown' or '/usr/local/sbin/powerdown' in your 'at' job.

[Squid]

Regarding powerdown vs poweroff, straight from the horse's mouth:

 

Note that /usr/local/sbin/powerdown is a script that just invokes either /sbin/reboot or /sbin/poweroff and has been deprecated.

 

'powerdown' is a script formerly used by webGui/emhttp to initiate a gracefull power off or reboot.  The problem is that in it's original form, 'powerdown' relied on emhttp process to sequence the operation, but there were cases where this could not happen or proceeded very slowly.  Ultimately the system commands 'poweroff' or 'reboot' were finally invoked to complete the operation.

 

Anyway, the whole shutdown/poweroff/reboot operation was re-coded a couple releases ago so that now the "stock" linux reboot and poweroff commands work properly to execute a "clean" reboot or poweroff, or at least time-out in a reasonable amount of time (before battery dies in UPS hopefully).

 

The point is you should use /sbin/poweroff instead of 'powerdown' or '/usr/local/sbin/powerdown' in your 'at' job.

muscle memory... (and LT can deprecate it as much as they want.  truth be told is that the community is so used to issuing powerdown commands that they will always keep the script)  powerdown actually calls reboot / poweroff to do its thing

#!/bin/bash
logger "/usr/local/sbin/powerdown has been deprecated"
if [[ "$1" == "-r" ]]; then
  /sbin/reboot
else
  /sbin/poweroff
fi

[John_M]

Who looks at diagnostics? 

 

Someone other than me has also downloaded it. Hardly surprising, I suppose. This is a community of data hoarders. 

 

[John_M]

So how's it doing, JohnnyT? Has it shutdown, rebooted or is it still sitting there?

 

[JohnnyT]

Still just hanging out. I tried a reboot command and nothing.

[Squid]

Who looks at diagnostics? 

 

Someone other than me has also downloaded it. Hardly surprising, I suppose. This is a community of data hoarders. 

You're correct.. The syslog is a nightmare full of OOM's  And with all that going on who knows the side effects. 

 

If the powerdown (don't care LT -> I'll keep saying that til my dying day)doesn't work then a hard reset is basically the only recourse

[JohnnyT]

How did it get to OOM hmm..... I just logged in, said yeah update, then it downloaded the update so I started array shutdown and BOOM broken. Is this going to require a parity rebuild?

[John_M]

That's the big question. Some OOMs are fairly obvious when people try to do too much with too little RAM - and unRAID doesn't use swap space by default. But recent releases seem to be more prone than others. There's a theory about memory fragmentation but I don't know. I thought with 64-bit addressing it's all mapped and managed anyway.

 

Your syslog has been HUPed so the problems are already happening when it starts. It's impossible even to guess.

[JohnnyT]

Ok, I will hard boot when I get home.  :'( I will keep you posted.

[JohnnyT]

Instead of waiting until I got home since I had to hard boot it anyways I issued the command below at the cmd prompt and it forced an unclean reboot. The machine is back up and doing a parity check.

 

echo b > /proc/sysrq-trigger

 

http://www.linuxjournal.com/content/rebooting-magic-way