Running 6.1.2, memory utilization at about 40%-50%, parity and SMART checks all come back clean, only plugins are powerdown and community applications. Docker is active with the img on a cache drive, but no docker apps actually installed/running.

 

If I call the mover from the web GUI while all the drives are spun up, everything works fine. However, if I call the mover when the drivers are spun down (or when it's called automatically at 3AM), the disks dont get spun up, and the mover runs endlessly in the background doing nothing, even after I spin up the disks. With the mover running in the background, I'm unable to hit the mover button again with the disks up, and it prevents me from stopping the array from the web gui, so I can't reboot the system. I telnet'd into the server and called powerdown, which proceeded to do nothing after a hour. I also tried to kill all the rsync and mover processing running the background with kill SIGTERM and kill SIGKILL, but a few of them refused to stop. Ultimately, I had to long-press the power button to force the power off. After I upgraded to v6, I've had the webgui and power button stop responding several times. I read that a few people were having similar issues, but supposedly they were fixed in 6.1.2. Is anyone else still having an issue like I am on 6.1.2? I'm not too keen on having to kill the power on my server anymore.

Next time it happens, go to Tools - Diagnostics and post the zip. If you can't get to the webui, you can also type "diagnostics" at the command line and it will create the diagnostics zip file on your flash drive.

As requested, here is the diagnostics. Normally after invoking the mover, there's nothing after the "mover started" message. However, this time it looks like there actually is an error message. I should also note that this is the first time I've notice it occur after all the drives were spun up. I've also just noticed that a lot of files that I changed/moved/renamed/added before the crash were deleted or restored to their original state after I forced the power off and restarted

tower-diagnostics-20150913-1228.zip

Have you run memtest recently?

Yeah, having gotten that error gave me something to search for, and a lot of what I found said to run memtest, so I started running it immediately. 2hrs and 5 passes later and there are no errors, but I'll keep running it for about another 20hrs, unless someones offers another idea of what the problem/solution might be.

Have you had any hardware changes or moves recently?

Nope. Closest thing to a hardware change I've had in the last year is the UPS battery dying and getting replaced, but I've been having this problem both before/after that. Last time I replaced a drive was probably also a year or two ago,. The server itself hasn't moved an inch or had any component changes since the day I got it years ago.

Small update: no memory errors overnight. I'm disabling the cache drive on all my user shares and seeing if that at least avoids the problem for now.

I am having an issue with this as well today. I noticed my plex was no longer working and the plex server (running as a docker by needo) was showing as down by all my devices. All the other dockers are running fine it looks like. I can access the shares on windows PCs AND the WEBGUI for unraid works fine as I move between the pages. I can telnet in as well. I cannot shut down the array, because the stop option is greyed out and it says "mover is running" next to the INVOKE button on the main page. Since my mover runs at 630 in the morning, I think its safe to say that its been running since this morning.... Thats about 12 hours ago. I have posted the diagnostics here, and I am wanting to know how I can kill the mover if possible so I can cleanly stop the array, and reboot it.

 

*EDIT*

Also, I should mention that all my drives are spun down when i got home from work today as well, so it would seem that nothing is being moved to the array. I spun them up but saw none of the numbers changing in a way that would tell me stuff was being moved around.

tower-diagnostics-20150915-1858.zip

Also I found this screenshot interesting. Something that I am unsure what it is, has one of my cores maxed out and has been holding it that way for over 12 hours. Maybe this is the mover? It has been about that long since it would have ran. Can/should I kill this in HTOP? Also, I just tried to stop the Plex docker and now my WEBGUI in unraid is toast. Just spins now like its still trying to close that docker but is failing to do so.

I've been examining the syslogs and ps reports of 3 different users (2 in this thread and one elsewhere), each with what appears to be the same problem - the Mover is stuck and a CPU is maxed, possibly other damage too.  Here are the relevant syslog parts, notice the similarities (show names abbreviated) -

scheduled, logging off, 100% CPU, plex tool running

root    31866  1648  0 06:30 ?        00:00:00 /bin/sh -c /usr/local/sbin/mover &> /dev/null

root    31867 31866  0 06:30 ?        00:00:00 /bin/bash /usr/local/sbin/mover

root    31870 31867  0 06:30 ?        00:00:00 find ./Shows/ -depth ( ( -type f ! -exec fuser -s {} ; ) -o ( -type d -empty ) ) -print ( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ ; -delete ) -o ( -type f -exec rm -f /mnt/user0/{} ; )

root    32199    2  0 06:30 ?        00:00:00 [kworker/3:0]

root    32343 31870  0 06:30 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./Shows/P...s - S04E20 - Free Fall HDTV-720p.mkv /mnt/user0/

root    32344 32343  0 06:30 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./Shows/P...s - S04E20 - Free Fall HDTV-720p.mkv /mnt/user0/

root    32347 32344  0 06:30 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./Shows/P...s - S04E20 - Free Fall HDTV-720p.mkv /mnt/user0/

scheduled, logging on, 100% CPU, plex tool running

root    32375  1455  0 01:30 ?        00:00:00 /bin/sh -c /usr/local/sbin/mover |& logger

root    32378 32375  0 01:30 ?        00:00:00 /bin/bash /usr/local/sbin/mover

root    32379 32375  0 01:30 ?        00:00:00 logger

root    32383 32378  0 01:30 ?        00:00:00 find ./TV/ -depth ( ( -type f ! -exec fuser -s {} ; ) -o ( -type d -empty ) ) -print ( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ ; -delete ) -o ( -type f -exec rm -f /mnt/user0/{} ; )

root    32613 32383  0 01:30 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./TV/L...r.mp4 /mnt/user0/

root    32614 32613  0 01:30 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./TV/L...r.mp4 /mnt/user0/

root    32615 32614  0 01:30 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./TV/L...r.mp4 /mnt/user0/

manual, logging on, very high CPU, simultaneous GPF, no plex tool running

root    25953    1  0 12:26 ?        00:00:00 /bin/bash /usr/local/sbin/mover

root    25954    1  0 12:26 ?        00:00:00 logger

root    25956 25953  0 12:26 ?        00:00:00 find ./PLEX_TV/ -depth ( ( -type f ! -exec fuser -s {} ; ) -o ( -type d -empty ) ) -print ( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ ; -delete ) -o ( -type f -exec rm -f /mnt/user0/{} ; )

root    25958 25956  0 12:26 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./PLEX_TV/A...D.mkv /mnt/user0/

root    25959 25958  0 12:26 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./PLEX_TV/A...D.mkv /mnt/user0/

root    25960 25959  0 12:26 ?        00:00:00 rsync -i -dIWRpEAXogt --numeric-ids --inplace ./PLEX_TV/A...D.mkv /mnt/user0/

 

Two are logged, one is not.  Two are scheduled, one is manual.  Two have screen shots of htop with the same shfs process associated with /mnt/user0 running at 100%, the other shows in their ps that that same line has the highest CPU time, so probably very high CPU.  This one also has an immediate GPF (general protection fault), the other 2 do not, but aren't functional, most likely due to runaway threads.  The Mover appears to be spinning its wheels, with no further copying, and all drives spin down.

 

The find and rsync commands and parameters used are beyond my expertise, so I'm really hoping some more experienced rsync and bash users will examine the situations above.  There appears to me to be a conflict here, some how between rsync and Plex?  One user has plexpy running, another has plex and plexwatch running, and the third, while no Plex tool is running, has the files in PLEX_TV, so Plex is involved some how.  Three different situations concern me, (1) is there an incompatibility between how Plex handles the metadata and how rsync does, and (2) is a Plex tool opening or changing files during rsync handling of the same, and (3) are files still being downloaded/added by other tools and then grabbed by rsync before they are finished/closed.

 

Questions -

* How does rsync handle a file that's open and growing?

* How does rsync handle a file that's being changed, or its metadata is being changed?

* Is there a bug in rsync?

* Is there a bug in FUSE?

 

To avoid these potential conflicts, I can think of a few alternatives -

* Don't use the Mover, turn off caching, and download direct to array drives

* Use modified Mover with different parameters

* Use modified Mover with replacement copy program for rsync (e.g. cp then rm)

* Don't use Plex?

* Other options?

Rob, you sound way smarter than I... Thank you very much for taking the time,  as I know how valuable it can be. Valuable points you have brought up,  and I'm hoping since this forum is a very active one,  those points you brought up will be talked about. I'm new to this,  but I'm willing to try my best to get the most information out when these crashes take place so that others don't see the same problems.

By the way,  I tried performing a clean shutdown with the powerdown plug-in but it failed. I was forced to perform a hard restart and my array is performing a parity check now. My mover will try to run again at 630 am central, but I'll disable that for tonight until a few more people post their thoughts. Thank you.

Check plex settings for thumbnail extraction and other background activities.

Check plex settings for thumbnail extraction and other background activities.

 

For another data point, I am one of the people having this same issue and I do not run Plex on my unraid.  All seems well when I turn off my mover script.

RobJ,

 

I have also been having issues with Unraid becoming unresponsive and CPU going to 100%.  Last night I decided to upgrade my CPU from a celeron E1200 to a Q9300 to see if it made a difference.  Prior to shutting down unraid, I telneted in to the console to see if my system had started spawning off SMBD processes as outlined in my other thread https://lime-technology.com/forum/index.php?topic=42900.0 ... when I did this, nothing was out of the ordinary.  I then hit stop the array and it got to the sync stage and my system locked up.  I was still able to telnet in, attached is my syslog. Also here is a dropbox link to a screen shot showing that sync is running with 100% of the CPU.  https://www.dropbox.com/home/Log_Website?preview=htopsynccpu9_15_15.jpg

 

I let unraid sit in this hung state with sync running for an hour, then I manually powered off, changed out the cpu, restarted unraid and the parity check began (which has now completed sucessfully).  Not sure if this is related, but thought that I should share it.

 

Dan

 

 

 

 

unraidshutdownfornewcpu9_15_15.txt

I manually ran the mover in the last few minutes and moved about 30GB to various drives in my array. Everything seems to have worked fine this time. Just adding this bit of information. I can post logs if that will help anybody. Should I turn on mover logging until this issue is resolved and people stop saying they have this problem?

 

The find and rsync commands and parameters used are beyond my expertise, so I'm really hoping some more experienced rsync and bash users will examine the situations above.  There appears to me to be a conflict here, some how between rsync and Plex?  One user has plexpy running, another has plex and plexwatch running, and the third, while no Plex tool is running, has the files in PLEX_TV, so Plex is involved some how.  Three different situations concern me, (1) is there an incompatibility between how Plex handles the metadata and how rsync does, and (2) is a Plex tool opening or changing files during rsync handling of the same, and (3) are files still being downloaded/added by other tools and then grabbed by rsync before they are finished/closed.

 

My understanding is that Plex does not store any Metadata with the video files themselves. Metadata should be in the Plex/Library/ directory that's created by the install not the folders video files you want to playback with plex are stored in.

 

That said, Plex has background processes (that I disabled) that can fetch metadata at specific intervals. Also my backup process using rysnc to backup the Plex/Library/ directory to my array did have issues when I was writing to a share instead of a disk. Writing to a specific disk fixed this issue. Not sure what was causing it or what the problem was, but it was very odd to me.

 

So it's possible these people have either intentionally or unintentionally configured there Plex Install to a location that Mover is attempting to move, causing a stall, or it might be due to a background process having the file open.

 

Personally I think it's likely the --inplace flag that's breaking things, as I have heard that --inplace can cause a whole host of problems if it's used on files that are being modified.

 

"--inplace This option changes how rsync transfers a file when the file's data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file. This has several effects: (1) in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash), (2) the file's data will be in an inconsistent state during the transfer, (3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails, (4) a file that does not have write permissions can not be updated, and (5) the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file (one exception to this is if you combine this option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer). WARNING: you should not use this option to update files that are being accessed by others, so be careful when choosing to use this for a copy. This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound. The option implies --partial (since an interrupted transfer does not delete the file), but conflicts with --partial-dir and --delay-updates. Prior to rsync 2.6.4 --inplace was also incompatible with --compare-dest and --link-dest. " -Rsync Man page

One user has plexpy running, another has plex and plexwatch running, and the third, while no Plex tool is running, has the files in PLEX_TV, so Plex is involved some how.

I'm the user with the files in PLEX_TV. If it helps troubleshoot the problem, I have Plex and Sonarr running on a separate PC, which has the PLEX_TV share mounted to a drive. (I know that's not a recommended setup, but it has served me well for years without issue until now).

 

So it's possible these people have either intentionally or unintentionally configured there Plex Install to a location that Mover is attempting to move, causing a stall, or it might be due to a background process having the file open.

See above, Plex is not even installed on the same PC.

I've been examining the syslogs and ps reports of 3 different users...

...

 

 

Questions -

* How does rsync handle a file that's open and growing?

* How does rsync handle a file that's being changed, or its metadata is being changed?

* Is there a bug in rsync?

* Is there a bug in FUSE?

 

To avoid these potential conflicts, I can think of a few alternatives -

* Don't use the Mover, turn off caching, and download direct to array drives

* Use modified Mover with different parameters

* Use modified Mover with replacement copy program for rsync (e.g. cp then rm)

* Don't use Plex?

* Other options?

 

From my quick review, the htop image below reveals the user share/fuse layer at 100%.

Not sure if it's a bug or there's an open file or some other tool keeping it busy (cache dirs?)

 

Perhaps turn off plex completely or boot in safe mode, then execute the mover.

 

the -exec fuser {} segment in the find is suppose to skip over open files.

 

Has anything thought to look at the file being copied to see if it is growing or what the size is in comparison to the original.

 

So it's possible these people have either intentionally or unintentionally configured there Plex Install to a location that Mover is attempting to move, causing a stall, or it might be due to a background process having the file open.

See above, Plex is not even installed on the same PC.

 

Well that addresses the misconfigured question, but I think it still could be due to Plex's secheduled services having files open while it is scanning for new files or downloading new metadata. I turned all the secheduled services off because I didn't want my disks to spin up frequently when I am not actively using my system.

Another update, with logs attached. Cache drive is no longer being used by any of the shares, but system still locked up. Cant login to WebUI or access files over the network. I could telnet in to run and get the diagnostics, but the powerdown script still hangs and does nothing. Sonarr and Plex did a few things throughout the day, so as suggested, it could be one of those. On the other hand, they also both did some work yesterday as well, and at night everything was still working. It seems like it's still ultimately a problem on the unraid side, since it never used to do this.

tower-diagnostics-20150917-1800.zip

Another update, with logs attached. Cache drive is no longer being used by any of the shares, but system still locked up. Cant login to WebUI or access files over the network. I could telnet in to run and get the diagnostics, but the powerdown script still hangs and does nothing. Sonarr and Plex did a few things throughout the day, so as suggested, it could be one of those. On the other hand, they also both did some work yesterday as well, and at night everything was still working. It seems like it's still ultimately a problem on the unraid side, since it never used to do this.

 

 

Looks like the mover wasn't running.  Next time try to capture the output of top.

Maybe shfs is having issues and that's what's causing the problem.

Another update, with logs attached. Cache drive is no longer being used by any of the shares, but system still locked up. Cant login to WebUI or access files over the network. I could telnet in to run and get the diagnostics, but the powerdown script still hangs and does nothing. Sonarr and Plex did a few things throughout the day, so as suggested, it could be one of those. On the other hand, they also both did some work yesterday as well, and at night everything was still working. It seems like it's still ultimately a problem on the unraid side, since it never used to do this.

 

From your diag info it appears you only have 1GB of RAM.  It's possible the system is running out of memory and dying before it can log that it's dying.  Is it possible to increase your RAM?  Another alternative is to try the swap file plugin:

http://lime-technology.com/forum/index.php?topic=25816.0

Looks like the mover wasn't running.  Next time try to capture the output of top.

Maybe shfs is having issues and that's what's causing the problem.

Huh, I would have thought that the diag info would have included output from top, for such a case. I'll be sure to add that to my post next time (not that I'm hoping there IS a next time...)

 

From your diag info it appears you only have 1GB of RAM.  It's possible the system is running out of memory and dying before it can log that it's dying.  Is it possible to increase your RAM?  Another alternative is to try the swap file plugin:

http://lime-technology.com/forum/index.php?topic=25816.0

I don't think it's a memory issue. The few times I was able to open the WebUI after everything else had locked up, the memory was sitting at 40%-50% capacity. I can probably manage to increase the RAM, but I'd rather not until we have firmer confirmation that it actually is a problem with lack of memory. I'll definitely give that plugin a shot though, thanks.

 

For what it's worth, there now seems to be several topics (spread across a few different forum subsections) that describe the same issue I'm having.