Hi All,

 

As the title suggests tower becomes unresponsive to all else other than access via telnet, even then reboot or powerdown commands does not work, leaving a hard reset to be the only option.

 

This started after I recently updated to 6.0 and started experimenting with docker. I have removed or uninstalled all the docker containers and apps but the problem still exists. From the log file I noticed 2 errors mainly (I'm not an expert on reading the linux logs), but these 2 had the error word in them and was repeated multiple times, so they stood out. The issue seems to occur after the mover is started and it gets hung trying to move some files.

 

Jul 11 03:43:43 Tower logger: rsync: write failed on "/mnt/user0/Other Media/Drive E - Users Folder/Downloads/cloneapp/CloneApp/Backup/Plex Media Server/Metadata/Movies/2/832559deeeede9c0e01e2ebdb02e0c1456f5fbe.bundle/Contents/_stored/art/com.plexapp.agents.localmedia_087e3fc04633581524755d42cf4d66ee2562c4a4": No space left on device (28)
Jul 11 03:43:43 Tower logger: rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0]

and

 

Jul 11 03:44:31 Tower logger: rsync: open "/mnt/user0/Other Media/Drive E - Users Folder/Downloads/cloneapp/CloneApp/Backup/Plex Media Server/Metadata/Movies/2/01d23aac16a3f46af7921fd57a44eeca75f2881.bundle/Contents/_combined/posters/com.plexapp.agents.localmedia_aa2482d036fe5152e328a0cb6fcebebeca32ed85" failed: No space left on device (28)
Jul 11 03:44:31 Tower logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]

 

The complete log file I copied is here and attached.

 

All help appreciated.

 

Thanks in advance, Abhi.

Logfile.zip

Just a quick comment, the built-in powerdown doesn't work if emhttp is hanging, waiting for a process to return.  Install the Powerdown plugin, which should always work.  See the Upgrading to UnRAID v6 guide, Other Plugins section.

Just a quick comment, the built-in powerdown doesn't work if emhttp is hanging, waiting for a process to return.  Install the Powerdown plugin, which should always work.  See the Upgrading to UnRAID v6 guide, Other Plugins section.

 

RobJ - Thanks for the link and the comment. Installed the updated version of the power down plugin and then a few others as well.

 

Any idea what those errors are caused by?

 

Edit: A couple of my disk drives are full or nearly full - could that be causing the issue seen, shouldn't the splitlevel and the high-water allocation method take care of that - or should go ahead and manually exclude those disks from the shares?

 

Jul 11 03:44:31 Tower shfs/user0: shfs_write: write: (28) No space left on device
Jul 11 03:44:31 Tower shfs/user0: shfs_write: write: (28) No space left on device

I'm probably not the best person to answer share based questions, as I have never used User Shares!

 

But I believe there's a setting for Minimum Free Space, that should keep a drive from being over-filled.  The advice has been to set it to be twice as large as the largest file you might save to the drive.  However, I know that the Reiser file system bogs down when too close to full, so it's best to keep 5% to 10% free on them, to avoid the possible problems when it tries to write files to an almost full disk.  You can see long delays then, which could appear as hangs.

Thanks RobJ.

 

Any help on this one. It is unresponsive now (again).

This appears to be a plex configuration problem. Post in the plex thread.

not to hijack the thread, but curious..

 

I have a QNAP TS-412, and when the free space got below a certain level, the SMB process would shoot cpu usage right up and make the NAS almost useless, does this happen with unRAID aswell?

This appears to be a plex configuration problem. Post in the plex thread.

 

I'm not an expert but it does not seem related the plex install. I had removed that and the file that the mover was stuck in was not anything related to a plex on unraid - it was a cloneapps (android app) backup folder in another one of my disk shares - which had backed up the android apps and their configs to a remote location.

 

Anyways, as per your reply I went ahead and did a complete fresh install  of unraid on the USB - from scratch (I had earlier upgraded via the plugin from 5 to 6) ad tried that out - but the same issue seems to be happening again especially when the mover starts running. Also I changed the allocation method from high water to most free (2 out of my 10 drives are almost full).

 

But it still freezes up, now the telnet session is also not connecting (host not found) and everything is down - smb, telnet, console, emhttp etc.

 

The syslog is attached - could someone please take a look and let me know what the issue is?

 

I have - 2 docker containers (mysql, and tonido) and a few plugins (mainly phaze's plugins - SAB, CP, SB, Marschino, PMS etc). The crashes were happening before the plugins were installed as well.

 

All help appreciated. Thanks.

 

syslog_07162015.zip

not to hijack the thread, but curious..

 

I have a QNAP TS-412, and when the free space got below a certain level, the SMB process would shoot cpu usage right up and make the NAS almost useless, does this happen with unRAID aswell?

 

This seems to be what is happening here too - my memory and cpu usage seems unusually high - but the I might be reading it all wrong - not an expert. But you got the almost useless part absolutely right - the tower is down.

 

I am waiting for someone who is more knowledgeable to take a look at the log and update me. Thanks for the reply.

not to hijack the thread, but curious..

 

I have a QNAP TS-412, and when the free space got below a certain level, the SMB process would shoot cpu usage right up and make the NAS almost useless, does this happen with unRAID aswell?

 

This seems to be what is happening here too - my memory and cpu usage seems unusually high - but the I might be reading it all wrong - not an expert. But you got the almost useless part absolutely right - the tower is down.

 

I am waiting for someone who is more knowledgeable to take a look at the log and update me. Thanks for the reply.

Boot in SAFE mode and if it continues to happen go to Tools - Diagnostics and post. Instead of a syslog, the more complete diagnostics using this method is preferred for v6.

 

You might also consider a memtest.

* Took a look at both syslogs, both are chopped off at the beginning, missing all of the boot and setup sections.  Hopefully the diagnostics will have a complete syslog, as we need to see that setup info.

 

* Both syslogs show file system corruption on the Cache drive, possibly an important factor in the troubles you're having.  I'd restart the array in Maintenance mode and Check Disk File systems on the flash drive, using its drive symbol of /dev/sdf1.  While there, because of all the restarts, I'd check the file systems on all of the drives (using /dev/md1 and up), just to be safe.  You may want to turn auto-start off, until system is fine again.  That wiki page needs to be updated for v6, reformatted to separate the current from old methods.

 

* One drive (on ata4.00) had a brief flurry of interface issues, but without setup info, cannot tell which drive it is.  No further problems after that, may or may not be significant.  Very unusual though, should not happen.  Occurred 3.6 minutes into parity check.  There were no good clues as to what the actual issue was.

 

* The second syslog shows you were getting OOM-killed, badly, effectively shutting you down.  Something (probably tonidoconsole) was using up all of your memory.

 

* About 2 hours after starting, the first OOM-kill call was made, and thereafter every 30 to 60 minutes, it was called again.  Each time, tonidoconsole was killed, but each time it appeared to restart with a new PID.  I'm no expert here, and didn't take the time to research the number formats, but it appears it was using over 2GB of RAM each time it was killed.  That would appear to indicate a very bad memory leak, or you are running too much for the RAM that's present.

not to hijack the thread, but curious..

 

I have a QNAP TS-412, and when the free space got below a certain level, the SMB process would shoot cpu usage right up and make the NAS almost useless, does this happen with unRAID aswell?

 

This seems to be what is happening here too - my memory and cpu usage seems unusually high - but the I might be reading it all wrong - not an expert. But you got the almost useless part absolutely right - the tower is down.

 

I am waiting for someone who is more knowledgeable to take a look at the log and update me. Thanks for the reply.

Boot in SAFE mode and if it continues to happen go to Tools - Diagnostics and post. Instead of a syslog, the more complete diagnostics using this method is preferred for v6.

 

You might also consider a memtest.

 

Thank you. attached is the diagnostics report (run while in maintenance mode - hope that is okay)!

 

I'm running the tower headless - will do the memtest as soon as I can move it and plug in a monitor and keyboard, and report.

tower-diagnostics-20150716-2214.zip

* Took a look at both syslogs, both are chopped off at the beginning, missing all of the boot and setup sections.  Hopefully the diagnostics will have a complete syslog, as we need to see that setup info.

 

Thanks again RobJ. I left the log window up and running on the browser, and was copying and pasting the syslog from there - hence the initial cutoff.  Since the system was becoming unresponsive and inaccessible (even via telnet) I did not know of another way to get to the syslog.

 

* Both syslogs show file system corruption on the Cache drive, possibly an important factor in the troubles you're having.  I'd restart the array in Maintenance mode and Check Disk File systems on the flash drive, using its drive symbol of /dev/sdf1.  While there, because of all the restarts, I'd check the file systems on all of the drives (using /dev/md1 and up), just to be safe.  You may want to turn auto-start off, until system is fine again.  That wiki page needs to be updated for v6, reformatted to separate the current from old methods.

 

Ran on the cache drive and disk 1 (sdf1, md1) - no corruptions found on either.

- Disk 1

- Cache

Running on md2 now - this drive did show some fails on the log and the SMART tests, so I'm not sure if there is anything wrong with it or is it just age. Will report once done. Auto start has been off always, I need to start the array after booting up.

 

* One drive (on ata4.00) had a brief flurry of interface issues, but without setup info, cannot tell which drive it is.  No further problems after that, may or may not be significant.  Very unusual though, should not happen.  Occurred 3.6 minutes into parity check.  There were no good clues as to what the actual issue was.

 

I posted the diagnostics report above - maybe that would be helpful, to see what's going on.

 

 

* The second syslog shows you were getting OOM-killed, badly, effectively shutting you down.  Something (probably tonidoconsole) was using up all of your memory.

 

I noticed this as well. This was installed in a docker container - I will remove the tonido container. However this was a recent addition (after the fresh clean install 2 days back), wasn't there before this, that is why you don't see it in the first log.

 

* About 2 hours after starting, the first OOM-kill call was made, and thereafter every 30 to 60 minutes, it was called again.  Each time, tonidoconsole was killed, but each time it appeared to restart with a new PID.  I'm no expert here, and didn't take the time to research the number formats, but it appears it was using over 2GB of RAM each time it was killed.  That would appear to indicate a very bad memory leak, or you are running too much for the RAM that's present.

 

Will run the memtest soon and see if there is a problem with the memory.

 

Thanks for all the help so far. Hope we can get tot he bottom of this.

 

* Usable RAM is 3.4GB, meaning .6GB (probably 640MB) reserved for graphics, which unRAID does not use.  Go into BIOS and knock the video RAM reservation down to its lowest possible value.  Even at its lowest, memory is wasted, because unRAID only works with a text console.  This should hopefully give you back at least 512MB more of RAM to use.

 

* You may want to consider increasing your RAM to 8GB.  Unless it's a memory hogging or memory leaking bug that gets fixed, you need more than you currently have.  Or uninstall the memory hog!

 

* Cache drive looks clean!  No more corruption evident.

 

* The drive on ata4.00 is the SSD.  The interface to it seems a little flaky.  It reported an "interface fatal error", then reads and writes failed until a hard reset, fine ever since.  You might check for a firmware update for it.

 

* Odd CPU 'Call Trace' as the array was stopping, popped up right after the first data drive was stopped.  This looks very much like a little bug that was happening in a late v6 beta or early v6 rc, but fixed by a later version.  Don't know what to say about it.  It doesn't seem harmful, but is not correct behavior.

* Usable RAM is 3.4GB, meaning .6GB (probably 640MB) reserved for graphics, which unRAID does not use.  Go into BIOS and knock the video RAM reservation down to its lowest possible value.  Even at its lowest, memory is wasted, because unRAID only works with a text console.  This should hopefully give you back at least 512MB more of RAM to use.

 

* You may want to consider increasing your RAM to 8GB.  Unless it's a memory hogging or memory leaking bug that gets fixed, you need more than you currently have.  Or uninstall the memory hog!

 

* Cache drive looks clean!  No more corruption evident.

 

* The drive on ata4.00 is the SSD.  The interface to it seems a little flaky.  It reported an "interface fatal error", then reads and writes failed until a hard reset, fine ever since.  You might check for a firmware update for it.

 

* Odd CPU 'Call Trace' as the array was stopping, popped up right after the first data drive was stopped.  This looks very much like a little bug that was happening in a late v6 beta or early v6 rc, but fixed by a later version.  Don't know what to say about it.  It doesn't seem harmful, but is not correct behavior.

 

I would be happy to increase the RAm - did not do it till now because I thought it did not make a difference beyond 4GB for unRaid. Is that not true anymore (after v6)?

I would be happy to increase the RAm - did not do it till now because I thought it did not make a difference beyond 4GB for unRaid. Is that not true anymore (after v6)?

 

That was only true in 32 bit v5 and earlier.  There's no such restriction in 64 bit v6!  It will use everything you give it.