Looks like the mover wasn't running.  Next time try to capture the output of top.

Maybe shfs is having issues and that's what's causing the problem.

Huh, I would have thought that the diag info would have included output from top, for such a case. I'll be sure to add that to my post next time (not that I'm hoping there IS a next time...)

 

From your diag info it appears you only have 1GB of RAM.  It's possible the system is running out of memory and dying before it can log that it's dying.  Is it possible to increase your RAM?  Another alternative is to try the swap file plugin:

http://lime-technology.com/forum/index.php?topic=25816.0

I don't think it's a memory issue. The few times I was able to open the WebUI after everything else had locked up, the memory was sitting at 40%-50% capacity. I can probably manage to increase the RAM, but I'd rather not until we have firmer confirmation that it actually is a problem with lack of memory. I'll definitely give that plugin a shot though, thanks.

 

For what it's worth, there now seems to be several topics (spread across a few different forum subsections) that describe the same issue I'm having.

The other topics reported on this issue aren't the same as your necessarily. Some have reported the issue with the mover is when a container is running at the same time, for which we are continuing to investigate. I have not seen other topics that mention exactly your issue (where disks being spun down plus moved being run causes this problem).  I also attempted to recreate your issue today and was unsuccessful.

 

Lastly, 1GB of RAM is the absolute minimum requirement for the OS.  When you add plugins on top of that, you are increasing the amount of memory consumption.  Can you try booting into safe mode and report back if the issue persists?

Upon further investigation, it seems like the memory usage in the WebUI is not being reported correctly (or I'm misunderstanding what is being reported). The on the dashboard of the WebUI, it shows:

Memory usage: (42%)

Memory size: allocated 980 MB

installed 1024 MB (max. 4 GB)

 

while going in from telnet and running top shows:

Mem:  1003500k total,  898860k used,  104640k free,    42108k buffers

Swap:        0k total,        0k used,        0k free,  699020k cached

 

Output from the WebUI and top were both obtained at the same time, and was taken while the system was still operating normally. The CPU utilization reported by the WebUI seems to fairly closely match what top says. Until now, I've only been using what the WebUI was reporting, and thinking I had ~600MB of free memory. Assuming that top is actually accurate , it looks like there is only ~100MB of free memory,  so maybe it is a lack of memory problem after all. I've attached the rest of the output from top to his post, just in case.

 

Please let me know if I'm misunderstanding what is being reported the WebUI/top, or if there's something strange going on that's causing them to not match.

top.txt

Upon further investigation, it seems like the memory usage in the WebUI is not being reported correctly (or I'm misunderstanding what is being reported). The on the dashboard of the WebUI, it shows:

Memory usage: (42%)

Memory size: allocated 980 MB

installed 1024 MB (max. 4 GB)

 

while going in from telnet and running top shows:

Mem:  1003500k total,  898860k used,  104640k free,    42108k buffers

Swap:        0k total,        0k used,        0k free,  699020k cached

 

Output from the WebUI and top were both obtained at the same time, and was taken while the system was still operating normally. The CPU utilization reported by the WebUI seems to fairly closely match what top says. Until now, I've only been using what the WebUI was reporting, and thinking I had ~600MB of free memory. Assuming that top is actually accurate , it looks like there is only ~100MB of free memory,  so maybe it is a lack of memory problem after all. I've attached the rest of the output from top to his post, just in case.

 

Please let me know if I'm misunderstanding what is being reported the WebUI/top, or if there's something strange going on that's causing them to not match.

 

Memory usage reporting in linux is notoriously tricky.  With 1GB installed there's no way you have 600MB free.

Here is another one. I invoked the mover and before I did that, I shutdown the Plex Docker. Plex watch was running though. Shares were still working and web GUI for unraid was fine again. Still had to force shutdown the array unclean though. All my drives are spun down again, so i know the mover stopped running.

Had to add diag file separately due to upload size.

tower-diagnostics-20150919-1745.zip

Here is another one. I invoked the mover and before I did that, I shutdown the Plex Docker. Plex watch was running though. Shares were still working and web GUI for unraid was fine again. Still had to force shutdown the array unclean though. All my drives are spun down again, so i know the mover stopped running.

 

Would you be able to reproduce this in 'Safe Mode'?  Meaning, is anything you were doing depending on a plugin?  It would be extremely useful to be able to rule out plugins from causing this issue.

As requested, here is the diagnostics. Normally after invoking the mover, there's nothing after the "mover started" message. However, this time it looks like there actually is an error message. I should also note that this is the first time I've notice it occur after all the drives were spun up. I've also just noticed that a lot of files that I changed/moved/renamed/added before the crash were deleted or restored to their original state after I forced the power off and restarted

 

This system log shows a ReiserFS crash.  Please boot server in Maintenance mode and then click each Disk page and select 'Check Fileystem' for each disk.

 

Were you running any of the 6.0-beta's?  In particular there was a nasty ReiserFS bug introduced in kernel 3.6.2 used by unRaid v6.0-beta7/8:

http://lime-technology.com/forum/index.php?topic=35161.0

 

The nature of this bug was that corruption was not necessarily detected by "File system check" - instead it would just crash.

Here is another one. I invoked the mover and before I did that, I shutdown the Plex Docker. Plex watch was running though. Shares were still working and web GUI for unraid was fine again. Still had to force shutdown the array unclean though. All my drives are spun down again, so i know the mover stopped running.

 

Would you be able to reproduce this in 'Safe Mode'?  Meaning, is anything you were doing depending on a plugin?  It would be extremely useful to be able to rule out plugins from causing this issue.

I will do that within the next few hours. I'll let you know the results.

Memory usage reporting in linux is notoriously tricky.  With 1GB installed there's no way you have 600MB free.

That being the case, I went ahead an ran out to buy 4GB of RAM. I'm running memtest overnight (about 12-16hrs) to make sure they're seated properly and aren't duds. However, might I suggest, maybe you should display a warning on the WebUI next to the memory usage display that the memory usage may not actually be accurate, so people don't rely on it for diagnostic purposes like I did.

 

This system log shows a ReiserFS crash.  Please boot server in Maintenance mode and then click each Disk page and select 'Check Fileystem' for each disk.

 

Were you running any of the 6.0-beta's?  In particular there was a nasty ReiserFS bug introduced in kernel 3.6.2 used by unRaid v6.0-beta7/8:

http://lime-technology.com/forum/index.php?topic=35161.0

 

The nature of this bug was that corruption was not necessarily detected by "File system check" - instead it would just crash.

 

I don't believe I installed any of the beta's, but I could be mistaken. Usually it's my policy to not perform any unnecessary or risky upgrades on my Unraid system, since it would be (and is) very inconvenient when something breaks. First thing tomorrow morning I'll stop the memtest, boot into Maintenance mode, and run the file system check to be safe. Hypothetically, if I had installed the beta and this is the cause of my problems, is there anything else I'll have to do after I run the file-system check to prevent it from reoccurring (I'm already on 6.1.2)? Thanks again for all the help.

Memcheck came back clean after 12hrs and 6 passes, and Check File System found no errors as well (see attached log). I'm going to turn the cache drive back on for my shares and see how things go.

 

However, after re-reading limetech's response, it seems that if I HAD installed that bad beta, the file-system check may not be finding/fixing the problem. Is that correct? It's a PITA, but one of my drives is basically empty. I could start shuffling data around so I can reformat each drive. Is there a tool/script/command anyone can recommend to make moving data between drives easier? I'd also like to know which filesystem do most people recommend for the most stability, least maintenance, and/or easiest/reliable to fix when something DOES go wrong? From a quick search, I'd lean towards converting to XFS, but I'm not familiar enough with filesytems to make a informed decision.

reisercheck.txt

I am having this same or a very similar issue.

 

6.1.3

All reiserfs formatted drives.

No plex, no docker, no plugins, no vms, pretty vanilla actually except for unmenu which I don't use often (mainly for smarthistory)

No recent hardware changes.  I replaced parity AFTER I started having this issue but it was due to having some larger spares on hand and needing to upgrade parity before I can use the spares. (As needed)

 

Never installed a 6.x beta, started the 6.x chain with 6.0 or 6.1.1 something thereabouts.  But I came from the 5.x chain.  This is an old hardware build.

 

I am using a cache drive and the mover seems to be the source of the issues.

 

When the event happens, drive shares stop responding but the gui is still ok for me. 

The find command is using a ton of CPU

TONS of smbd processes hanging around

Some rsyncs

 

next time it happens I will do the diagnostics.

Right now I have the array in maint mode and I am checking the reiserfs filesystem on each drive.  There are 18 so it will take a while.  So far so good.

 

Just wanted to throw my hat in here.  I don't have logs yet because I just rebooted.

Well, shoot!

 

[pre]reiserfsck 3.6.24

 

Will read-only check consistency of the filesystem on /dev/md5

Will put log info to 'stdout'

###########

reiserfsck --check started at Thu Oct  8 18:29:12 2015

###########

Replaying journal:

Replaying journal: Done.

Reiserfs journal '/dev/md5' in blocks [18..8211]: 0 transactions replayed

Checking internal tree..  finished

Comparing bitmaps..Bad nodes were found, Semantic pass skipped

1 found corruptions can be fixed only when running with --rebuild-tree

###########

reiserfsck finished at Thu Oct  8 19:31:00 2015

###########

block 467509177: The level of the node (56396) is not correct, (1) expected

the problem in the internal node occured (467509177), whole subtree is skipped

vpf-10640: The on-disk and the correct bitmaps differs.[/pre]

 

So I guess I have to run rebuild tree?  Docs say that is bad. 

Can I do that from the web gui on 6.1.3 in maint mode?  I assume that is why the options field exists.

One more but my impression from the docs is that this one is minor.  ?

 

UPDATE: I did the repair on this drive and it worked fine.  Still cautious about the one that needs rebuild tree.  Would like some feedback on it before I do it.  Array is still in maint mode.

 

[pre]reiserfsck 3.6.24

 

Will read-only check consistency of the filesystem on /dev/md12

Will put log info to 'stdout'

###########

reiserfsck --check started at Fri Oct  9 02:07:11 2015

###########

Replaying journal:

Replaying journal: Done.

Reiserfs journal '/dev/md12' in blocks [18..8211]: 0 transactions replayed

Checking internal tree..  finished

Comparing bitmaps..Checking Semantic tree:

finished

2 found corruptions can be fixed when running with --fix-fixable

###########

reiserfsck finished at Fri Oct  9 03:26:59 2015

###########

vpf-10640: The on-disk and the correct bitmaps differs.[/pre]

Ok guys I am still having problems.  I was able to repair disk 5 and it SEEMS I didn't lose any files.  So all disk FS should be 100% ok.  Also all smart is 100% ok.

 

However samba is borked again and I cannot reach the shares.  Normally I have to POWER OFF because killing processes and trying to shutdown normally does nothing.  It just hangs.  I am going to leave it up this time but I really need help.

 

tower2-diagnostics-20151012-1727.zip

I'm curious hypyke, were you ever able to get this resolved?  I'm having similar issues... (Running disk tests now)

I am having this same or a very similar issue.

 

6.1.3

All reiserfs formatted drives.

No plex, no docker, no plugins, no vms, pretty vanilla actually except for unmenu which I don't use often (mainly for smarthistory)

No recent hardware changes.  I replaced parity AFTER I started having this issue but it was due to having some larger spares on hand and needing to upgrade parity before I can use the spares. (As needed)

 

Never installed a 6.x beta, started the 6.x chain with 6.0 or 6.1.1 something thereabouts.  But I came from the 5.x chain.  This is an old hardware build.

 

I am using a cache drive and the mover seems to be the source of the issues.

 

When the event happens, drive shares stop responding but the gui is still ok for me. 

The find command is using a ton of CPU

TONS of smbd processes hanging around

Some rsyncs

 

next time it happens I will do the diagnostics.

Right now I have the array in maint mode and I am checking the reiserfs filesystem on each drive.  There are 18 so it will take a while.  So far so good.

 

Just wanted to throw my hat in here.  I don't have logs yet because I just rebooted.

 

I also continue to have this problem (exact same symptoms as you describe, also no dockers, etc as I use unRAID purely for storage). I also went direct from v5 (5.0.5) to v6.1.3.

 

Previously, I disabled cache for a while and still experienced the issue while writing direct to shares. So I think it's more related to having 6.1.3, a large # of reiserfs drives in your array and writing to it rather than the mover itself. It feels a bit like a memory leak would (not saying its a memory leak...). Like there's a trigger and things start to spiral out of control until it all finally hangs. When it happens to me and I telnet in and run "uptime" the system load is well into three digits accompanied by huge numbers of SMBD processes and some find & rsync.

 

I decided to disable the mover over the holiday (~ 3 weeks), writing ONLY to the xfs cache drive during that time... leaving all other activity the same (watched plenty of movies via Plex, backed-up some files on unRAID to other systems, etc). I had no issues during this period.

 

Now I'm home again. I thought it may have been related to my SAS2LP-MV8 cards since they have had other issues, so I replaced those with m1015s the day I returned. I turn on the mover (nightly) and within 2 days my system load is up to 150, my shares are not accessible, I cannot power down gracefully and I must hard reboot. With one hang already, I'm confident that it's not specific to the HBA type.

 

Someone I PM'd with mentioned that these symptoms went away when they moved from rfs->xfs so that is my next step. I just ordered a 3TB Red. This will take 2-3 weeks to complete, so I'm really hoping to find a way around it.

 

If I end up with an all XFS array and this continues to happen, I'm not sure what I can do at that point.

 

I'd be curious to hear if someone has a 6.1.3 (or higher) 10+ disk system with all ReiserFS data disks and is NOT having this problem. @jonp - does LT have such a test system? Any thoughts on this issue? I've run reiserfsck and found no errors on my disks. System logs don't appear to reveal much either. Happy to grab them if you think they'll help but I doubt they're much different than what @hypyke posted.

 

 

My Parity and cache drives are plugged into the motherboard, I wonder if that makes a difference. I suppose I could plug ALL disks into my m1015s before going down the RFS -> XFS conversion path.

I'm curious hypyke, were you ever able to get this resolved?  I'm having similar issues... (Running disk tests now)

 

What are your system specs?