I woke up this morning to find my unRAID shares unresponsive. Neither telnet or the web interface would work.

 

So I plugged in my monitor and keyboard and was confronted with a screen full of info in the console ending in a line of [end trace]

 

Now I assume the OS itself has crashed and that was the end of the stack trace (I couldn't scroll up so couldn't see the start)? I had to hard poweroff and reboot. The problem is that I've no idea what happened, and I assume the syslogs etc will have been lost in the reboot, thus leaving me with no way to diagnose and prevent it happening again. The array started up again fine after the reboot and when I left for work was running a parity check.

 

Any thoughts on what I can do next? This is my first unRAID install (4.4.2) and it's only been running a couple of days. I've got unMenu installed and a few of the downloadable packages that that allows. The crash has made me nervous

 

Thanks in advance,

 

Justin

If i were you i would:

 

Install beta 4

Uninstall all user addons

 

aka test stability with the official release first free of any other factors.

 

Tis will simplify support

Ok that makes sense thanks.

 

Perhaps I'm going too fast trying out user mods etc until I'm happy that everything is working ok.

 

I avoided installing the beta version though as I assumed that would be more risky than 4.4.2. Is the beta release actually regarded to be more stable then?

 

Thanks!

Ok that makes sense thanks.

 

Perhaps I'm going too fast trying out user mods etc until I'm happy that everything is working ok.

 

I avoided installing the beta version though as I assumed that would be more risky than 4.4.2. Is the beta release actually regarded to be more stable then?

 

Thanks!

Unlike many other systems, your assumption about release numbering was flawed.  The most stable is probably 4.5--beta4, believe it or not.

The beta-releases are just as often more bug-fixes than new features.

 

However... a hard-crash of the type you are describing are almost always related to a problem with memory.   Your very first step, especially since yours is a new server, is to go into your BIOS and check the memory voltage, timings, and multiplier (speed) settings.  I know many BIOS have an AUTO setting, but many get it wrong.   See this thread for an example.

Then... run the memory test, for several cycles...  If you have memory errors, then nothing else can work properly.

 

The other way to crash is to run out of memory (a process allocated more memory that you have)  An add-on could do this...  example, you create a huge database and allocate cursors, but do not free them... eventually you will crash.  Usually the server will terminate processes trying to free up memory and you will lose the ability to connect to the server... this does not usually give the same symptoms you described with a kernel exception and a resulting stack trace.

 

If you can take a screen-shot with a camera of the crash it will sometimes help... Otherwise, first thing... get your hardware stable.

 

Finally,  To upgrade you only need to replace the bzroot and bzimage files from the newest distribution. (and 4.5-beta5 is due out shortly.  I'd let some of us with stable systems use it first, but once it is out for a week or so, it too will probably be a good upgrade choice.)

 

Joe L.

Many thanks for the detailed reply Joe!

 

The hardware is 2nd hand (from a mate) as I wanted to build the server on a budget, so it's more that possible that either it's dodgy or I've not set the BIOS up correctly.

 

Tonight I'm going to install the Beta4 as suggested, remove unMenu and anything else I installed from that and then run a few memtests.

 

How do I know what the timings for the RAM should be? Will it be on a label on the stick?

I'm running 2x512mb of DDR2. Is that enough?

 

Disks are 3x 1TB Hitachis and unRAID is installed on a Sandisk Micro Cruzer (which I believe is recommended).

 

I'd recommend that, before upgrading unRAID or removing unMENU, that you run the memory test.  If that runs successfully overnight, then you should pursue these other options.

 

While I agree that 4.5B4 is stable, so was 4.4.2.  The only fix that I am aware of that might impact this issue is the so called "log explotion" bug.  The syslog is actually kept in memory.  If it gets too big it can cause Linux to start killing processes and lead to an unresponsive system or crash.  But the syslog does not grow unless something else is wrong.  The most common cause related to errors from the "mover" script.  (The mover script is invoked every night to move things off of the cache disk and onto the array.)  If you don't have a cache disk, this is not the cause.

 

But with each new release of unRAID, the latest and greatest release of Linux is included.  Bugs are fixed (and new ones created too  ) with each release.  Trying the latest release is certainly worthwhile.

 

Might want to check out this post concerning beta vs release versions of unRAID.

 

 

Tonight I'm going to install the Beta4 as suggested, remove unMenu and anything else I installed from that and then run a few memtests.

I agree with bjp999, perform the memory test first.  Since that is run on its own, before loading any OS, you do not need to remove anything from the flash drive at all to run it with no interaction from anything you might have added.

How do I know what the timings for the RAM should be? Will it be on a label on the stick?

The timings are not likely to be on the label, but the model number will be, and with it you can look up the manufacturer's recommended timings, voltage, and clock speed.  Then you can verify the settings in the BIOS.  Beware though, many older motherboards have no way to set memory voltage, so in their slots you must use memory that takes the voltage it supplies.

I'm running 2x512mb of DDR2. Is that enough?

I've only got 512MB of ram in my server.  You should be fine. (Unless your system log grows to use it all up, but that is as a result of other issues.)

Disks are 3x 1TB Hitachis and unRAID is installed on a Sandisk Micro Cruzer (which I believe is recommended).

The specific disks and flash drive do not matter too much... (as long as the server boots, the flash drive is fine)

 

If several passes of memory test comes up clean, then post a copy of your system log.  It might provide the clues needed to figure out what is happening.

 

Joe L.

Ok will do. Should I expect memtest to take hours?

 

The ram and mobo were used before by my friend without any issues, but that's not to say it's not the memory so definitely best to check that out first.

Ok will do. Should I expect memtest to take hours?

 

The ram and mobo were used before by my friend without any issues, but that's not to say it's not the memory so definitely best to check that out first.

 

Memtest will run for as long as you let it.  Length of one cycle varies with amount of memory and system speed.  I'd expect one cycle to take 10-20 minutes on average.  YMMV.

 

I'd recommend an overnight test.

Ok I found some better quality ram (Corsair PRO XMS stuff that were awesome DDR at the time) and have memtest on that with the correct timings entered into the BIOS (2-2-2-5). No errors in 5 runs (about 1hr20min).

 

So now I've removed all my unMenu stuff and installed Beta4. Attached is my syslog from the first boot of that. If someone wouldn't mind taking a look that would be fantastic 

oh also when refreshing the main web admin screen I get an errror in Safari that http://tower/style.css failed to load:

 

Safari can’t open the page “http://tower/style.css” because the server unexpectedly dropped the connection. This sometimes occurs when the server is busy. Wait for a few minutes, and then try again.

 

No idea if this is a known issue or not, just thought I'd mention it.

You are the first to mention it, the other browsers ignore the fact they cannot load the style sheet. (I don't think Tom ever provided one)

 

He did mention it in the release notes for 4.5-beta2 though...

System Management Utility will now use a CSS style sheet file on the Flash (config/style.css) if present.

 

Not tested, but you can probably silence the error by creating an empty file there like this:

touch /boot/config/style.css

I'm not sure that would work. Surely it's looking for style.css in the same place it gets main.html etc from? I'll have a play.

 

*** update - that actually did fix it, but I'm not sure how ***

 

Don't suppose you had a quick scan of my syslog did you Joe?

I'm not sure that would work. Surely it's looking for style.css in the same place it gets main.html etc from? I'll have a play.

 

*** update - that actually did fix it, but I'm not sure how ***

 

Don't suppose you had a quick scan of my syslog did you Joe?

Since it specifically said in the release notes that it would use one from config/style.css, I figured that would be where it is looking.

All the "html" is internal to /usr/local/sbin/emhttp.  It is also the web-server, so it can re-route any request for a file to wherever it likes, including providing the normal management pages from itself.

 

The browser is simply putting in a request for the file, so its error message is all it knows. (It thinks the style sheet is at the web-root)

 

In any case, your syslog looked pretty good to me.  No glaring errors.

 

Now for the tough questions...

What were you running as add-ons previously?

 

I know you said you enabled some through unMENU, but specifically which ones?  And which others (torrent feeds?, etc?)

 

Joe L.

Ahh I must have missed that bit in the release notes. Apologies for doubting you Joe 

 

As far as unMenu goes, I believe I only downloaded and enabled 2 packages:

 

lsof, so I could use the option to show open files

cxxlibs, as I think I needed that for temp display in the unmenu interface

 

unMENU was the only user mod I added, so no torrent client or anything like that.

Ahh I must have missed that bit in the release notes. Apologies for doubting you Joe 

 

As far as unMenu goes, I believe I only downloaded and enabled 2 packages:

 

lsof, so I could use the option to show open files

cxxlibs, as I think I needed that for temp display in the unmenu interface

 

unMENU was the only user mod I added, so no torrent client or anything like that.

Good news is, neither of those would cause you to run out of memory... and I'll bet most people have been running them. Also, the cxxlibs is in the new release, as is lsof.  So you don't need to load them going forward once you upgrade to 4.5-beta4 (or 4.5-beta5, apparently due out any minute now today, if Tom keeps to his predicted release time)

 

Joe L.

I think the only other thing I did was use the User Scripts section of unMENU to change the read ahead to 2048 (I think standard is 1024).

 

Not sure if I want/need to do that but I'm sure I read performance was better with the higher number. Don't know how to set it outside of unMENU and it didn't seem to remember it after a reboot.

 

The other thing I've seen mentioned are some changes to /etc/hosts and nameservers that are supposed to improve perfomance? I might be mistaken.

 

My write speeds do seem slower tonight. Last night I was peaking writes across gigabit of ~22MB/s. Tonight seems slower, like ~12MB/s most of the time. No idea why.

Just a note to say that since going to Beta4 and changing the RAM for my better set I've had no issues whatsoever....

 

I'm going to leave it like this for a bit before trying to change anything.

 

Thanks for the help!

 

Justin

Ahh I must have missed that bit in the release notes. Apologies for doubting you Joe 

 

As far as unMenu goes, I believe I only downloaded and enabled 2 packages:

 

lsof, so I could use the option to show open files

cxxlibs, as I think I needed that for temp display in the unmenu interface

 

unMENU was the only user mod I added, so no torrent client or anything like that.

 

Both lsof and cxxlibs are built-in to the 4.5-beta series.  Also a fix for non-present style.css will be in next beta.

Thanks! I've just spotted that there's a beta6 out so I'm going to upgrade to that this evening...