I've had the trial version of the unRAID server (version 4.3-beta6) running for several weeks without any problems. I only have a 3 disk system (3 x 1TB drives) and wanted to test it out for a little while. 

 

Everything has been running great. I transferred my media from another JBOD "server" and have been ripping both DVDs and CDs that I can finally put online because I finally have more space. I also just purchased another 1 TB drive and was planning to upgrade to the plus version so I could increase my storage space.

 

However, this morning, something strange happened. I was working on my office computer and listening to some music. I have J.River Media Center 12 running on one of the other machines on my network. JRMC12 is also set to run in server mode, so other machines can use a single media library. The office machine was using that other machine's library. All of the music is stored on the unRAID server. Anyhow, when I was listening, I noticed that there was some unusual distortion. It sounded like weird distortion now and then. Not your normal dropouts and skips, but more like shrieks and scratches. So I tried playing the same song on different machines. And it had the same distortion. Then I tried playing the same song from the original CD on my computer. The CD played flawlessly, so I tried re-ripping it. While ripping, JRMC12 crashed. I restarted JRMC12 and tried again, but this time it reported that it could not find the destination folder.

 

Then I tried accessing my unRAID server from the file system (Network Places). It couldn't find it. I tried accessing it from the web interface. That didn't work. I tried other machines on the network. It appeared that the unRAID server stopped working. I couldn't use the web interface to stop the array and restart the server. I checked the actual box, which is located in my basement, and it appeared to be running fine. I tried restarting my Netgear gigabit switch and my Linksys router, but that didn't help either.

 

I decided to shut down the unRAID server. After shutting it down, I opened it up and installed the new 1 TB drive. I did not connect it because I want to make sure everything is ok before I make any changes. Besides, I still have to purchase the plus version. I connected the machine to a  keyboard and monitor and restarted to see if it booted ok. It booted up and seemed to take a little while to mount the drives, but eventually it was running fine. I tried accessing it from another PC on the network and it is there. The main screen from the web interface now shows that Parity-Check is in progress and it shows 1 sync error. In the disk status section, none of the 3 drives report any errors. I will let it complete the parity check before doing anything else.

 

I have no idea why the unRAID server stopped working. Does anyone have some suggestions that might help me diagnose potential problems so this will not happen again? Now I'm a little nervous about purchasing the plus version and connecting the new drive. Any advice would be greatly appreciated.

 

Thanks for your help.

 

update: after running the parity check for about 1/2 hour and pressing the 'Refresh' button on the main screen, the command area now shows 55 sync errors.

It is too bad that you were not aware of the need to capture the syslog before rebooting, as that might have showed us what went wrong.  But since unRAID runs entirely in RAM, the log is gone.  It would be useful to capture it now though, and attach it, so we can see if anything is wrong with your setup.  For various ways to view and capture the syslog, see my sig and this thread:  http://lime-technology.com/wiki/index.php?title=Viewing_the_System_Log.

 

GreggP -

 

How did you shutdown your unRAID server?  Shutting down from the command interface requires a program called "powerdown" which can be downloaded from this forum.  If you just use the normal shutdown command (or worse yet hit the big red switch), you will leave your array "open" and when you reboot it will force a parity check.  It is not unusual to have sync errors after an unclear shutdown.  I wouldn't worry too much about your sync errors - I once had a similar thing happen and no data was corrupted or lost.

 

As RobJ said, capturing your syslog would have been smart.  Knowing what happened based on a "blow by blow" user experience just isn't that useful.  We have to see what the computer said was happening - and only the syslog can tell us that.

 

Let your parity check finish, and then run another one to make sure all is well.  Then go ahead a add your third disk (or run for a while to see if your problem recurrs).  If it happens again, make sure to capture the syslog.

RobJ - Thanks for the advice and information for capturing the syslog. I'll read through it to make sure I understand the process.

 

bjp999 - Since I couldn't find any way to communicate to the unRAID server from another machine using the web interface and I was running my unRAID server without a monitor or keyboard, I decided to just turn the system off manually. I wanted to use the web interface to stop the array and shutdown the server, but it must have lost its connection to the network. That's why I tried restarting the router and switch.

 

So I don't think I could have captured the syslog with a telnet session from another PC. Or can you accomplish this under these circumstances. I could have connected a monitor and keyboard, but I don't know how I could capture the syslog to another PC on the network.

 

I now have the unRAID server running in my office with a keyboard and monitor. The console is showing the following message:

 

Fixing recursive fault but reboot is needed!

md0: parity incorrect: 1189243096

 

I have a telnet client called PuTTY, but I will need to read up on this to understand how to use it.

Okay, something weird is going on. I was just experimenting with running a telnet session using PuTTY. I could log on to the unRAID server and view the syslog. I was trying to figure out how to capture the syslog to a file so I would understand the procedure. I am also watching the text that appears on the unRAID server's monitor. That's how I read the message quoted in the previous message.

 

I've also been monitoring the progress of the parity check and the number of sync errors. I just tried to refresh the screen to see how many more minutes were estimated to complete the parity check and to see if there were any more sync errors. Now it seems to have lost its connection to the unRAID server. The telnet session crapped out as well.

 

Now the console on the unRAID server shows this text:

 

Kernal panic - not syncing: Fatal exception in interrupt

 

I can't figure out how to capture the syslog via a telnet session. Now what should I do?

Search the forum for "kernel panic".  Use advanced search.  This has come up from time to time, and I think there are some good explanations of what it means and things to do to stop these errors.

Perhaps the instructions in the wiki might help:

http://lime-technology.com/wiki/index.php?title=Troubleshooting#Capturing_your_syslog

 

Joe L.

It looks like it is impossible to capture a syslog. I cannot get a telnet session working.

 

I also did a search for "Kernel panic" and found one thread about the same message I see - "Kernal panic - not syncing: Fatal exception in interrupt"

 

I have a new motherboard, power supply, RAM, hard drives, etc. Everything seems to be running fine. The fans are all working, the hard drive temps were all under 25° C. It looks like this is going to be pretty difficult to troubleshoot. Damn...

log in on the system console and type:

 

ifconfig

 

You should see a section for eth0... does it have an IP address assigned?  If so, you have network connectivity.

also type:

 

ls /boot

 

You should see the files on your flash drive, if you do not, then something is keeping it from being mounted properly.

 

 

and lastly, if you see files on the flash drive... type:

 

cp  /var/log/syslog  /boot/syslog.txt

 

Then, at the prompt type

 

samba stop

mdcmd stop

 

and

poweroff

 

Then, transfer the flash drive to your PC, and the syslog.txt file should be there.  (you might have to enable viewing of hidden and system files on the PC for it to be visible)

 

Joe L.

log in on the system console and type:

 

ifconfig

 

You should see a section for eth0... does it have an IP address assigned?  If so, you have network connectivity.

also type:

 

ls /boot

 

You should see the files on your flash drive, if you do not, then something is keeping it from being mounted properly.

 

 

and lastly, if you see files on the flash drive... type:

 

cp  /var/log/syslog  /boot/syslog.txt

 

Then, at the prompt type

 

samba stop

mdcmd stop

 

and

poweroff

 

Then, transfer the flash drive to your PC, and the syslog.txt file should be there.  (you might have to enable viewing of hidden and system files on the PC for it to be visible)

 

Joe L.

 

Joe-

 

After restarting my system manually again (after the "Kernal panic" the keyboard wasn't responding, in fact the indicator lights were just flashing, and I couldn't use telnet or the web interface from another machine to get a syslog before shutting it down) and was able to get to a prompt after logging in.

 

ifconfig returns an eth0 section and displays an address (inet addr:192.168.1.100)

 

I am able to see the files on my flash drive and generate a syslog.txt file.  I've attached the file to this message.

 

Thanks!

After manually restarting the server and following Joe's instructions, which included the samba stop, mdcmd stop and poweroff commands, and then restarting the server, it booted up and ran the unRAID software. From a web interface I could get to the main screen it showed that it was running the parity check again. This time, it wasn't showing any errors. So I decided to just let it run. After a couple hours it still looked fine, but then suddenly, it stopped working. I can no longer access the unRAID server from the web interface.

 

I cannot get a telnet session from another machine either.

 

And, when I look at the console of the unRAID server, it shows a bunch of text that doesn't mean anything to me. It includes a Call Trace list and some Code listing and ends with

 

EIP: [<c0190d53>] search_by_key+0x66a/0xcd8 SS:ESP 0068:df5ddcf8

---[ end trace f9126f14440350dd ]---

 

The cursor is just flashing at the bottom of the screen and the system doesn't respond to typing on the keyboard. When I try to telnet into it from another machine, I cannot get to the login prompt.

 

Is their some way I can capture the syslog before shutting it down?

 

 

Not without a keyboard.  Something about your system is not working correctly, or incompatible with the Linux kernel.  Your syslog was captured shortly after unRAID boot, so does not capture any of the errors you have been seeing.  It looks completely fine, except for the following section of 4 lines, of ACPI errors and exceptions:

ACPI Error (dswstate-0226): No result objects! State=f7f71a00 [20070126]

ACPI Exception (dsutils-0642): AE_AML_NO_RETURN_VALUE, Missing or null operand [20070126]

ACPI Exception (dsutils-0735): AE_AML_NO_RETURN_VALUE, While creating Arg 1 [20070126]

ACPI Error (psparse-0537): Method parse/execution failed [\_TZ_.THRM._TMP] (Node f7c16078), AE_AML_NO_RETURN_VALUE

 

I don't know the significance of these errors.  I do know I have never seen them before, and therefore they may be indicating a BIOS or motherboard defect or incompatibility.  I just did some Googling, and now I suspect you may need a BIOS update.  I don't know if these ACPI errors are related to your crashes, but did run across someone else with similar errors who also got a locked keyboard.

 

I updated the BIOS just after I assembled the server and before trying to setup unRAID. Other people who were using this Abit AB9 Pro motherboard reported some problems with the onboard LAN controller that was resolved with the update. So I made sure this was up to date. This was only about a month ago. I''ll have to see if there have been any recent updates.

 

I wonder why it would run perfectly fine for the last month and now start to give me problems. When there is a problem with the BIOS, wouldn't it have caused problems from the start? I would think this is more like some kind of defect, but it seems impossible to diagnose without replacing components until the problems disappear. Unfortunately, it is difficult to know where to start. Since reporting this problem earlier this morning, the only component that I've swapped out is the video card. That didn't make any difference, so that leaves the motherboard, RAM, CPU and hard drives.

 

This is extremely frustrating. I wish I could at least capture the syslog after unRAID stops working, but that doesn't seem possible. When this system crashes it leaves me with a frozen system. The last thing I typed into the unRAID console was "tail -f /var/log/syslog" so whatever is on my screen should be the last screen of the syslog before it froze the keyboard. Is there anything I should look for on this screen that might give me a clue as to what is wrong?

 

I can also use a telnet client to open a telnet session from another PC. This just gives me a blank screen with a blinking cursor, so none of the OS prompts are displayed and I can't type in any commands to get a syslog.

 

I really appreciate your advice, hopefully I can figure this out without having to spend the money to replace every component.

 

Thanks,

Gregg

 

update: I just checked the Abit website to see if there is a newer BIOS for the AB9 Pro mobo and the latest is version 22, which I updated to when I built this system a month ago.

 

You know you're getting desperate when you type every character you see on the console into notepad on another machine. It's moments like this that I can really appreciate the cut and paste features in Windows.

 

Here's the last screen displayed on my unRAID console:

 

[pre]ESI: 00000000 EDI: f525a460 EBP: f7df6e00 ESP: df5ddcf8

DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068

Process shfs (pid: 1842, ti=df5dc000 task=f7cc0530 task.ti=df5dc000)

Stack: 00001000 00000000 00000000 0000443b 7acd4b80 00000000 0000000a df5ddd00

      00200086 0000000a df5ddef8 00000010 c2100018 00000001 00000001 00000018

Call Trace:

[<c0184e56>] reiserfs_init_locked_inode+0x0/0xe

[<c0133e07>] mempool_free+0x4e/0x52

[<c01e125d>] elv_queue_empty+0x1d/0x1e

[<c01598a5>] __d_lookup+0x84/0xc6

[<c017f7e6>] search_by_entry_key+0x19/0x240

[<c01866cc>] reiserfs_readdir+0xd9/0x431

[<c01554b0>] filldir64+0x0/0xc5

[<c0155678>] vfs_readdir+0x4a/0x74

[<c01554b0>] filldir64+0x0/0xc5

[<c0155705>] sys_getdents64+0x63/0xa5

[<c0103b4e>] syscall_call+0x7/0xb

==================

Code: 89 d1 31 db 8b 6c 24 24 31 d2 39 d3 0f b6 45 10 7f 0f 7c 04 39 c1 77 09 8b

44 24 20 e9 b8 f9 ff ff 8b bc 24 e4 00 00 00 8b 5f 18 <0f> b7 03 3b 44 24 60 89

44 24 30 74 25 8b 4c 24 60 89 44 24 08

EIP: [<c0190d53>] search_by_key+0x66a/0xcd8 SS:ESP 0068:df5ddcf8

---[ end trace f9126f14440350dd ]---[/pre]

When something like this happened to me, I used the camera in my cell phone to take a picture of the text on the screenand posted the picture.  You might be able to do the same.

 

It looks like you experienced a "Kernel Panic"

 

Kernel panic's are often affiliated with memory errors.  I recommend as a first step you run the memtest from the unRAID boot menu and see if it shows anything.

 

Joe L.

 

 

Wow!  What a lot of typing!  And, as you probably feared, mostly wasted, unusable!  Sorry.  There are 3 things to note however, to compare with similar crash screens, to determine if the crash happens in the same place, or is completely random.  The first is the process near the top, shfs.  The second is the current EIP near the bottom, search_by_key, just the label, don't care about the offset after.  And the third is just the first item in the Call stack, reiserfs_init_locked_inode.  If you can recall from previous crashes seeing anything similar, that would be helpful.

 

*If* the keyboard were working at all, you can use the Shift-PgUp key combination to page up through the screen buffer, to see the previous screens.  PgUp does not work here, but Shift-PgUp acts like a PgUp.

 

As Joe says, a digital pic is the easiest, but it does take a little experimentation, to get a clear, detailed, focused image, that is as clear as possible of glare, especially flash glare.

 

I recommend adding the --lines parameter to the tail command, because it gives you much more to page back through, with the Shift-PgUp key.  I use tail -f --lines=99 /var/log/syslog.

 

I'm sorry a BIOS update wasn't available.  I still hold a little hope that a future BIOS update will fix it, as some of the online discussion, currently ongoing, had to do with problems in the tables provided by manufacturers that ACPI uses.  There is a very very small chance that going back to a previous BIOS might help.  The current discussion is very low-level core stuff, and has not produced a solution yet, and any patch produced will take awhile to be available in a general kernel release.

 

Hopefully, it is just bad memory, not a good thing, but easily fixed.  Otherwise, a motherboard replacement may be in your future.

It looks like you experienced a "Kernel Panic"

 

Kernel panic's are often affiliated with memory errors.  I recommend as a first step you run the memtest from the unRAID boot menu and see if it shows anything.

 

I'm not sure how you deduced there was a Kernel panic from that screen, but you are right. After I finally manually restarted the server, and tried to boot into unRAID, the boot process froze and the text at the bottom of the screen reported "Kernel panic - not syncing: broken padding". A Google search for this showed that someone solved it by running memtest (in their case v3.2), which showed one of memory modules had gone bad.

 

I'm not sure if this is related to the previous Kernel panic I reported in my 2nd post. That one was "Kernal panic - not syncing: Fatal exception in interrupt". A search on the unRAID forums seems to indicate that this one was due to some other hardware problems like a bad PSU.

 

I shut down the server and removed one of the 1Gb memory sticks (Crucial 2GB DDR2 804 240 Pin DIMM Ballistix Kit). Then I restarted the server to see if it could boot  into unRAID without the Kernel panic. It worked fine, so I swapped memory sticks and tried booting into unRAID again. This one was fine as well. So I put them both back in and tried again. And again, it booted into unRAID without the "Kernal panic".  Anyhow, instead of trying to run unRAID another time, I decided to run Memtest86. I started the test around 12:30 AM and it is still running this morning and it reports:

 

WallTime = 9:05:12

Cached = 2047M

RsvdMem = 276M

MemMap = e820-Std

Cache = on

ECC = off

Test = Std

Pass = 21

Errors = 0

 

So it looks like the memory sticks are working. Maybe the memory sticks weren't seated well and swapping them around fixed the problem.

 

Yesterday, I was also able to restart unRAID normally and the problems didn't occur until it was running for several hours. It usually happened when I tried to read the contents of a folder from one of my other computers. Before the last crash, I was trying to view one of the directories on my unRAID server from my office PC. I picked 'Network' from the Start menu and it displayed all my network devices, including the unRAID server, which I've renamed from Tower to Media. I double clicked on Media and it displayed the folder's that I've created as shares. These include "Movies", "Video" and "Music". The "Music" folder is physically located in disk1 and the "Movie" and "Video" directories are located in disk2. If I double clicked on either the "Movie" or "Video" directories, it displayed the contents of the folders. When i clicked on "Music" it reported an error that I might not have the correct permissions. This was strange, because all of the shares had identical settings, except the physical drive assignments. So then I tried typing "\\Media\disk1\" and/or "\\Media\disk1\Music" in the address bar and both of these let me display the contents. I went to a few other computers on the network and tried the same test. I couldn't display the contents of the Music shares using any of these methods. Then I went back to my office PC to see if I could still look at the Music folders and it no longer let me. And, I couldn't view the contents of "Movies" or "Videos" either. The unRAID web interface no longer responded and it could no longer find a connection to \\Media. I then saw that the unRAID console was frozen and the text that I eventually copied was on the screen.

 

So I'm going to stop the memory test for now and try running unRAID and actively monitor the Syslog again. This time with RobJ's added parameters.

 

Thanks again for your help.

GreggP -

 

You started this post saying that your unRAID server had been running for several weeks without error, and then you started having this problem.  Your memory must have been seated properly at that time.  I think you have isolated the problem to a particular RAM stick.  I (personally) would be VERY skeptical about believing that the RAM stick was suddenly good.

 

RAM can be finicky.  I've seen people in overclocking forums that run RAM tests and get errors after several DAYS of continuous pounding on the memory.

 

unRAID will run fine with 1G.  Why don't you contact Crucial and get a warranty exchange on the bad stick?

It would not hurt to run a file-system check on your disks either, just in case the panic stops have resulted in any corruption.

 

The instructions are here in the wili: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

 

It is done with the array started, from a telnet prompt, on each of your data disks, md1 through md15  (or rather, on those you have disks assigned to)

When prompted by the reiserfsck program to continue, you must respond by typing "Yes" (with a capital "Y" and lower case "es")

 

Joe L.

 

GreggP -

 

You started this post saying that your unRAID server had been running for several weeks without error, and then you started having this problem.  Your memory must have been seated properly at that time.  I think you have isolated the problem to a particular RAM stick.  I (personally) would be VERY skeptical about believing that the RAM stick was suddenly good.

 

RAM can be finicky.  I've seen people in overclocking forums that run RAM tests and get errors after several DAYS of continuous pounding on the memory.

 

unRAID will run fine with 1G.  Why don't you contact Crucial and get a warranty exchange on the bad stick?

 

How would you identify which stick is bad?

I shut down the server and removed one of the 1Gb memory sticks (Crucial 2GB DDR2 804 240 Pin DIMM Ballistix Kit). Then I restarted the server to see if it could boot  into unRAID without the Kernel panic. It worked fine, so I swapped memory sticks and tried booting into unRAID again. This one was fine as well. So I put them both back in and tried again. And again, it booted into unRAID without the "Kernal panic".  Anyhow, instead of trying to run unRAID another time, I decided to run Memtest86. I started the test around 12:30 AM and it is still running this morning and it reports:

I just read this again carefully.  My thought would be the first stick you pulled out was bad, but after re-reading this section, I guess it would be hard to know for sure.  You are probably doing the right thing running your memory tests.  I would run it overnight tonight.

Alright, it got through the parity check and it reported 1 error near the very end.

 

The console reported - "Fixing recursive fault but reboot is needed!"

 

To capture the syslog, I typed "cp /var/log/syslog /boot/syslog.txt" and I also tried to capture it to the disk1 file share, so I typed

"cp /var/log/syslog /mnt/disk1/syslog.txt"

 

the console then reported

 

md0: parity incorrect: 1946157840

md: sync done. time=16844sec rate=57988K/sec

md: recovery thread sync completion status: 0

 

I then checked my disk1 share to see if the syslog.txt file was there. It wasn't so I tried again with this -

"cp /var/log/syslog /media/disk1/syslog.txt" and the file still didn't appear on the disk1's root.

 

I figured it would be on the flash, so I started the process of shutting down the server. Using the web interface, I stopped the array. It showed 'Media emhttp: shcmd (13): killall -w smdb nmbd' at the console. Then I lost control from the web interface. It didn't shut down the computer so I used the process Joe listed in a previous post:

 

samba stop

mdcmd stop

poweroff

 

None of these commands appeared to do anything. Then I realized that I was probably still in the active monitoring mode, so I typed ^C to get a prompt. Then I tried the 'samba stop' etc. The console than responded "nmbd: no process killed'. Then I tried 'poweroff' and the console then showed:

Broadcast message from root (vc/1) (Sat May 10 15:09:21 2008):

The system is going down for system halt NOW!

INIT: Switching  to runlevel: 0

INIT: Sending processes the TERM signal

INIT: Sending processes the KILL signal

Running shutdown script /etc/rc.d/rc.0:

Saving system time to the hardware clock (UTC).

nmdb: no process killed

Unmounting remote filesystems.

BUG: unable to handle kernel paging request at virtual address 0044a94e

printing eip: c0155e27 *pde = 00000000

Oops: 0000 [#2]

Modules linked in: md_mod fuse pata_jmicron jmicron ide_core ahci sata_sil24 lib

ata r8169

 

There are several more screens of info after this.

 

After this, I realized that the syslog was probably not captured because I wasn't typing it from the command prompt and was still in the monitoring mode. So I tried typing 'cp /var/log/syslog /boot/syslog.txt' and it just returned to a flashing cursor. So I think I screwed up AGAIN!!!

 

Aaaaarrrrggggg. I did take some screen shots with my digital camera and I'll attach the first 2 to this post. I can post others if it will provide any more info.

 

I still have a flashing cursor on the server's console and I can PageUp and PageDown to view any screen. Is there a way to capture the syslog while I'm in this state, or am I too late? I'll keep the server running until I hear back from one of you.

 

Thanks again,

Gregg

I thought you were going to run a memory test overnight?

Yes, sorry for the misunderstanding. I did that last night. That test did not return any errors. However, after restarting the server, while booting in unRAID, it gave me another "kernel panic = not syncing: broken padding" message. I restarted and then ran another memory test. This time all sorts of errors appeared. So, I removed one stick of memory and tried again. Immediately, the test was showing errors. Then, I swapped memory sticks and ran Memtest again. With this stick, there were no errors. At least not so far. Hopefully, I have found the problem. I will continue testing the single memory stick for awhile to make sure it is ok.

 

It was strange that last night's test with both sticks didn't find any errors. I've never known memory to have this sort of erratic behavior. I thought that either it works or it doesn't.

Some memory sticks need higher voltage to be able to work properly.  It is possible one of yours is working, but just barely.  When you use the other, or when things heat up, or cool down, or heavy disk activity occur it might just push it outside of its safe working environment.

 

Check ithe bios settings if you moitherboard has them, it might be as simple as raising the voltage a tenth of a volt or so to the memory, and both memory sticks will work.

As many have said, unRAID work fine with 1Gig... in fact, it works fine with 512Meg (the amount I've had in my server for the past 2 years or so)

 

Joe L