Recently, I found that my unRAID server had a problem: the shared directories not longer appeared on my LAN (smb).  I rebooted and found that it did not mount the disks nor the shares and I could not access the web GUI in a browser.  I can telnet to the server and log in, but many commands will cause the telnet session to hang (ls of some directories, cat, less, etc).  Very hard to debug in this state.

 

I copied the syslog to the Flash and then mounted the flash on my PC.  This is the only way I can get data to/from the server.

 

I suspect a hardware problem of some kind, but the server boots into the BIOS and POSTS and passes the memory test, so I don't have any clue what might be wrong so I am trying to parse the system log.  Now, I had another log from a previous date when I was debugging another problem (slow writes) but the system was working, so I had the idea to compare, line by line, the old (call this good) log with the current (call this one bad) log using the cygwin sdiff command.  I have attached both logs and the sdiff output.  In the sdiff, the good log is on the left and the bad log is on the right and I removed the date and time at the start of each line because that would cause all the lines to mis-compare. The problem is, I am not a Linux expert so I don't really know what to look for.

 

I think some of the differences are due to the way Linux derives some value.  Lines 146, 147 are different because it calculated the CPU frequency slightly differently.

 

Many of the differences are due to things happening is a slightly different order. For instance 596-601 in the bad log happen at lines 622-626 in the good log.  I think these are not important, right?

 

So, reading down the file, many of the diffs I think are unimportant.  The same disks are discovered (in slight different order), the same processes are started etc.

 

One place that seems wrong is lines 3 and 4.  In the old (good) log, rc.inet1 seems to be configuring the loop back IP port and the bad one does not ever do this.  No idea why.

At line 859 unmenu is started in the good log, but I have disabled it so it does not show in the bad log.  At line 940, the bad log says that auto start is disabled, that would explain why the shares are not mounted.  But why does the web GUI not respond?  I can't start the array without being able to access the GUI.  From there on, the good log shows the array startup, etc.  Line 1095 shows me telneting into Tower from my PC, but I can't really do much in the telnet session.

 

Can someone has some experience take a look at tell me what I'm missing?

 

Details:

 

No web GUI, no shares mounted.

Telnet works but most command cause the session to hang (cant even cntrl-C out).

 

Commands that work:

ls /home, ls /root, ls /share, ls /sys

Commands that hang:

ls /dev, ls /bin, ls /etc, ls /sbin, ls -R /sys, ps -e, less, cat of any file.

 

I have reformatted the Flash and copied fresh version of the files thinking that some file may have been corrupted, but it did not change the outcome.

 

Thanks for any suggestions.

sdiff_of_logs.zip

I'm still thinking a hardware problem. When you say it passes the memory test, do you mean you booted into memtest? If so, how long did you let it run?

 

When you say you copied fresh version of the files, what files do you mean? bzimage and bzroot are the important files as far as corruption goes. The OS is unpacked from bzroot.

 

Are you absolutely sure you got a good download? Did you check the md5?

 

You might also try a new flash with a completely new installation. Your drives will not be assigned but you should be able to get the webGUI.

 

 

I'm still thinking a hardware problem. When you say it passes the memory test, do you mean you booted into memtest? If so, how long did you let it run?

 

When you say you copied fresh version of the files, what files do you mean? bzimage and bzroot are the important files as far as corruption goes. The OS is unpacked from bzroot.

 

Are you absolutely sure you got a good download? Did you check the md5?

 

You might also try a new flash with a completely new installation. Your drives will not be assigned but you should be able to get the webGUI.

 

Ran the memtest for 16 hours, it passed.

Copied bzimage, bzroot and readme, per the instructions.  I just checked the md5 and it checks, so I think I have a good copy.  I re-formatted the Flash and then copied the files over.

 

Is it possible that one of the config files is corrupted?  I checked the "text" configs, disk.cfg, network.cfg, share.cfg but I can't check the "binary" configs, secrets.tdb and super.dat.  If one of those was corrupted, could it cause the emhttp gui to fail and all the other weird effects?

 

I am inclined to agree that it is a hardware problem.  There was brief period when the server would not boot.  Switching on the power resulted in one of the fans "twitching" but no boot.  I opened the box and started to get ready to dissasemble it but tried once more and it booted and POSTED fine.  I ran the memtest overnight then and it has booted fine ever since.  I checked the BIOS settings and they are also OK.  Don't know how else to debug a hardware problem that has no symptoms.

 

One thing I thought I might do is to start over: copy fresh files, configs, etc.  If my disks are attached, what would effect would that have?  Of course, I am most worried about losing data.  Is this the same as the old "restore"?  If so, maybe I should disconnect all the disk from there SATA ports?

 

Anyway, thanks for the reply.  Did you get a chance to look at the sdiff of the logs?  I just don't see anything funny in the "failing" log.  It seems to start the unraid program and then just stop.

super.dat is your drive configuration, which drive is in what slot. It keeps track of drive serial numbers, and whether the array is stopped or started, probably some other things. Not sure about secrets.tdb

 

If you start with a completely new install, backup your flash somewhere. You only need to reassign your drives and use New Config from the Utils page. The main thing you need to be concerned with is assigning your parity drive correctly. You don't want to accidentally assign a data drive as parity. If in doubt, try to add all the disks to the array as data drives. The one that shows up as unformatted is parity, since it doesn't have a file system, and your data drives are already formatted as reiserFS. You should probably rebuild parity, but if you think your parity is good you can check the box. You should at least check parity when you get the array started again.

 

If you get your array back up you will have to set up your shares and users again. You can probably use the .cfg files from the backup to figure out what you had before.

 

If you want to try this and it doesn't seem to be working right or you aren't sure, just ask.

 

As you already found out, for some reason your emhttp is not starting.

Something must be wrong with your config.

Do you have a backup of your thumbdrive?

 

Save your actual content of your thumbdrive and then narrow the issue down by deleting the config files.

Start with the samba config.

Then remove the secrets.db

 

But I have to admit, that your console problems are somewhat strange!

 

You could try a different thumbdrive with  a stock install of unRAID.

Just to check if that one boots up the GUI. If so, it's probably "only" a config issue.

 

 

As you already found out, for some reason your emhttp is not starting.

Something must be wrong with your config.

Do you have a backup of your thumbdrive?

 

Save your actual content of your thumbdrive and then narrow the issue down by deleting the config files.

Start with the samba config.

Then remove the secrets.db

 

But I have to admit, that your console problems are somewhat strange!

 

You could try a different thumbdrive with  a stock install of unRAID.

Just to check if that one boots up the GUI. If so, it's probably "only" a config issue.

 

Thanks.  That was going to be my next plan. I have a backup of the flash, but I have made changes to the system (added disks, etc) since then.  This points up the need to do regular backups of this data, something I never thought about before.

 

Since I can read the "textual" config files (disk.cfg, etc) I don't think that they are corrupted.  Can't check secrets.tdb or super.dat.  What happens if I delete secrets.tdb?  Is this what you meant by the "samba config"?  If has no possibility of destroying data on my disk, then that would be a good next action to take.

 

Thanks for your help.

 

Set up a clean install. The disks will need to be assigned but the shares will appear with default configs.

The module that creates the syslog (syslogd I think) starts well after the boot is initiated, so has to catch up, creating the initial syslog.  And because initialization happens simultaneously with the many modules and kernel and user drivers, the lines do get scrambled.  As you noted, the inet ones often get scattered through the syslog, with 1 or all 3 often ending at the very top of the syslog, impossible of course but quite common in many syslogs.  All of the lines do occur in both of your syslogs.

 

I don't see anything particularly wrong in either syslog, but in both the network support (from your Realtek chipset) is really slow to get going, no idea why.  It would be useful to see an original syslog with the times intact, so we can check how long some of the events took.  UnRAID's emhttp is running in both, apparently without issue.

 

For those who want to read and compare syslogs, I recommend installing WinMerge (free, for Windows, don't know if other versions are available).  Then add prediffer files to get the comparison to ignore the first characters, the date and time.  Attached is WinMergePlugins.zip containing the 2 I use most often, just add them to the MergePlugins subfolder of the WinMerge program folder, then after selecting 2 files for comparison, click the Plugins tab then the Prediffer item and select which one you want.  This way you keep the timestamps (often useful), but they're ignored for comparison.

 

As you did above, comparing known good and bad syslogs can often be the fastest way to isolate an issue.  I do it all the time, using WinMerge.  It does a great job showing what's different and what's just moved.

WinMergePlugins.zip

Set up a clean install. The disks will need to be assigned but the shares will appear with default configs.

 

This is the conclusion I reached, as well.

To be clear: by "clean install" you mean, write only the clean unRAID install files to the flash, with none of the the config files, etc, including super.dat.

 

At this point I should expect to be able to connect to the unraid GUI via http and re-assign all my disks.  The problem is that I have to assign them to same "diskx" slot that they were in.  Is this correct?  What happens if I mix one or more?  I know that I *have* to get the parity disk right, but what happens if I mix one of the others?

 

Then I have to redefine all the shares?  Someone once told me that any top level directory would cause a share to be defined.  Is this correct?

 

Thanks for all the help.

I would first try keeping the config directory except the plugins subdirectory and keep the go file from the new install zip. Try keeping it all and if that doesn't work then delete parts and try again. I think just keeping super.dat, disk.cfg, share.cfg and the shares directory would keep the basic setup intact. replace the ones you don't keep with the defaults from the zip.

 

To try and answer your questions about starting over.

 

Yes, you'd have to assign the disks to the correct slots if you wanted them to be in those slots.

 

It doesn't matter if you mix the disks up, but make 100% absolutely positively you DO NOT assign a data disk to the parity slot. That is the only mis-assignment that will cause a big problem. You can first assign all the disks to be data disks and start the array. If you get one disk that shows up as unformatted then that is the parity disk.

 

The top level directories will create user shares since the top level directories are user shares. They would have default settings so you'd have to change those settings.