I am having a strange problem.  After my box boots up, I can access the shares without any problem.  But after a prolonged period of time, the shares are no longer available.  I reboot, and it works again.  It is extremely frustrating as my backups are failing in the middle of the backup.  Any ideas?

There are various possible conditions that could fill all of memory, and cause the server to close down processes, including the networking.  We have also seen cases of flash drive problems, where it would work initially to boot the system, but fail later, possibly causing loss of networking.  For these and other issues, we need to see a syslog, see my sig for the Troubleshooting link.

Thank you for your help.  Here is what I've figured out.

 

My log file was too big to fit on my flash drive.  I have put it on one of my disks, and it is over 800MB.  This was after one of my huge backups, (hundreds of thousands to millions of files.).

 

Currently, my lime box won't even let me access it via the browser, but I can still get in through tellnet.  Using the "free" command, I was able to figure out that I am using 988,588 of 1,000,676.  I am thinking that being out of memory is why I am having to re-boot. 

 

Is this huge 800MB log file stored in memory?  It has to be, since it isn't on the flash drive and it isn't on any of the disks.  Is there a way to limit how big this file can get?  Is there any way I can free memory up from the command line so I can access the box via the network again?

 

<<Edit>>

 

This will clear your log file for anybody looking to do it.  > /var/log/syslog

 

I still would like to know if there is a way to limit the logfile size.

log files should never get this big under normal conditions. deal with the cause not the symptom.... read your log file and fix each problem as you find it.

 

also fyi syslogs compress massively (think 95% + compression easily)

Once you've truncated the syslog, you will probably get back some memory.

 

You can re-start the unRAID web-interface by typing

nohup /usr/local/sbin/emhttp &

 

The bigger issue is why your syslog filled to where it used all the available memory.

 

Can you do a

cd /mnt/disk_where_you_put_your_syslog

tail -500 syslog

 

To identify the error messages that caused it to fill?

 

Joe L.

I wound up rebooting in order to get my web interface back.  Immediately after reboot, it starts running a parity check.  Is this normal?  I will let it finish the parity check - will probably take all day - and then I'll run a backup to re-create the enormous log file, and run your command to get the last of it. 

 

Many thanks for all the help.

time to start posting syslogs so the helpful peeps here can deal with specifics.

 

If you dont know how to do anything like this just ask

I wound up rebooting in order to get my web interface back.  Immediately after reboot, it starts running a parity check.  Is this normal?  I will let it finish the parity check - will probably take all day - and then I'll run a backup to re-create the enormous log file, and run your command to get the last of it. 

 

Many thanks for all the help.

Yes, it is very normal for the parity check, as the system was not stopped cleanly.

 

Too bad you did not see my prior post in time, or ask how to shut down cleanly from the telnet prompt.

 

It is time to post your current syslog... now, before you do your copy.  That way, we have a baseline to compare to and can spot any errors before you do a massive file copy.

 

Then, while you do the copy, you can also keep an eye on the syslog.  If it starts growing, it's time to get advice on how to proceed.  We'll need a new copy then. (not all 800 Meg, but a sample of the errors... preferably when they first start)

 

I suggest you install one of the packages I developed to help you in keeping an eye on your server.  It is "UnMENU", an improved management interface to unRAID. Built into it is a syslog viewer, and a link to easily download a copy of the syslog to your PC (it will download only the last 10000 lines, so don't worry, you will never get 800Meg)    That UnMENU interface is described here in the wiki. http://lime-technology.com/wiki/index.php?title=UnRAID_Add_Ons#UnMENU  (It points to the thread where the download link exists)

You can install it now, while parity is being calculated.  The main page even includes a view of the last few lines of the syslog, automatically updated every 60 seconds, and color coded for errors it recognizes.

 

Joe L.

1.  Current Log File - ftp://ftp.kellpro.com/Pub/NS_Tools/syslog.txt

2.  Awesome, I'll look around and install the UnViewer.

3.  Just for the record, how do I shut the server down via telnet?  I just used shutdown -r -t: "now"

 

Again, thanks for the help...

1.  Current Log File - ftp://ftp.kellpro.com/Pub/NS_Tools/syslog.txt

2.  Awesome, I'll look around and install the UnViewer.

3.  Just for the record, how do I shut the server down via telnet?  I just used shutdown -r -t: "now"

 

Again, thanks for the help...

Your command will shut down the server, but not cleanly stop the array, so you will be faced with a full parity calc on reboot.

 

I'd install the powerdown package and use it.  Then it will stop the array when you use shutdown, as it has hooked into the shutdown script.  If it is not installed, In a pinch, you can look at the instructions here: http://lime-technology.com/forum/index.php?topic=2781.0

Syslog looks OK so far.

 

15 terabyte server, trying not to be envious here... 

 

Were you able to save that 800MB syslog to a permanent location?  If so, can you extract the first couple of thousand lines, zip them, and attach or make available?

Ohhh, I like the UnMenu!  I am having a problem with the part of the page that is supposed to read the syslog file because it can't find tower.  My box is named "Lime1".  I've gone through all the files and replaced any \\tower with \\lime1, but it doesn't seem to have fixed the problem.  Is there something I need to change?

 

ftp://ftp.kellpro.com/Pub/NS_Tools/screenshot.gif

 

<<Edit>>  This issue has been addressed in this thread & fixes the issue.  http://lime-technology.com/forum/index.php?topic=2595.60

 

I do still have that huge syslog file.  I will see if I can get some of it.

 

Well, I talked my employer to get it the lime box for backing up several million images after we had a corruption problem that randomly corrupted 10% of our production images across hundreds of jobs - so it isn't like I have this beast hooked up to my TV - I wouldn't mind having one though   I have been impressed at how robust the system is, and how inexpensive it is in comparison to other backup solutions out there.  One of the things I noticed is that the latest version, 4.3, is supposed to handle huge folder structures better.  I am also going to try to upgrade.

Ok, here is the first 3000 lines ftp://ftp.kellpro.com/Pub/NS_Tools/syslog1.txt

Ohhh, I like the UnMenu!  I am having a problem with the part of the page that is supposed to read the syslog file because it can't find tower.  My box is named "Lime1".  I've gone through all the files and replaced any \\tower with \\lime1, but it doesn't seem to have fixed the problem.  Is there something I need to change?

You can invoke the unmenu command with the correct server name as an argument, or create an unmenu_local.conf file as a copy of unmenu.conf and change one line in it. 

 

To invoke with the alternate server name use

awk -v MyHost=Lime1 -W re-interval -f unmenu.awk

 

or...

There is only one line you need to change in a configuration file, it is in the unmenu.conf file near the top.  But rather than change it, copy unmenu.conf to unmenu_local.conf and change it there.  Options changed there override those in unmenu.conf  (That way, next time you download an updated unmenu.conf file, it will not overwrite your local changes)

 

The line in unmenu.conf is close to the top of the file and currently is commented out with a leading "#"

You need to delete the "#" and change the name as needed.

# Unmenu will discover for itself your host name, but if you wish to set it yourself, or

# your host name is not set, then set the value of MyHost here.

#MyHost = MyTower

You want to change it to:

# Unmenu will discover for itself your host name, but if you wish to set it yourself, or

# your host name is not set, then set the value of MyHost here.

MyHost = Lime1

(Don't forget to delete the leading "#" on that line in unmenu_local.conf to make it active.

 

Oh yes,  you probably need to undo all the other lines you changed, or download a new copy of them.

ftp://ftp.kellpro.com/Pub/NS_Tools/screenshot.gif

Looks good.  It will look better once you create the unmenu_local.conf file.

 

I do still have that huge syslog file.  I will see if I can get some of it.

 

Well, I talked my employer to get it the lime box for backing up several million images after we had a corruption problem that randomly corrupted 10% of our production images across hundreds of jobs - so it isn't like I have this beast hooked up to my TV - I wouldn't mind having one though   I have been impressed at how robust the system is, and how inexpensive it is in comparison to other backup solutions out there.  One of the things I noticed is that the latest version, 4.3, is supposed to handle huge folder structures better.  I am also going to try to upgrade.

Yes, you should upgrade to at least 4.3.3-final

Since it is a "production box, I'd wait till Tom releases 4.4 final before installing it, should be any day now.

 

Joe L.[/b]

It looks like you have a very nice 14TB server there, and it is half full.  That is a lot of image files, even if huge RAW images.

 

It sounds as if you are using TrueImage for backups.  (and the reason for the large file)  You might want to try their incremental backups once you get the initial once in place.

 

Yes, the later version of unRAID does a lot better with huge number of files.  You will want to upgrade.  Usually, it involves only replacing two files on the flash drive and rebooting.  bzimage and bzroot.  I usually re-name the existing two files to bzroot433 and bzrimage433  (naming them after the old release number), then copy the new files from the downloaded distribution to the flash drive, then reboot.  It is as easy as that.

 

You will want a screen-shot of the current drive assignments, in case the newer version of Linux scans them in a different order. (Worst case, the array will not start, you will need to go to the "devices" page and re-assign the drives back to their original slots, and then start it)

 

Joe L.

(I have about 6TB of storage on my unRAID server, and a fairly large collection of DVD ISO images on it... it is very nice as a home media server)

You are getting killed by duplication, in the User Shares.  It appears you are probably making a full backup to disks 2, 3, 7, 9, and sometimes 4, but the disks and folders being used are also in the same User Share.  The way a User Share works is to create a virtual folder system, of the merged contents of all of the constituent folders, but all of the usual file system rules still apply.  You cannot have the identical file name more than once within the same folder.  In this case, unRAID is logging a warning about the duplicate objects when it builds the virtual folders for the User Share.

 

Since you must be making these duplicate copies or backups directly to the Disks (instead of a User Share), you should consider turning off User Shares.  You probably did not create a User Share called NS_Computer, but once the folder of that name had been created at root level on multiple disks, unRAID automatically creates the NS_Computer share, if User Shares are turned on.

 

There may be alternative ideas as to the best way to deal with this.  One that comes to mind would be to rename NS_Computer to .NS_Computer on each unRAID disk, and then adjust the backup program to use the altered destination path.  The leading period causes unRAID to ignore it as a User Share.  Another idea would be to change the names more radically, for example, make one disk hold /Tuesday/NS_Computer, another /Friday/NS_Computer, etc.  If you turn off User Shares, then none of this is necessary.

 

Something I don't understand is why the system waited 10 hours before reporting all of these duplicates.  It reported a few simple ones immediately, but may have been building an array of these warnings for the entire NS_Computer share, for almost 10 hours!

This puzzles me.  I am backing up to user share, not to the individual disks.  I have highwater as my setting in all my share types.  On a nightly basis, I try to run three backups, all go into the shares, not to the disks.  As far as I can remember, I've never copied to the disks themselves.

 

If I understand what you are saying, it is warning me that I can't have the same file names in both places.  I did start having this problem about the time I started backup my computer up to the limebox, (NS_Computer), and so maybe it just has to do with this.  This is just a backup of my hard drive, so I can delete it and re-run it to another share without a problem.

 

If the problem is with my image files, I hope I can figure out some way to fix it short of a full re-copy or I'm screwed...

 

Thanks for all the help.  It's been fun.  When I'm finished with this parity check, I will modify my local backup, and see if I'm getting the same errors for the big backups of the image files.

 

I don't know if it's relevant, but I'm using a program called SyncBackPro to run my backups. 

 

P.S.  Last time I checked, there were > 10,000,000 image files...

You have massive duplication, so something is not working right.

 

A Bandaid 'solution': somewhere there is a similar discussion, and I believe that a command was found that would disable the logging of the duplication.  Joe or NAS may know what it is.  It does not fix anything, but would save all of the memory lost to logging messages!

 

Unfortunately, you have 2 problems now.  One is to figure out (not by yourself!) why it is happening, what is wrong.  The other is to begin deleting the duplicates, a delicate and often time-consuming operation, since you will want to be careful not to delete a more recent tree of files.  I would use MC, see the top of the Transferring Files Within the unRAID Server page.  I would set both windows to sort files by most recent date, and compare the same folders on different drives, and determine which are the oldest and can be deleted.  Sorry, but this can be a huge job!

This puzzles me.  I am backing up to user share, not to the individual disks.  I have highwater as my setting in all my share types.  On a nightly basis, I try to run three backups, all go into the shares, not to the disks.  As far as I can remember, I've never copied to the disks themselves.

 

If I understand what you are saying, it is warning me that I can't have the same file names in both places.  I did start having this problem about the time I started backup my computer up to the limebox, (NS_Computer), and so maybe it just has to do with this.  This is just a backup of my hard drive, so I can delete it and re-run it to another share without a problem.

 

If the problem is with my image files, I hope I can figure out some way to fix it short of a full re-copy or I'm screwed...

 

Thanks for all the help.  It's been fun.  When I'm finished with this parity check, I will modify my local backup, and see if I'm getting the same errors for the big backups of the image files.

 

I don't know if it's relevant, but I'm using a program called SyncBackPro to run my backups. 

 

P.S.  Last time I checked, there were > 10,000,000 image files...

If using "user-shares" you can have (and will have) the two files with the same name . One in the disk share, and the other in the user share.  That is by design.   When you copy a file to the user share, it is actually copied to a disk share and a "virtual" link to it is presented to you in the user share.  The user-share link exists only in memory.

 

What you can't have is two disk shares, with the same hierarchy, both having the same file name in the same place in the directory hierarchy.  It is only possible to make a "virtual" link to one of them for the user share, the other is only accessible in the disk share.

 

It is possible that with 10,000,000 files you are running out of memory to create the virtual links for the user-shares.  That might get things really confused.

 

How much RAM do you have in your server?  (You probably should have 2 to 4 Gig if it is supported by your motherboard)

 

Look on the bright side, once you get the duplicates resolved, you will likely have a lot more disk space available for your files.

 

You really need to automate the resolution of your duplicates.  Let me think of what can be done with a script.  If you have two (or more) files that are identical, (same md5 checksum) then all but one can be automatically deleted (don't delete the one in the user share... it is just a link to one of the others) .  If a different checksum is found, then they cannot be auto-deleted and you will need to resolve which is the one you desire.

 

At least a script can tell you what needs to be done, even if it does not do it itself.

 

Joe L.

Joe L.

I use and like and recommend SyncBackSE, but have not upgraded to the new SyncBackPro.  But even if it were buggy, or misconfigured, it would not be at fault for creating this duplication.

 

Because it looks like it took about 10 hours before, and you have over 10 million files in this share, I'm beginning to think it may have choked on it, may not have built the complete virtual tree in time.  Spacewise, from your screen shot, it looks like highwater is working great.  But your 1GB of RAM and your Celeron may not be up to the task.  You really need more memory, and this is one time a faster CPU would really help.  A swap file may help, but others would be better for advice on that.

 

[i'll stop, I see you are getting better advice.]

I use and like and recommend SyncBackSE, but have not upgraded to the new SyncBackPro.  But even if it were buggy, or misconfigured, it would not be at fault for creating this duplication.

 

Because it looks like it took about 10 hours before, and you have over 10 million files in this share, I'm beginning to think it may have choked on it, may not have built the complete virtual tree in time.  Spacewise, from your screen shot, it looks like highwater is working great.  But your 1GB of RAM and your Celeron may not be up to the task.  You really need more memory, and this is one time a faster CPU would really help.  A swap file may help, but others would be better for advice on that.

 

[i'll stop, I see you are getting better advice.]

You might be right on there.  If it attempted to copy to disk1 one day, and did not have enough memory to complete the user-share tree, the next time it might try disk2, since it had the most free space, and so on, and so on.

 

So... first step, get your parity re-calculated.

Then, I'd suggest you upgrade to 4.3.3final (as you said it had improvement in handling user-shares from the 4.3.2 version you are running.  Specifically this one

Bug fix: Turn off user share file system debugging output accidentally left on - doh!

and this one

    * Bug Fix: Fix crash that may occur if there is a large number of user shares.

and this one

Bug fix: fix problem in user share file system with 'chown' command.

) 

 

The debugging output alone might be a lot of your syslog, although it might be the initial cause of the filling of memory, and subsequent crash of user-shares (your in-ability to complete a huge transfer) it does not correct the duplicate files you currently have on your server.

 

After you have parity calculated, and then upgrade to 4.3.3 then we can look at the syslog once more.  By then, I'll figure out some script that might help.

 

Joe L.

 

We ordered one of the pre-configured lime boxes from Lime Tech.  It came with 1GB of ram.  I am going to try to get an order in today and get it kicked up to 4GB.

 

When the parity check is done, I will do some comparison on the different disks to check for duplicate files.  It will be clunky, but with dir commands plus some sql server skills, I can do a mass compare on all 14 drives.

 

I will update the OS, and start one of the jobs.  I will monitor the syslog to see if it is writing stuff for each file.

 

Many thanks for all of the help and ideas. 

For duplicate detection and control, I use this program. It's worked great for me.

There are some settings that are a little too conservative out of the box and need to

be tweaked in order to be totally effective.

 

http://clonespy.de/

Just an update for anybody else who runs across this post.  Here is what I did to fix the problem - Well...I think it's fixed anyway

 

1 - I updated to UnRAID.  See http://lime-technology.com/forum/index.php?topic=2235.0 for update instructions.  The only trouble I had with upgrading is that the upgrade included a file that overwrote the name of my server and workgroup with defaults.  I would recommend creating a backup of your flash drive, (luckily I did) and using a compare program to compare the files, (I used Beyond Compare 3) to easily find the configuration files that are different, and port your old settings over to the flash drive.

 

2 - I deleted the files that had been duplicated from each individual drive.  I have only tried this on my smaller share that I will be able to re-run a full backup on in a reasonable amount of time.  If you have several terabytes of data, you will probably have to figure out a way to look for duplicate files on drives, and delete all but one copy.  If I understand correctly, based on previous posts, the share is a collection of "shortcuts" to the actual file.  Somehow, you would have to delete the files that were not being used by the sharefile(s), and leave the one the share is pointing to, or you might have to delete the share, clean up your files, then re-create your share.  Thankfully, this isn't something I've run into, (yet).  If I do, I will post back here.

 

3 - I cleared my log file, (> /var/log/syslog), and monitored it with the UnMenu add-on, (http://lime-technology.com/forum/index.php?topic=2595.msg20982#msg20982).  I also re-wrote it to my flash drive just to make sure.  (cp /var/log/syslog /boot/syslog.txt from the console) and found that it was indeed empty.

 

Anyway, many many many thanks for all the help.  I have learned a ton, and you guys have been extremely patient with me.

Glad to hear you got it solved.

 

A few posts have been made about syslog explotion due to dupe file detection. I believe there is still a problem with the daily logrotate that could be addressed in the next beta to lessen the impact  but I still think that anything that could cause unRAID to fail should be considered a bug. Its fair to say that there needs to be a bit of a perfect storm for this to happen but some thought should be given to how to solve this reliably in the long term. A syslog daemon that can generate hundreds of megs of duplicate logs needs a bit of taming (especially since each log eats precious RAM)