No entries in syslog - it stopped logging? disk disabled

[hilly]

Hi. So today at work i got mails to say a disk in my backup server had been disabled. I have come home and disk 3 is disabled and has 364 errors. This alone would be something i would search and resolve as i have a spare disk, just not sure how to do a smart test on the disabled disk as its not giving anything back when i go into health to check - so thats the first part

 

But here is an interesting one - attached is my syslog. given my server is still up and runing, has not had a restart and i have touched nothing you will see it had a number of entries about the UPS (its slaved off my main server and so when this is off it complains about beign able to see the UPS, i know about this, its not a problem) What IS a problem is it seems my syslog has stopped writing in mid line and did so nearly a month ago !! i have not heard of people not able to check the syslog when there is an error because there is no syslog to check !!

 

Tried to run a smart report from putty and got "failed:No such device"

 

So before i do anything about the disabled disk what is the view about this? anything i can check etc. I am assuming if i shut down, replace the failed disk (can i check it somehow with it disabled?) with a spare and restart the syslog will start also.

 

 

EDIT: As this is my backup server i wanted to get it on the road quickly and so i stopped the array, removed the disabled disk and put in the spare. Data rebuild is now in progress without issue so far - so it seems at first not to be a tray or cable issue. The syslog has not seen any of this and is still not logging, it is exactly the same as it was before i took any action. Should i restart the server after the rebuild or is there something to check here first about why logging has just stopped (not good to try and fault find if the map is not there to try to follow !)

syslog-2015-05-21.txt

[RobJ]

It looks like it has been rotated.  If messages fill the syslog too big, then it's renamed and a new one is started.  You can check the /var/log folder and see if the original syslog is there, with a variant name.  And since it's in memory, lowmem in v5, it's possible it ran out of memory, blocking further syslog writing.  A reboot will fix it, plus figuring out what is stuffing it so full, and stopping that.  The UPS messages may not be a serious problem, but if they are using up all lowmem, then they *are* a problem.

 

Once a drive is disabled, only a reboot brings it back online, for the array and for a SMART report.  Can't think of anything else to save, except for the original syslog if you find it.

[hilly]

I checked the syslogs. There are 2 older versions noted as syslog_1 and Syslog_2 but these are older, the one in the OP is the most recent but seems it stopped logging very soon after a reboot and has been doing nothing else since then. I currently have the rebuild of my failed drive ongoing but when this is finished i will do a restart and check if logging is working again. Just never heard of the logging stopping like this and a syslog is always asked for to do problem analysis.

 

Strange .....

[RobJ]

Copy off both those syslogs, so they can be zipped and attached here.  I don't relish examining them, as they are full of repeated garbage.  As I said, I think you ran out of memory for them (in their particular section of RAM), as they are in RAM, not on disk.

[hilly]

As the 2 syslogs are older would they still tell you anything? if it was just full memory then i am fine with that as an explination - i am assuming you get no warning your syslog is no longer being written to for whatever reason?. It is when the primary server is down i get so many UPS messages. going to do a conversion of that server to 6.0 shortly and add more storage and work with only that server so the piggy back issue will go away. will then use the backup for just that reason.

 

I have done a restart now the disabled disk has been rebuilt and the the syslog seems to be writing perfectly fine now - could these 2 things be related? The syslog from the restart is attached but it looks fine to my untrained eye and the server is running and all showing green.

syslog-2015-05-22.txt

[RobJ]

Those aren't older syslogs, as older syslogs are lost every boot.  Your syslog is syslog_1 plus syslog_2 plus the syslog piece you attached in first post.  So yes, we want to see them, because with most issues, you want to see the very first time it occurs.  The first occurrence is often by far the most important.

 

An example, a loose SATA cable falls off due to vibration, causing the kernel to disable the non-responsive drive.  That only causes a few (but very important) messages in the syslog, but what may follow is megabytes of repeated error messages involving that drive, "can't access SMART", "read errors", "can't write", "mover errors", etc, etc, etc.  The only thing useful is the very first evidence of non-response, and the kernel disabling the drive.

 

Please make sure you zip these, before attaching.

 

Just a note - you are installing Powerdown 2.06 (the latest is 2.16), but it looks like an older version of apcupsd is then installing Powerdown 1.02, and replacing 2.06.  I would remove both Powerdown and apcupsd and reinstall both, with the latest versions for v5.  Hopefully, the new version would use your previous UPS settings, but to be safe, you may want to make notes of them.

 

Otherwise, your new syslog looks clean!

[hilly]

Thanks for your input and help RobJ but i did a reboot to kick start the syslog and of course now the others are gone - never though to copy them somewhere safe first - Damn, its not like i dont know that bit !!

 

Also thanks for the info on the packages, i have left the systems alone for a long time now and they have kind of got behind or are a bit untidy. I will be clearing everything off anf going to 6.0 very soon - remembering to make a full backup of the flash from each  machine first !! i hope then to use new apps and keep them up to date inside of unraid

 

But for this issue i think i just holed my boat !!

[RobJ]

Also thanks for the info on the packages, i have left the systems alone for a long time now and they have kind of got behind or are a bit untidy. I will be clearing everything off anf going to 6.0 very soon - remembering to make a full backup of the flash from each  machine first !! i hope then to use new apps and keep them up to date inside of unraid

 

Make sure you follow the Upgrading to UnRAID v6 guide and reformat (after backing up!).  Syslinux files have moved out of the root in v6.

[hilly]

Yes, been reading the guide so I know the steps involved and I need things like crashplan and to adopt my current backup so need to learn a bit about dockers, sequence to do things etc.

 

One question is still not clear to me - file system format. Should I use the chance to change? Will it cause me some problems in the future if there is a need to change, it it worth changing the cache disk format etc. There is info on the different ones but no definative statements or clear recommendations. Because of this I dont know whether to do an as is upgrade or get a couple of larger disks and effectively start again .... What is your view?

[RobJ]

One question is still not clear to me - file system format. Should I use the chance to change? Will it cause me some problems in the future if there is a need to change, it it worth changing the cache disk format etc. There is info on the different ones but no definative statements or clear recommendations. Because of this I dont know whether to do an as is upgrade or get a couple of larger disks and effectively start again .... What is your view?

 

There is discussion of that near the bottom of that guide, representing what I feel the current consensus is.  It's new territory yet, and we are all still learning, watching what others are doing and monitoring their success and issues.  If there's no hurry for you, keep reading and watching, learning.  You can upgrade without any format changes, then change when you are ready and sure of what you want.