Hey guys,

 

Thought I should post this because I've seen multiple threads/posts referencing hard crashes on rc9, many of which do not contain any diagnostics for us to review.  Logically this is because when a system hard crashes, you can no longer get to the UI which prevents you from obtaining those logs.  Without a log event, diagnosing, reproducing, and resolving a hard crash issue can be neigh impossible.  As such, if you can repeatedly cause a hard crash like this, you will need to attach a monitor and keyboard directly to your unRAID system and reboot the system.  From the console, login with your root user and type the following command:

tail /var/log/syslog -f

When the system reaches a hard lockup, take a picture of what's on the screen and post a new message in this forum with a prefix indicating it is a hard crash.  Also include your diagnostics from a clean boot so we can review your system configuration as well.  Thanks everyone!

How about installing "Nerd Tools" plugin which has Screen. Starting a screen session with

tail /var/log/syslog > /mnt/disk1/syslog.txt

detaching from the Screen and plodding along like normal? If/when it hard crashes again, the file would be available after the boot on /mnt/disk1/syslog.txt?

30 minutes ago, dkerlee said:

How about installing "Nerd Tools" plugin which has Screen. Starting a screen session with

tail /var/log/syslog > /mnt/disk1/syslog.txt

detaching from the Screen and plodding along like normal? If/when it hard crashes again, the file would be available after the boot on /mnt/disk1/syslog.txt?

 

That is definitely an alternate approach, but requires a plugin install + more steps.  In addition, if the crash occurs too quickly, it may not make it to the file.

Jon - For all my hard crashes so far in the 6.4 series,  the console has been inaccessible with keyboard/monitor

Edited September 27, 20178 yr by Dephcon

Found this in FAQ from user @Squid 

 

 

Edited October 2, 20178 yr by dkerlee

Is it possible to add more standard system utility with standard delivery, e.g. strace, gstack, perf, iostat or even dstat?

Because if you work with Linux, these standard tools are very useful to troubleshoot, or let's say, take necessary diagnostic information for such kind of issue when it occurring.

Edited September 30, 20178 yr by rowid_alex

There's a nice plugin you can get called Geek Tools or something. In there are loads of standard linux CLI that you might want. Nerdpack. Support thread but saw some of those you mentioned in there.

21 hours ago, dkerlee said:

There's a nice plugin you can get called Geek Tools or something. In there are loads of standard linux CLI that you might want. Nerdpack. Support thread but saw some of those you mentioned in there.

 

Thanks a lot! I followed spaceinvaderone's video so I installed neadpack already. But forgot I have to explicitly switch them on, if I need some of these tools. There is no gstack, but other tools indeed much help. Thanks a again for the reminder!

anyway to gather this info if the console is also crashed/acting up? For instance, I am on the latest 6.4 RC, the web ui is currently down and the console lets me type my user and pass in and then does nothing.

2 minutes ago, wickedathletes said:

anyway to gather this info if the console is also crashed/acting up? For instance, I am on the latest 6.4 RC, the web ui is currently down and the console lets me type my user and pass in and then does nothing.

If you can't even get into the console, I'd make another unRaid flash drive, boot off that and see where you can go from there.

Just now, dkerlee said:

If you can't even get into the console, I'd make another unRaid flash drive, boot off that and see where you can go from there.

 

I assume I just crashed, just wanted to know if there was a way to find out why since a reboot will surely bring it back online. I'd say on average this happens to me 5-6 times a year.

 

So, I left putty open for about 40 minutes and it finally loaded... what the heck would cause that?

@wickedathletes 40 minutes is a weird amount of time. I bet some process is waiting for something to finish, that hangs it all up. When you look at your logs, look at the time stamps. Find that 40 minute gap, and see what the last few entries right before it were. 5-6 times a year ain't horrible, but it shouldn't happen.

1 hour ago, dkerlee said:

@wickedathletes 40 minutes is a weird amount of time. I bet some process is waiting for something to finish, that hangs it all up. When you look at your logs, look at the time stamps. Find that 40 minute gap, and see what the last few entries right before it were. 5-6 times a year ain't horrible, but it shouldn't happen.

 

I am 75% sure i have 2 faulting SATA ports causing my lock ups. Ive swapped my drives around and the same ports fail. Of course one is on the mobo and one is on my pcie sata card. Since everything I have read says it should not cause harm to the drives I am just biding my time until I upgrade my system in early 2018 (my system is maxed on storage slots anyways.

 

kind of off topic, but are their ways to test the USB stick? That thing has been in my server since 3.x.  Every time I buy a replacement it ends up being re-purposed haha.

Edited October 11, 20178 yr by wickedathletes

@wickedathletes Testing a USB drive? Nope. You're on your own there. I did read here somewhere that when booting unRaid, it only touches the USB once while it boots, then everything is ran from memory. So that's very few reads/writes to the USB drive, and an old usb 2.0 won't impact speed except boot.

Nice job finding those hardware fubars. You could look forward to that next build and get a part that you can use both now, and then. Like this: https://www.newegg.com/Product/Product.aspx?Item=N82E16816101792. it's $112 plus another 15-30 for a pair of mini-sas > 4xsata cables. It will get you 8 new sata ports. If you're planning on keeping those older hard drives for the new build, you're going to need many sata ports. I've had one of those SAS2LP-MV8, and it's been rock solid, for years.

At bootup the unRaid files are read. When settings or drive configurations are changed, config files are updated. Plugins are installed on the flash disk. When array is started, a single file called config/super.dat is updated. When array is stopped, that file is updated again. When diagnostics is run, the file is written to flash. When preclear runs, smart reports are stored on the flash. Some plug-ins can use the flash to maintain state.

 

That pretty much covers everything I know that causes USB accesses.

 

Btw, since a few files in the config collar are updated over and over, I periodically copy the config folder (with array stopped) and rename the config folder to config_old or something, and the rename the new copy to the name config. Not deleting old one prevents the sectors those files used from being reused. Provides a backup if ever needed. And forces the writes onto other sectors on the USB. Probably completely unnecessary, but couldn't hurt. I've probably got 10 old config directories, and still plenty of real estate even on my modest 4G USB.

19 hours ago, SSD said:

Btw, since a few files in the config collar are updated over and over, I periodically copy the config folder (with array stopped) and rename the config folder to config_old or something, and the rename the new copy to the name config. Not deleting old one prevents the sectors those files used from being reused. Provides a backup if ever needed. And forces the writes onto other sectors on the USB. Probably completely unnecessary, but couldn't hurt. I've probably got 10 old config directories, and still plenty of real estate even on my modest 4G USB.

Sounds like a great idea to me.  You could probably do this once a month with dated renames of the config folder and still have lots of space after 5 years.

In my opinion there should be something like this included by default in unRAID. Crashes sometimes are unpredictable, and if a system crashes for a user, enabling the logging after that might not do that much good because the crash may not even happen for another few months. I had a system crash on me twice (luckily no data was lost), after that I enabled logging, and what do you know, it never crashed after that... To this day I don't know for sure what caused the crashes.

@Krzaku I'm with ya on that one, but there was some banter (I forget here above, or on a similar hijacked thread) that the nature of crashes will frequently keep anything helpful from being written to an actual log file. You can capture some stuff, which I think they should! But it seems like the important stuff is only on the physical screen - total pain in the ass for headless servers in the basement!

this is my 6th or so KP with the 6.4 series, finally got something captured from console.

 

Edited October 27, 20178 yr by Dephcon

I've been having hard crashes constantly - always within an hour of starting the machine, usually much less.

tower-diagnostics-20171030-1937.zip

Start with doing a memory test and verify your RAM is okay.

 

12 hours ago, bonienl said:

Start with doing a memory test and verify your RAM is okay.

 

 

I should have mentioned I did a 1 pass memory test but just to be sure I ran another 5 passes overnight with no issues. The issue seems to be RCU callbacks on Ryzen (discussion in the main rc10b thread)

My dad's server crashes every 1 or 2 days. Usually it just locks up. No errors on the console, it just becomes totally unresponsive. Lose access by SSH and samba, but can sometimes type something into the terminal, but then it freezes completely when entering the command.

 

It's been doing this for a while. I even replaced the MoBo, RAM, CPU, and a couple disks (and ran RAM tests, with no errors). Same thing. If I can ever see some sort of error or anything useful in a log, I'll post it.

@BBoYTuRBo is it a Ryzen processor by chance? If it is, you need a pre-release for stability to support that Ryzen processor. 

I went 30 days without a crash running on RC9, updated to 10, waited a few days and then turned on bridging for one of my dockers (plex) instead of host mode. It ran for about a day, then crashed shortly after applying an update to the docker this morning. Earlier in the beta I had been running bridge mode on all of Dockers and would crash constantly. I have disabled this mode again and will advise if another crash occurs. 

 

I hope this helps.