May 20, 201511 yr Per dgaschk's request, I've opened a separate ticket. Syslog attached (syslog-2015-05-13.txt) which captures the Disabling IRQ 16 - on unRAID v6b15 WITH the AOC SAS2LP-MV8 installed and all the drives running. Yesterday, I stripped the system down to the following: > Fresh install on new USB (SanDisk Ultra Fit 32gb) running unRAID v6 RC 2.0 > Motherboard Asus P8Z68-V Pro > Motherboard NIC activated > Removed pro/1000 card > Removed Supermicro AOC-SAS2LP-MV8 > Using 3 Different unused disks 1TB for Parity, 500GB and 1TB for data > Not running unmenu yet. System loads without issue (latest syslog20150518.txt attached). (had some problems with the USB in USB 2.0 slots, but resolved when I put it in USB 3.0 slot (see http://lime-technology.com/forum/index.php?topic=40009.msg375704#msg375704)) Formatted data disks, Parity has completed with no errors. The system ran EXACTLY 24 hours (as far as I can tell to the second!), then I get IRQ 16 (Nobody Cares). Its not crashed yet, so I captured the syslog (attached syslog-nobody cares.txt). I'll leave the system running for now until it either crashes or someone recommends a next step. ===== Update - the forum only allows 192kb uploads. So I'll post the other two syslogs in following posts. syslog-2015-05-13_0459.txt
May 20, 201511 yr Author Update, got a few Segfaults error 4...starting from where the previous syslog left off... May 19 20:35:10 HunterNAS kernel: Disabling IRQ #16 May 19 20:58:51 HunterNAS emhttp: /usr/bin/tail -n 42 -f /var/log/syslog 2>&1 May 20 01:03:28 HunterNAS kernel: sh[27181]: segfault at 56dee3801084 ip 00002b6f71be3f07 sp 00007ffd1181db90 error 4 in ld-2.17.so[2b6f71bdd000+23000] May 20 01:03:29 HunterNAS kernel: sh[27182]: segfault at 5667a85a1084 ip 00002b33d42b3f07 sp 00007ffe3811a400 error 4 in ld-2.17.so[2b33d42ad000+23000] May 20 01:03:30 HunterNAS kernel: sh[27183]: segfault at 563f56497084 ip 00002b1fab22ef07 sp 00007fff067293c0 error 4 in ld-2.17.so[2b1fab228000+23000] May 20 01:03:31 HunterNAS kernel: sh[27184]: segfault at 55b49dd63084 ip 00002ada4ee94f07 sp 00007fff6f81cb50 error 4 in ld-2.17.so[2ada4ee8e000+23000] May 20 01:03:32 HunterNAS kernel: sh[27186]: segfault at 55ab76f81084 ip 00002ad5bb7a3f07 sp 00007fff0c627c70 error 4 in ld-2.17.so[2ad5bb79d000+23000] May 20 01:03:33 HunterNAS kernel: sh[27188]: segfault at 56307a947084 ip 00002b183d486f07 sp 00007fff44235aa0 error 4 in ld-2.17.so[2b183d480000+23000] May 20 01:03:34 HunterNAS kernel: sh[27190]: segfault at 5675a79b1084 ip 00002b3ad3cbbf07 sp 00007ffc9709b330 error 4 in ld-2.17.so[2b3ad3cb5000+23000] May 20 01:03:35 HunterNAS kernel: sh[27191]: segfault at 55e62dcbb084 ip 00002af316e40f07 sp 00007ffd5554cef0 error 4 in ld-2.17.so[2af316e3a000+23000] May 20 01:03:36 HunterNAS kernel: sh[27192]: segfault at 56ca1f225084 ip 00002b650f8f5f07 sp 00007ffe7d9f87c0 error 4 in ld-2.17.so[2b650f8ef000+23000] May 20 01:03:37 HunterNAS kernel: sh[27193]: segfault at 561bc797d084 ip 00002b0de3ca1f07 sp 00007ffd17581360 error 4 in ld-2.17.so[2b0de3c9b000+23000] May 20 01:03:38 HunterNAS kernel: sh[27194]: segfault at 56ae44327084 ip 00002b5722176f07 sp 00007ffe8f531eb0 error 4 in ld-2.17.so[2b5722170000+23000] May 20 01:03:39 HunterNAS kernel: sh[27195]: segfault at 570a719f5084 ip 00002b8538cddf07 sp 00007ffe674e2d00 error 4 in ld-2.17.so[2b8538cd7000+23000] May 20 01:03:40 HunterNAS kernel: sh[27196]: segfault at 5577ecf4d084 ip 00002abbf6789f07 sp 00007ffc670abc00 error 4 in ld-2.17.so[2abbf6783000+23000] May 20 01:03:41 HunterNAS kernel: sh[27197]: segfault at 56cb325f5084 ip 00002b65992ddf07 sp 00007ffc5b01f490 error 4 in ld-2.17.so[2b65992d7000+23000] May 20 01:03:42 HunterNAS kernel: sh[27198]: segfault at 56a1d4d29084 ip 00002b50ea677f07 sp 00007ffed7729780 error 4 in ld-2.17.so[2b50ea671000+23000] May 20 01:03:43 HunterNAS kernel: sh[27199]: segfault at 5687571e9084 ip 00002b43ab8d7f07 sp 00007ffc5fec1ba0 error 4 in ld-2.17.so[2b43ab8d1000+23000] May 20 01:03:44 HunterNAS kernel: sh[27200]: segfault at 5743c170d084 ip 00002ba1e0b69f07 sp 00007ffd298fb130 error 4 in ld-2.17.so[2ba1e0b63000+23000] May 20 01:03:45 HunterNAS kernel: sh[27201]: segfault at 56624c1b9084 ip 00002b31260bff07 sp 00007fffc4658b40 error 4 in ld-2.17.so[2b31260b9000+23000] May 20 01:03:46 HunterNAS kernel: sh[27202]: segfault at 56fdfb899084 ip 00002b7efdc2ff07 sp 00007ffc1ab2a3e0 error 4 in ld-2.17.so[2b7efdc29000+23000] May 20 01:03:47 HunterNAS kernel: sh[27203]: segfault at 573dc769b084 ip 00002b9ee3b30f07 sp 00007ffe43482090 error 4 in ld-2.17.so[2b9ee3b2a000+23000] May 20 01:03:48 HunterNAS kernel: sh[27204]: segfault at 5555587f9084 ip 00002aaaac3dff07 sp 00007ffc53b52e60 error 4 in ld-2.17.so[2aaaac3d9000+23000] May 20 01:03:49 HunterNAS kernel: sh[27205]: segfault at 561c17bd5084 ip 00002b0e0bdcdf07 sp 00007ffd4a371c40 error 4 in ld-2.17.so[2b0e0bdc7000+23000] May 20 01:03:50 HunterNAS kernel: sh[27206]: segfault at 55e628537084 ip 00002af31427ef07 sp 00007ffdfdd8dff0 error 4 in ld-2.17.so[2af314278000+23000] May 20 01:03:51 HunterNAS kernel: sh[27208]: segfault at 56556ec95084 ip 00002b2ab762df07 sp 00007ffe2c6072c0 error 4 in ld-2.17.so[2b2ab7627000+23000] May 20 01:03:52 HunterNAS kernel: sh[27209]: segfault at 558c21709084 ip 00002ac610b67f07 sp 00007ffeb1f190c0 error 4 in ld-2.17.so[2ac610b61000+23000] May 20 01:03:53 HunterNAS kernel: sh[27210]: segfault at 57497b167084 ip 00002ba4bd896f07 sp 00007ffc486f0690 error 4 in ld-2.17.so[2ba4bd890000+23000] May 20 01:03:54 HunterNAS kernel: sh[27211]: segfault at 55f9eeb13084 ip 00002afcf756cf07 sp 00007ffd292804b0 error 4 in ld-2.17.so[2afcf7566000+23000] May 20 01:03:55 HunterNAS kernel: sh[27213]: segfault at 5651bc8a1084 ip 00002b28de433f07 sp 00007ffd74f5b9c0 error 4 in ld-2.17.so[2b28de42d000+23000] May 20 01:03:56 HunterNAS kernel: sh[27214]: segfault at 571f12f55084 ip 00002b8f8978df07 sp 00007ffde67252c0 error 4 in ld-2.17.so[2b8f89787000+23000] May 20 01:03:57 HunterNAS kernel: sh[27215]: segfault at 565de22b9084 ip 00002b2ef113ff07 sp 00007ffdc9b00880 error 4 in ld-2.17.so[2b2ef1139000+23000] May 20 01:03:58 HunterNAS kernel: sh[27216]: segfault at 564c1ecf7084 ip 00002b260f65ef07 sp 00007ffe5598e440 error 4 in ld-2.17.so[2b260f658000+23000] May 20 01:03:59 HunterNAS kernel: sh[27217]: segfault at 55613478b084 ip 00002ab09a3a8f07 sp 00007ffde743c4c0 error 4 in ld-2.17.so[2ab09a3a2000+23000] May 20 01:04:00 HunterNAS kernel: sh[27218]: segfault at 562ee21f5084 ip 00002b17710ddf07 sp 00007ffccc409f40 error 4 in ld-2.17.so[2b17710d7000+23000] May 20 01:04:01 HunterNAS kernel: sh[27219]: segfault at 565a4f709084 ip 00002b2d27b67f07 sp 00007ffd724f7a30 error 4 in ld-2.17.so[2b2d27b61000+23000] May 20 01:04:02 HunterNAS kernel: sh[27220]: segfault at 55c42cad3084 ip 00002ae21654cf07 sp 00007ffe05f4d160 error 4 in ld-2.17.so[2ae216546000+23000] May 20 01:04:03 HunterNAS kernel: sh[27221]: segfault at 561764b9b084 ip 00002b0bb25b0f07 sp 00007ffe85017d70 error 4 in ld-2.17.so[2b0bb25aa000+23000] May 20 01:04:04 HunterNAS kernel: sh[27223]: segfault at 568caa95f084 ip 00002b4655492f07 sp 00007fff36c27a50 error 4 in ld-2.17.so[2b465548c000+23000]
May 20, 201511 yr The system ran EXACTLY 24 hours (as far as I can tell to the second!), then I get IRQ 16 (Nobody Cares). Its not crashed yet, so I captured the syslog (attached syslog-nobody cares.txt). I care mate, you've really had some problems here, hope you get it sorted out.
May 20, 201511 yr Can't look at it at the moment, sorry, hoping someone else can... Otherwise tomorrow... Another Afib episode is wiping me out, killing my concentration, and full syslog analysis is beyond me for now. Since I can't work 'til it's over, I'm trying to help where I can, in little easy bits and drabs.
May 20, 201511 yr Can't look at it at the moment, sorry, hoping someone else can... Otherwise tomorrow... Another Afib episode is wiping me out, killing my concentration, and full syslog analysis is beyond me for now. Since I can't work 'til it's over, I'm trying to help where I can, in little easy bits and drabs. Aren't you on anything for rate control? Digoxin, B-Blocker, Amiodarone or something like that? Not good if you keep having episodes.
May 20, 201511 yr Aren't you on anything for rate control? Digoxin, B-Blocker, Amiodarone or something like that? Not good if you keep having episodes. Sotalol limits the highest ventricle BPM's, makes it safer but doesn't fix anything, might make them slightly shorter, hard to tell. They're still roughly a weekly occurrence. Recently, I got my first insurance card in a very long time, so am starting the process toward an ablation, we'll see. Thanks for the concern! And apologies to Jeff for the OT tangent, my fault. Edit: Episode already over! Working on this ...
May 20, 201511 yr Update - the forum only allows 192kb uploads. So I'll post the other two syslogs in following posts. Just a reminder, we generally prefer zipped syslogs (per the Need help guide), for the huge space savings and the integrity check and the original capture date. The Syslog icon on the Tools page has a Download button that provides a zipped and downloaded syslog, ready for attaching.
May 20, 201511 yr There are 2 issues, probably not related, but not impossible. One is the "IRQ 16: nobody cared" and the other is the segfaults involving ld.so. One big thing, great in some ways, is that this build exonerates the SAS card and the network cards! The first thing I checked was whether the IRQ issue occurred exactly 24 hours later, but while it's close, it missed by about 15 minutes, so probably not a factor. Good thinking though. It did occur shortly after a bunch of segfaults, so it's not impossible there's a connection. I haven't looked back in your older syslogs to see if that's always the case. But if it was connected, then fixing the segfaults might fix everything. Here are the first segfaults - May 18 20:54:51 HunterNAS-6 login[1491]: ROOT LOGIN on '/dev/tty1' ... May 18 20:57:04 HunterNAS-6 kernel: awk[4298]: segfault at 7ffe7d2568f3 ip 00002aeae67caea7 sp 00007ffebca01f10 error 6 in ld-2.17.so[2aeae67c0000+23000] May 18 20:57:04 HunterNAS-6 kernel: sleep[4299]: segfault at 7ffd2d15e893 ip 00002b5cbe8c2ea7 sp 00007ffd6c909eb0 error 6 in ld-2.17.so[2b5cbe8b8000+23000] May 18 20:57:04 HunterNAS-6 kernel: awk[4300]: segfault at 7ffcab727613 ip 00002b09151d7ea7 sp 00007ffceaed2c30 error 6 in ld-2.17.so[2b09151cd000+23000] As best as I can remember, segfaults involving a library like the above were a problem earlier with mixed plugins with conflicting dependency requirements. One plugin (or the system) would install a library, then a different plugin would install a different version of the library, leaving the first plugin or system module to call a library that was not linked the same. That doesn't seem possible in your case, because there's no evidence of any plugins loaded. But then I noticed that the first segfaults occurred just 2 minutes after you logged in. Can you recall everything you ran about 2 minutes after logging into the console? I suspect you executed something that's not compatible, perhaps 32 bit from v5, or perhaps a tool you like. This first occurrence of segfaults involve an awk call, then a sleep, then another awk call. I have to go to work shortly ...
May 21, 201511 yr Author Interesting analysis. In this build there are no plugins, no unmenu, vanilla. I'm not sophisticated enough at this point to use any favorite tools other than things like ethtool, ls, cp, mcedit, etc...all basic stuff... And in this build I've purposely stayed away from anything that isn't core. The only thing I noticed in the older build is that failures were very common a few minutes after the parity check completed, which I found interesting. I'd be happy to try anything to force the failure...
May 21, 201511 yr Author I ran a tail -f to try to capture a panic as it happened. Not sure I captured anything valuable, but syslog attached. Here's what was on the screen... http://my.jetscreenshot.com/12412/20150521-9xf6-181kb Hope it helps. Not sure what I can do next... syslog20150520.txt
Archived
This topic is now archived and is closed to further replies.