dirtysanchez Posted November 17, 2015 Share Posted November 17, 2015 Hi all, Server has been rock solid for 3 years, never crashed once. In the span of about 4 weeks, it has now crashed 4 times. Once crashed webGUI becomes unresponsive, unable to access shares, but dockers are still running, and can SSH into box. All attempts to shut down the system via command line fail and I have to power cycle the server, although it's entirely possible I'm not doing it correctly via cmd line. It just crashed again this afternoon, and here is the relevant portion of the syslog which I pulled before power cycling the server. Nov 16 14:09:28 Landfill kernel: general protection fault: 0000 [#1] PREEMPT SMP Nov 16 14:09:28 Landfill kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables kvm_intel kvm vhost_net vhost macvtap macvlan tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod i2c_i801 ahci libahci r8169 mii [last unloaded: md_mod] Nov 16 14:09:28 Landfill kernel: CPU: 1 PID: 13845 Comm: shfs Not tainted 4.1.7-unRAID #3 Nov 16 14:09:28 Landfill kernel: Hardware name: System manufacturer System Product Name/P8H77-I, BIOS 0904 10/15/2012 Nov 16 14:09:28 Landfill kernel: task: ffff88002ce3baa0 ti: ffff880006fd0000 task.ti: ffff880006fd0000 Nov 16 14:09:28 Landfill kernel: RIP: 0010:[<ffffffff811535ac>] [<ffffffff811535ac>] __discard_prealloc+0x98/0xb3 Nov 16 14:09:28 Landfill kernel: RSP: 0018:ffff880006fd3bb8 EFLAGS: 00010246 Nov 16 14:09:28 Landfill kernel: RAX: ffff88011a2af468 RBX: ffff88011a2af440 RCX: d1e528b18ebe2731 Nov 16 14:09:28 Landfill kernel: RDX: 1b1530a073d5c084 RSI: ffff88011a2af440 RDI: ffff880006fd3d00 Nov 16 14:09:28 Landfill kernel: RBP: ffff880006fd3be8 R08: 0000000000001199 R09: 000000000003ad6e Nov 16 14:09:28 Landfill kernel: R10: 00000000ffffffff R11: ffff880153d82e38 R12: ffff880006fd3d00 Nov 16 14:09:28 Landfill kernel: R13: ffff88011a2af4e0 R14: ffff880006fd3d00 R15: 00000000aec273c8 Nov 16 14:09:28 Landfill kernel: FS: 00002b7615618700(0000) GS:ffff88021fa80000(0000) knlGS:0000000000000000 Nov 16 14:09:28 Landfill kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 16 14:09:28 Landfill kernel: CR2: 00000000006fb808 CR3: 0000000208e3a000 CR4: 00000000001406e0 Nov 16 14:09:28 Landfill kernel: Stack: Nov 16 14:09:28 Landfill kernel: ffff88021416b200 ffff880006fd3d00 ffffc90001599000 ffffc900015b91e8 Nov 16 14:09:28 Landfill kernel: ffff880006fd3d00 ffff880214148800 ffff880006fd3c18 ffffffff8115362b Nov 16 14:09:28 Landfill kernel: ffff880006fd3d00 ffff88002ce3baa0 ffffc90001599000 ffffc90001599000 Nov 16 14:09:28 Landfill kernel: Call Trace: Nov 16 14:09:28 Landfill kernel: [<ffffffff8115362b>] reiserfs_discard_all_prealloc+0x44/0x4e Nov 16 14:09:28 Landfill kernel: [<ffffffff8116fe6c>] do_journal_end+0x4e7/0xc78 Nov 16 14:09:28 Landfill kernel: [<ffffffff81170b5c>] journal_end+0xae/0xb6 Nov 16 14:09:28 Landfill kernel: [<ffffffff81161249>] reiserfs_dirty_inode+0x6c/0x7c Nov 16 14:09:28 Landfill kernel: [<ffffffff8104d390>] ? ns_capable+0x3a/0x4f Nov 16 14:09:28 Landfill kernel: [<ffffffff8111cb25>] __mark_inode_dirty+0x2f/0x265 Nov 16 14:09:28 Landfill kernel: [<ffffffff8115cfd7>] reiserfs_setattr+0x262/0x297 Nov 16 14:09:28 Landfill kernel: [<ffffffff810ff408>] ? __sb_start_write+0x9a/0xce Nov 16 14:09:28 Landfill kernel: [<ffffffff8111367a>] notify_change+0x1db/0x2dc Nov 16 14:09:28 Landfill kernel: [<ffffffff811164d8>] ? __mnt_want_write+0x4a/0x61 Nov 16 14:09:28 Landfill kernel: [<ffffffff810fbe72>] chmod_common+0x7a/0x102 Nov 16 14:09:28 Landfill kernel: [<ffffffff810fcd75>] SyS_fchmodat+0x3f/0x77 Nov 16 14:09:28 Landfill kernel: [<ffffffff810fcdc1>] SyS_chmod+0x14/0x16 Nov 16 14:09:28 Landfill kernel: [<ffffffff815f562e>] system_call_fastpath+0x12/0x71 Nov 16 14:09:28 Landfill kernel: Code: 1c 75 bb 0f 0b 85 c0 74 12 48 8b 93 e8 00 00 00 4c 89 ee 4c 89 e7 e8 be 6e 00 00 48 8b 4b 28 44 89 7b 1c 48 8d 43 28 48 8b 53 30 <48> 89 51 08 48 89 0a 48 89 43 28 48 89 43 30 58 5b 41 5c 41 5d Nov 16 14:09:28 Landfill kernel: RIP [<ffffffff811535ac>] __discard_prealloc+0x98/0xb3 Nov 16 14:09:28 Landfill kernel: RSP <ffff880006fd3bb8> Nov 16 14:09:28 Landfill kernel: ---[ end trace 9726fa9b66052d85 ]--- There is nothing in the log prior that appears to be related to the issue. Nothing with the server has changed recently, hardware-wise, and software-wise the only differences in the last few months would be updated plugins and/or dockers. No new plugins or dockers installed in at least 3 months. Possible RAM issue? I'll run a memtest overnight. Beyond that I'm at a loss as far as troubleshooting Linux kernel panics. Any help would be appreciated. Quote Link to comment
lordsiris Posted December 13, 2015 Share Posted December 13, 2015 Hello, I have been having similar issues but failed to get the syslog before hard rebooting (no other method to reboot, just sits at command line not doing anything). I do remember seeing the "PREEMPT SMP" message though in the syslog at some point (idiotic that i didn't save it before power cycling). I have also been using unraid successfully for a number of years without issue. Since moving to 6.x a month or so ago I have had multiple crashes, once about every week or 1.5 weeks. Last was Dec 3rd, then again today. I was a little hesitant in upgrading to 6.x as I really prefer my file server be simply be a file server and do one and one thing really well, which unraid has for years. I upgraded anyway with absolutely no intention of using docker or any of the virtual machine stuff as I have a separate xen server for that. FYI, server is on UPS and has no power issues. Anyone have any thoughts on the crashes? Sanchez have you figured anything out or resolved the issue? Thanks Quote Link to comment
trurl Posted December 13, 2015 Share Posted December 13, 2015 Hello, I have been having similar issues but failed to get the syslog before hard rebooting (no other method to reboot, just sits at command line not doing anything). I do remember seeing the "PREEMPT SMP" message though in the syslog at some point (idiotic that i didn't save it before power cycling). I have also been using unraid successfully for a number of years without issue. Since moving to 6.x a month or so ago I have had multiple crashes, once about every week or 1.5 weeks. Last was Dec 3rd, then again today. I was a little hesitant in upgrading to 6.x as I really prefer my file server be simply be a file server and do one and one thing really well, which unraid has for years. I upgraded anyway with absolutely no intention of using docker or any of the virtual machine stuff as I have a separate xen server for that. FYI, server is on UPS and has no power issues. Anyone have any thoughts on the crashes? Sanchez have you figured anything out or resolved the issue? Thanks Memtest Quote Link to comment
lordsiris Posted December 13, 2015 Share Posted December 13, 2015 Memtest Hello, just completed 4 total passes (2 normal, 2 force smp) with 0 errors on all 4 tests. Any other thoughts? Thanks! Quote Link to comment
dirtysanchez Posted December 17, 2015 Author Share Posted December 17, 2015 I haven't had the crash since I removed the cachedirs plugin. It looks like that's what was causing it for me. Quote Link to comment
lordsiris Posted December 17, 2015 Share Posted December 17, 2015 Unfortunately I don't have that plugin installed, just the UPS "Powerdown Package". Quote Link to comment
warhead Posted December 18, 2015 Share Posted December 18, 2015 I've been experiencing the same crashes. Same type of error in the syslog just before things start to hang. Same hardware that was running fine on 5.05, once I upgraded to 6.1.3 these crashes tend to happen every couple of weeks. I've got Sabnzbd, sickrage and transmission dockers running. Also running a SAS2LP card in my system, been reading that it might have other issues besides the slow parity check speeds under 6.x, not sure if it's related to this though. Already done several passes of memtest with no errors. Quote Link to comment
trurl Posted December 18, 2015 Share Posted December 18, 2015 What filesystems does everyone on this thread have? I know there have been some who have reported they stopped having these issues after they converted all ReiserFS to XFS. And others who report they never have these problem despite keeping ReiserFS. I personally converted everything to XFS except for btrfs cache pretty early and my experience has been that v6 is more stable than v5 ever was. Quote Link to comment
dirtysanchez Posted December 18, 2015 Author Share Posted December 18, 2015 In my system all array drives are ReiserFS, cache drive is XFS. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.