RFehr

@RobJ - thx a bunch. Definitely a hardware problem - likely do to our 'balmy' new ambient temps. Lifted the lid and performed a standard server BJ, re-seated SAS card and cable - issue gone. However, no reiserfs partition detected on sdd - but the physical disk looks fine. Partition table moved/bent - looks like I need to dust off testdisk (or the "screw it" option and rebuild from parity). Thanks again.

Folks, Here in Calgary, we have a short summer - or at least this year we have. The days of +25C with sunshine seem to be numbered - hence I'm dumping a log file here without really even trying to look at it myself. Like... not at all. So... I do feel guilty - I really hate dumping problems over the wall unless I've at least put in an honest effort. But, I'm confident that this feeling will pass with the assistance of a cold pint or two. In short, b4/Xen with a single Arch VM running all plugins has been crashing every day or two - BRILLIANTLY stable prior to about a week ago. I've changed nothing over this period. All pluggins except Plex Media Server are running in the Arch VM (PMS performs better with direct disk access for transcoding). Other than that, it's "off the rack" b4/Xen. See attached. UnRAID_log.zip

I'm not sure if enough kudos has been heft upon IconicBadger et. al. so here's some more: Excellent info, guides, youtube videos (OMG, WTF, etc.), and support in this and other threads. I went from a standing start with 5.0 avec PMS/SB/CP/Sab to 6.0b3 with PMS and an Arch VM running SB/CP/Sab in a few fun and informative hours with narry a 'hard' issue. Not completely challenge free - but all issues definitely in the "fun" v.s. "frustrating" category. Also have a Win7 VM just for kicks (not sure exactly what it might be good for as yet - but it's nice to have a windows OS around to look at - even if just for historical "remember when" moments). Unlike many, I've had no issues nfs - perhaps my meager 600 file library is not as taxing as others. Migrating SB/CP databases (v.s. fresh install/config) has thrown up the most significant issues - setting up 6b3 and Arch was completely trouble free. There is certainly some fine tuning to do on this front - and to be honest, I'm not sure that Arch is the right distro for those not very linux savvy - but the move to 6b3+archVM has been far less painfull than the struggle to get 'mandatory' apps running natively in unraid back in the 4.x days. So... hats off. Point me to the donate button.

First off, don't use port forwarding to anything that you deem important - regardless of whether the "important" resource is "password protected" or not. Instead, establish a VPN to your home network (ssl or IPSec) and manage LAN resources via the VPN. Much simpler to administer and much more secure. Port forwarding without deep knowledge of the target and ongoing diligence is just a bad idea.

I'm a little short on time but I thought I'd dump what I do know into a post just incase any of you are running into this issue as well. In short, the net-snmp 5.5 package that I grabbed (referenced in the mrtg unmenu package) does not appear to have been compiled with lmsensors module support - and hence does not work. root@UnRaid:/usr/bin# sensors -v sensors version 3.1.2 with libsensors version 3.1.2 root@UnRaid:/usr/bin# sensors k8temp-pci-00c3 Adapter: PCI adapter Core0 Temp: +33.0 C Core1 Temp: +41.0 C k8temp-pci-00cb Adapter: PCI adapter Core0 Temp: +45.0 C Core1 Temp: +50.0 C root@UnRaid:/usr/bin# net-snmp-config --version 5.5 root@UnRaid:/usr/bin# snmpwalk -v 2c -c public localhost lmSensors LM-SENSORS-MIB::lmSensors = No Such Object available on this agent at this OID root@UnRaid:/usr/bin# net-snmp-config --snmpd-module-list | grep -e [ss]ensors root@UnRaid:/usr/bin# There is also an (old - but perhaps still around) debian bug related to net-snmp and lmsensors discussed here http://web.archiveorange.com/archive/v/4feNTcfPu0Y6uzfgPo4z I don't have a (good) unraid build environment set up at the moment - and to be honest, I've not looked around for any other net-snmp package other than the one referenced in the mrtg install.

I'd managed to get nagios (2) mostly working in the 4.x days and in general, I find that I don't really care about the web server - I'm far happier with a really good, super tiny, super reliable monitoring platform that can let me know when something goes wrong. Frankly, other than usual old disk problems, my production UnRAID server is one of the most reliable systems I have - hence it's a great home for nagios. I did let my nagios efforts fall to the side when I started delving into the 5.x betas. Probably time to pick it back up again. IMO, first step is a simple plg with just nagios core and no web server. Real men use the command line. Weebotech - had you started any sort of plg effort or just install based on the slackware scripts?

Isn't the alerting repeat configurable configurable? As is the ability to configure service dependencies, so that if it stops responding to ping you don't get network service alerts? It is possible this is now an option. I have not used Nagios in several years. Was a huge annoyance to the IT team where I worked about 5 years years ago. Frequency of Nagios notifications is COMPLETELY configurable - as is everything about nagios. The only valid criticism of nagios is that the learning curve TO DO THINGS RIGHT is pretty steep. Part of the problem is that it is very easy to get nagios to "just work", but it takes a lot of planning and thinking about how to get nagios to perform exactly as you'd like it to. For example, for most "issues", I have nagios inform me ONCE when it happens - as long as it's within reasonable hours. If the issue is not resolved, I'll get a reminder email ONCE every 24hrs (again, within reasonable hours - i.e. not when I'm trying to sleep). If I've ignored reminder emails for 3 consecutive days, the issue is escalated (i.e. someone else gets the email). Also, the service dependancy construction is very important - i.e. if 10 different monitored devices are connected to a single ethernet switch, I get ONE notification if the switch is down - not 11 notifications that all connected devices AND the switch is down. Nagios is vastly powerful and configurable - but it needs a substantial intellectual investment up front.

I too am having what appears to be a related problem - but with CrashPlan backups (to a cache drive). This has been happening for awhile - finally narrowed the problem down to the CrashPlan backup dir, and to the backup of one particular machine - an MBP. Is this the same issue?? Nov 3 13:43:17 UnRaid kernel: mdcmd (130): spindown 2 Nov 3 13:43:17 UnRaid kernel: mdcmd (131): spindown 4 Nov 3 13:43:46 UnRaid kernel: mdcmd (132): spindown 0 Nov 3 14:16:22 UnRaid kernel: mdcmd (133): spindown 7 Nov 3 14:16:42 UnRaid kernel: mdcmd (134): spindown 3 Nov 3 14:16:42 UnRaid kernel: mdcmd (135): spindown 5 Nov 3 14:16:47 UnRaid kernel: mdcmd (136): spindown 1 Nov 3 14:16:47 UnRaid kernel: mdcmd (137): spindown 6 Nov 3 14:23:59 UnRaid kernel: scsi_verify_blk_ioctl: 14 callbacks suppressed Nov 3 14:23:59 UnRaid kernel: hdparm: sending ioctl 2285 to a partition! Nov 3 14:23:59 UnRaid last message repeated 5 times Nov 3 14:23:59 UnRaid kernel: smartctl: sending ioctl 2285 to a partition! Nov 3 14:23:59 UnRaid last message repeated 3 times Nov 3 15:05:44 UnRaid avahi-daemon[10911]: Invalid response packet from host 192.168.250.211. Nov 3 15:05:44 UnRaid avahi-daemon[10911]: Invalid response packet from host 192.168.250.205. Nov 3 15:21:08 UnRaid emhttp: shcmd (70): /usr/local/sbin/mover |& logger & Nov 3 15:21:08 UnRaid logger: mover started Nov 3 15:21:08 UnRaid logger: moving Data/ Nov 3 15:21:08 UnRaid logger: ./Data/Backups/507568011094850252/cpbp Nov 3 15:21:09 UnRaid logger: .d..t...... ./ Nov 3 15:21:17 UnRaid logger: rsync: get_xattr_names: llistxattr("Data",1024) failed: Software caused connection abort (103) Nov 3 15:21:17 UnRaid logger: .d..t...... Data/ Nov 3 15:21:17 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:17 UnRaid logger: rsync: failed to set times on "/mnt/user0/Data": Transport endpoint is not connected (107) Nov 3 15:21:17 UnRaid logger: cd+++++++++ Data/Backups/ Nov 3 15:21:17 UnRaid logger: rsync: recv_generator: mkdir "/mnt/user0/Data/Backups" failed: Transport endpoint is not connected (107) Nov 3 15:21:17 UnRaid logger: *** Skipping any contents from this failed directory *** Nov 3 15:21:17 UnRaid kernel: shfs[11046]: segfault at 8000000 ip b74b668c sp b740e068 error 4 in libc-2.11.1.so[b7440000+15c000] Nov 3 15:21:18 UnRaid logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Nov 3 15:21:18 UnRaid logger: ./Data/Backups/507568011094850252/cprp Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr(".",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d.......a. ./ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_set_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_delete_def_file(.): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr("Data",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d..t....a. Data/ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: failed to set times on "/mnt/user0/Data": Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: cd+++++++++ Data/Backups/ Nov 3 15:21:18 UnRaid logger: rsync: recv_generator: mkdir "/mnt/user0/Data/Backups" failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: *** Skipping any contents from this failed directory *** Nov 3 15:21:18 UnRaid logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Nov 3 15:21:18 UnRaid logger: ./Data/Backups/507568011094850252/cpfmf Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr(".",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d.......a. ./ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_set_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_delete_def_file(.): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr("Data",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d..t....a. Data/ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: failed to set times on "/mnt/user0/Data": Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: cd+++++++++ Data/Backups/ Nov 3 15:21:18 UnRaid logger: rsync: recv_generator: mkdir "/mnt/user0/Data/Backups" failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: *** Skipping any contents from this failed directory *** Nov 3 15:21:18 UnRaid logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Nov 3 15:21:18 UnRaid logger: ./Data/Backups/507568011094850252/cphdf Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr(".",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d.......a. ./ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_set_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_delete_def_file(.): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr("Data",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d..t....a. Data/ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: failed to set times on "/mnt/user0/Data": Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: cd+++++++++ Data/Backups/ Nov 3 15:21:18 UnRaid logger: rsync: recv_generator: mkdir "/mnt/user0/Data/Backups" failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: *** Skipping any contents from this failed directory *** Nov 3 15:21:18 UnRaid logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Nov 3 15:21:18 UnRaid logger: ./Data/Backups/507568011094850252/cpfmfp Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr(".",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d.......a. ./ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_set_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_delete_def_file(.): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr("Data",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d..t....a. Data/ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: failed to set times on "/mnt/user0/Data": Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: cd+++++++++ Data/Backups/ Nov 3 15:21:18 UnRaid logger: rsync: recv_generator: mkdir "/mnt/user0/Data/Backups" failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: *** Skipping any contents from this failed directory *** Nov 3 15:21:18 UnRaid logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Nov 3 15:21:18 UnRaid logger: ./Data/Backups/507568011094850252/cpfmfx Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr(".",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d.......a. ./ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_set_file(., ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: set_acl: sys_acl_delete_def_file(.): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: get_xattr_names: llistxattr("Data",1024) failed: Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: .d..t....a. Data/ Nov 3 15:21:18 UnRaid logger: rsync: get_acl: sys_acl_get_file(Data, ACL_TYPE_ACCESS): Transport endpoint is not connected (107) Nov 3 15:21:18 UnRaid logger: rsync: failed to set times on "/mnt/user0/Data": Transport endpoint is not connected (107) --- more of the same snipped --- Nov 3 15:21:35 UnRaid logger: rsync: ERROR: cannot stat destination "/mnt/user0/": Transport endpoint is not connected (107) Nov 3 15:21:35 UnRaid logger: rsync error: errors selecting input/output files, dirs (code 3) at main.c(555) [Receiver=3.0.7] Nov 3 15:21:36 UnRaid logger: rsync: connection unexpectedly closed (9 bytes received so far) [sender] Nov 3 15:21:36 UnRaid logger: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7] Nov 3 15:21:36 UnRaid logger: skipping apps/ Nov 3 15:21:36 UnRaid logger: mover finished Nov 3 15:24:48 UnRaid kernel: BUG: unable to handle kernel paging request at 73750075 Nov 3 15:24:48 UnRaid kernel: IP: [<c108326c>] __kmalloc_track_caller+0xb2/0xf2 Nov 3 15:24:48 UnRaid kernel: *pdpt = 0000000033f25001 *pde = 0000000000000000 Nov 3 15:24:48 UnRaid kernel: Oops: 0000 [#1] SMP Nov 3 15:24:48 UnRaid kernel: Modules linked in: md_mod xor sg asus_atk0110 sata_promise r8168(O) i2c_i801 i2c_core ata_piix coretemp hwmon mperf Nov 3 15:24:48 UnRaid kernel: Nov 3 15:24:48 UnRaid kernel: Pid: 3714, comm: Plex DLNA Serve Tainted: G O 3.4.11-unRAID #1 System manufacturer System Product Name/P5QL/EPU Nov 3 15:24:48 UnRaid kernel: EIP: 0060:[<c108326c>] EFLAGS: 00210202 CPU: 1 Nov 3 15:24:48 UnRaid kernel: EIP is at __kmalloc_track_caller+0xb2/0xf2 Nov 3 15:24:48 UnRaid kernel: EAX: 00000000 EBX: f46900c0 ECX: 73750075 EDX: 00009a6b Nov 3 15:24:48 UnRaid kernel: ESI: f4402580 EDI: c14aafc8 EBP: efd23b88 ESP: efd23b70 Nov 3 15:24:48 UnRaid kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Nov 3 15:24:48 UnRaid kernel: CR0: 8005003b CR2: 73750075 CR3: 323ee000 CR4: 000007f0 Nov 3 15:24:48 UnRaid kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Nov 3 15:24:48 UnRaid kernel: DR6: ffff0ff0 DR7: 00000400 Nov 3 15:24:48 UnRaid kernel: Process Plex DLNA Serve (pid: 3714, ti=efd22000 task=ea40b2a0 task.ti=efd22000) Nov 3 15:24:48 UnRaid kernel: Stack: Nov 3 15:24:48 UnRaid kernel: c129c7e2 000004d0 73750075 f46900c0 000004d0 f4402180 efd23ba4 c12a082d Nov 3 15:24:48 UnRaid kernel: 00000000 00000186 f4635b00 00000000 efd23c9c efd23bf8 c129c7e2 ffffffff Nov 3 15:24:48 UnRaid kernel: efd23bd8 ea40b2a0 00000186 f4635b90 f4635b78 ea40b2a0 f4635bf0 00000000 Message from syslogd@UnRaid at Sat Nov 3 15:24:48 2012 ... UnRaid kernel: Process Plex DLNA Serve (pid: 3714, ti=efd22000 task=ea40b2a0 task.ti=efd22000) Nov 3 15:24:48 UnRaid kernel: Call Trace: Nov 3 15:24:48 UnRaid kernel: [<c129c7e2>] ? sock_alloc_send_pskb+0xc5/0x26b Nov 3 15:24:48 UnRaid kernel: [<c12a082d>] __alloc_skb+0x50/0xf5 Nov 3 15:24:48 UnRaid kernel: [<c129c7e2>] sock_alloc_send_pskb+0xc5/0x26b Nov 3 15:24:48 UnRaid kernel: [<c129c996>] sock_alloc_send_skb+0xe/0x10 Nov 3 15:24:48 UnRaid kernel: [<c12c15f6>] __ip_append_data+0x2db/0x6da Message from syslogd@UnRaid at Sat Nov 3 15:24:48 2012 ... UnRaid kernel: Call Trace: Nov 3 15:24:48 UnRaid kernel: [<c12c1e6d>] ip_make_skb+0x75/0xa5 Nov 3 15:24:48 UnRaid kernel: [<c12c2287>] ? ip_append_page+0x3ea/0x3ea Nov 3 15:24:48 UnRaid kernel: [<c12c2287>] ? ip_append_page+0x3ea/0x3ea Nov 3 15:24:48 UnRaid kernel: [<c12dc5f3>] udp_sendmsg+0x4a1/0x6be Nov 3 15:24:48 UnRaid kernel: [<c12c2287>] ? ip_append_page+0x3ea/0x3ea Nov 3 15:24:48 UnRaid kernel: [<c103b132>] ? check_preempt_curr+0x29/0x64 Message from syslogd@UnRaid at Sat Nov 3 15:24:48 2012 ... UnRaid kernel: Stack: Nov 3 15:24:48 UnRaid kernel: [<c10400bf>] ? move_task+0x37/0x3b Nov 3 15:24:48 UnRaid kernel: [<c12e2002>] inet_sendmsg+0x6f/0x78 Nov 3 15:24:48 UnRaid kernel: [<c1299f4f>] sock_sendmsg+0xa8/0xbe Nov 3 15:24:48 UnRaid kernel: [<c1063383>] ? __alloc_pages_nodemask+0xd1/0x4a5 Nov 3 15:24:48 UnRaid kernel: [<c1092440>] ? core_sys_select+0x1f0/0x217 Nov 3 15:24:48 UnRaid kernel: [<c1299fb2>] ? sockfd_lookup_light+0x16/0x44 Nov 3 15:24:48 UnRaid kernel: [<c129a4c2>] sys_sendto+0xb1/0xd0 Nov 3 15:24:48 UnRaid kernel: [<c12997d7>] ? move_addr_to_user+0x39/0x50 Nov 3 15:24:48 UnRaid kernel: [<c129a55f>] ? sys_getpeername+0x64/0x6f Nov 3 15:24:48 UnRaid kernel: [<c131fa7c>] ? _raw_spin_unlock_bh+0xe/0x10 Nov 3 15:24:48 UnRaid kernel: [<c129c20b>] ? release_sock+0xd0/0xd8 Nov 3 15:24:48 UnRaid kernel: [<c129a4f9>] sys_send+0x18/0x1a Nov 3 15:24:48 UnRaid kernel: [<c129b41f>] sys_socketcall+0xc9/0x19c Nov 3 15:24:48 UnRaid kernel: [<c131fbb5>] syscall_call+0x7/0xb Nov 3 15:24:48 UnRaid kernel: [<c1310000>] ? alloc_node_mem_map+0x70/0x8a Nov 3 15:24:48 UnRaid kernel: Code: c1 8b 50 04 8b 08 85 c9 89 4d f0 75 16 8b 7d e8 8b 55 ec 50 89 f0 89 f9 e8 0d ed ff ff 59 89 45 f0 eb 25 8b 46 14 8b 4d f0 8b 3e <8b> 1c 01 8d 4a 01 8b 45 f0 64 0f c7 0f 0f 94 c0 88 c1 fe c9 75 Nov 3 15:24:48 UnRaid kernel: EIP: [<c108326c>] __kmalloc_track_caller+0xb2/0xf2 SS:ESP 0068:efd23b70 Nov 3 15:24:48 UnRaid kernel: CR2: 0000000073750075 Nov 3 15:24:48 UnRaid kernel: ---[ end trace 086405a4cd354af6 ]--- Message from syslogd@UnRaid at Sat Nov 3 15:24:48 2012 ... UnRaid kernel: Code: c1 8b 50 04 8b 08 85 c9 89 4d f0 75 16 8b 7d e8 8b 55 ec 50 89 f0 89 f9 e8 0d ed ff ff 59 89 45 f0 eb 25 8b 46 14 8b 4d f0 8b 3e <8b> 1c 01 8d 4a 01 8b 45 f0 64 0f c7 0f 0f 94 c0 88 c1 fe c9 75 Message from syslogd@UnRaid at Sat Nov 3 15:24:48 2012 ... UnRaid kernel: EIP: [<c108326c>] __kmalloc_track_caller+0xb2/0xf2 SS:ESP 0068:efd23b70 Nov 3 15:24:49 UnRaid kernel: BUG: unable to handle kernel paging request at 73750075 Nov 3 15:24:49 UnRaid kernel: IP: [<c108326c>] __kmalloc_track_caller+0xb2/0xf2 Nov 3 15:24:49 UnRaid kernel: *pdpt = 0000000033f25001 *pde = 0000000000000000 Nov 3 15:24:49 UnRaid kernel: Oops: 0000 [#2] SMP Nov 3 15:24:49 UnRaid kernel: Modules linked in: md_mod xor sg asus_atk0110 sata_promise r8168(O) i2c_i801 i2c_core ata_piix coretemp hwmon mperf Nov 3 15:24:49 UnRaid kernel: Nov 3 15:24:49 UnRaid kernel: Pid: 3709, comm: Plex DLNA Serve Tainted: G D O 3.4.11-unRAID #1 System manufacturer System Product Name/P5QL/EPU Nov 3 15:24:49 UnRaid kernel: EIP: 0060:[<c108326c>] EFLAGS: 00210202 CPU: 1 Nov 3 15:24:49 UnRaid kernel: EIP is at __kmalloc_track_caller+0xb2/0xf2

Presume this is RAM related? Only 2MB in this server. All I did was stop the array. Oct 26 12:29:31 UnRaid emhttp: shcmd (76): /usr/local/sbin/emhttp_event stopping_svcs Oct 26 12:29:31 UnRaid emhttp_event: stopping_svcs Oct 26 12:29:31 UnRaid emhttp: Stop AVAHI... Oct 26 12:29:31 UnRaid emhttp: shcmd (77): /etc/rc.d/rc.avahidaemon stop |& logger Oct 26 12:29:31 UnRaid logger: Stopping Avahi mDNS/DNS-SD Daemon: stopped Oct 26 12:29:31 UnRaid avahi-daemon[20815]: Got SIGTERM, quitting. Oct 26 12:29:31 UnRaid avahi-dnsconfd[20825]: read(): EOF Oct 26 12:29:31 UnRaid avahi-daemon[20815]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.250.219. Oct 26 12:29:32 UnRaid avahi-daemon[20815]: avahi-daemon 0.6.31 exiting. Oct 26 12:29:32 UnRaid emhttp: shcmd (78): /etc/rc.d/rc.avahidnsconfd stop |& logger Oct 26 12:29:32 UnRaid logger: Stopping Avahi mDNS/DNS-SD DNS Server Configuration Daemon: stopped Oct 26 12:29:32 UnRaid emhttp: shcmd (79): rm /etc/avahi/services/afp.service &> /dev/null Oct 26 12:29:32 UnRaid emhttp: shcmd (80): rm /etc/avahi/services/smb.service &> /dev/null Oct 26 12:29:32 UnRaid emhttp: Stop AFP... Oct 26 12:29:32 UnRaid emhttp: shcmd (81): /etc/rc.d/rc.atalk stop |& logger Oct 26 12:29:32 UnRaid emhttp: shcmd (82): ps axc | grep -q rpc.mountd Oct 26 12:29:32 UnRaid emhttp: _shcmd: shcmd (82): exit status: 1 Oct 26 12:29:32 UnRaid emhttp: Stop SMB... Oct 26 12:29:32 UnRaid emhttp: shcmd (83): /etc/rc.d/rc.samba stop |& logger Oct 26 12:29:32 UnRaid emhttp: Spinning up all drives... Oct 26 12:29:32 UnRaid emhttp: shcmd (84): /usr/sbin/hdparm -S0 /dev/sdi &> /dev/null Oct 26 12:29:32 UnRaid kernel: mdcmd (77): spinup 0 Oct 26 12:29:32 UnRaid kernel: mdcmd (78): spinup 1 Oct 26 12:29:32 UnRaid kernel: mdcmd (79): spinup 2 Oct 26 12:29:32 UnRaid kernel: mdcmd (80): spinup 3 Oct 26 12:29:32 UnRaid kernel: mdcmd (81): spinup 4 Oct 26 12:29:32 UnRaid kernel: mdcmd (82): spinup 5 Oct 26 12:29:32 UnRaid kernel: mdcmd (83): spinup 6 Oct 26 12:29:32 UnRaid kernel: mdcmd (84): spinup 7 Oct 26 12:29:32 UnRaid emhttp: Sync filesystems... Oct 26 12:29:32 UnRaid emhttp: shcmd (85): sync Oct 26 12:29:44 UnRaid emhttp: shcmd (86): /usr/local/sbin/emhttp_event unmounting_disks Oct 26 12:29:44 UnRaid emhttp_event: unmounting_disks Oct 26 12:29:59 UnRaid pms: Stopping Plex... Oct 26 12:30:19 UnRaid emhttp: shcmd (87): set -o pipefail ; umount /mnt/user |& logger Oct 26 12:30:19 UnRaid emhttp: shcmd (88): rmdir /mnt/user |& logger Oct 26 12:30:19 UnRaid emhttp: shcmd (89): crontab -c /etc/cron.d -d &> /dev/null Oct 26 12:30:19 UnRaid emhttp: Unmounting disks... Oct 26 12:30:19 UnRaid emhttp: shcmd (90): umount /mnt/disk1 |& logger Oct 26 12:30:19 UnRaid emhttp: shcmd (91): rmdir /mnt/disk1 |& logger Oct 26 12:30:19 UnRaid emhttp: shcmd (92): umount /mnt/disk2 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (93): rmdir /mnt/disk2 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (94): umount /mnt/disk3 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (95): rmdir /mnt/disk3 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (96): umount /mnt/disk4 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (97): rmdir /mnt/disk4 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (98): umount /mnt/disk5 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (99): rmdir /mnt/disk5 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (100): umount /mnt/disk6 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (101): rmdir /mnt/disk6 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (102): umount /mnt/disk7 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (103): rmdir /mnt/disk7 |& logger Oct 26 12:30:20 UnRaid emhttp: shcmd (104): umount /mnt/cache |& logger Oct 26 12:30:21 UnRaid emhttp: shcmd (105): rmdir /mnt/cache |& logger Oct 26 12:30:21 UnRaid emhttp: shcmd (106): /usr/local/sbin/emhttp_event stopping_array Oct 26 12:30:21 UnRaid emhttp_event: stopping_array Oct 26 12:30:21 UnRaid kernel: mdcmd (85): stop Oct 26 12:30:21 UnRaid kernel: md1: stopping Oct 26 12:30:21 UnRaid kernel: md2: stopping Oct 26 12:30:21 UnRaid kernel: md3: stopping Oct 26 12:30:21 UnRaid kernel: md4: stopping Oct 26 12:30:21 UnRaid kernel: md5: stopping Oct 26 12:30:21 UnRaid kernel: md6: stopping Oct 26 12:30:21 UnRaid kernel: md7: stopping Oct 26 12:30:21 UnRaid emhttp: shcmd (107): rmmod md-mod |& logger Oct 26 12:30:21 UnRaid kernel: md: unRAID driver removed Oct 26 12:30:21 UnRaid kernel: BUG: unable to handle kernel paging request at 73750075 Oct 26 12:30:21 UnRaid kernel: IP: [<c11a98c7>] strlen+0xb/0x15 Oct 26 12:30:21 UnRaid kernel: *pdpt = 0000000031c89001 *pde = 0000000000000000 Oct 26 12:30:21 UnRaid kernel: Oops: 0000 [#1] SMP Oct 26 12:30:21 UnRaid kernel: Modules linked in: xor pata_jmicron sg coretemp asus_atk0110 hwmon r8168(O) sata_promise jmicron i2c_i801 i2c_core ata_piix acpi_cpufreq mperf [last unloaded: md_mod] Oct 26 12:30:21 UnRaid kernel: Oct 26 12:30:21 UnRaid kernel: Pid: 7361, comm: rmmod Tainted: G O 3.4.11-unRAID #1 System manufacturer System Product Name/P5QL/EPU Oct 26 12:30:21 UnRaid kernel: EIP: 0060:[<c11a98c7>] EFLAGS: 00210246 CPU: 1 Oct 26 12:30:21 UnRaid kernel: EIP is at strlen+0xb/0x15 Oct 26 12:30:21 UnRaid kernel: EAX: 00000000 EBX: 00000000 ECX: ffffffff EDX: 73750075 Oct 26 12:30:21 UnRaid kernel: ESI: 73750075 EDI: 73750075 EBP: f1a59ec8 ESP: f1a59ec4 Oct 26 12:30:21 UnRaid kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Oct 26 12:30:21 UnRaid kernel: CR0: 8005003b CR2: 73750075 CR3: 27e05000 CR4: 000007f0 Oct 26 12:30:21 UnRaid kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Oct 26 12:30:21 UnRaid kernel: DR6: ffff0ff0 DR7: 00000400 Oct 26 12:30:21 UnRaid kernel: Process rmmod (pid: 7361, ti=f1a58000 task=ee314d80 task.ti=f1a58000) Message from syslogd@UnRaid at Fri Oct 26 12:30:21 2012 ... UnRaid kernel: Process rmmod (pid: 7361, ti=f1a58000 task=ee314d80 task.ti=f1a58000) Oct 26 12:30:21 UnRaid emhttp: shcmd (108): modprobe md-mod super=/boot/config/super.dat slots=24 |& logger Oct 26 12:30:21 UnRaid kernel: Stack: Oct 26 12:30:21 UnRaid kernel: 00000000 f1a59edc c10c2917 f3f5a100 f3f5a850 00000000 f1a59ef8 c10c2b14 Oct 26 12:30:21 UnRaid kernel: c1081f49 73750075 f3f5a100 f3f5a100 f751a800 f1a59f0c c10c2c55 73750075 Oct 26 12:30:21 UnRaid kernel: 00000000 f8538870 f1a59f24 c10c449d f85388b8 f8538870 f46d6ca8 000000a8 Oct 26 12:30:21 UnRaid kernel: Call Trace: Oct 26 12:30:21 UnRaid kernel: [<c10c2917>] sysfs_name_hash+0x11/0x5b Oct 26 12:30:21 UnRaid kernel: [<c10c2b14>] sysfs_find_dirent+0x65/0xb4 Oct 26 12:30:21 UnRaid kernel: [<c1081f49>] ? kfree+0xa7/0xb0 Oct 26 12:30:21 UnRaid kernel: [<c10c2c55>] sysfs_get_dirent+0x26/0x55 Oct 26 12:30:21 UnRaid kernel: [<c10c449d>] sysfs_remove_group+0x20/0xa7 Oct 26 12:30:21 UnRaid kernel: [<c104ec74>] mod_sysfs_teardown+0xc7/0xfe Oct 26 12:30:21 UnRaid kernel: [<c104f02c>] free_module+0x2f/0x81 Oct 26 12:30:21 UnRaid kernel: [<c104fb63>] sys_delete_module+0x1a0/0x1c0 Oct 26 12:30:21 UnRaid kernel: [<c1075c49>] ? do_munmap+0x1e2/0x1fc Oct 26 12:30:21 UnRaid kernel: [<c131fbb5>] syscall_call+0x7/0xb Oct 26 12:30:21 UnRaid kernel: Code: 5d c3 55 89 e5 56 89 c6 89 d0 88 c4 ac 38 e0 74 09 84 c0 75 f7 be 01 00 00 00 89 f0 48 5e 5d c3 55 83 c9 ff 89 e5 57 89 c7 31 c0 <f2> ae f7 d1 8d 41 ff 5f 5d c3 55 89 e5 57 31 ff 85 c9 74 0e 89 Oct 26 12:30:21 UnRaid kernel: EIP: [<c11a98c7>] strlen+0xb/0x15 SS:ESP 0068:f1a59ec4 Oct 26 12:30:21 UnRaid kernel: CR2: 0000000073750075 Oct 26 12:30:21 UnRaid kernel: ---[ end trace 1e693e7eb90d87ef ]--- Message from syslogd@UnRaid at Fri Oct 26 12:30:21 2012 ... UnRaid kernel: Stack: Message from syslogd@UnRaid at Fri Oct 26 12:30:21 2012 ... UnRaid kernel: EIP: [<c11a98c7>] strlen+0xb/0x15 SS:ESP 0068:f1a59ec4 Message from syslogd@UnRaid at Fri Oct 26 12:30:21 2012 ... UnRaid kernel: Code: 5d c3 55 89 e5 56 89 c6 89 d0 88 c4 ac 38 e0 74 09 84 c0 75 f7 be 01 00 00 00 89 f0 48 5e 5d c3 55 83 c9 ff 89 e5 57 89 c7 31 c0 <f2> ae f7 d1 8d 41 ff 5f 5d c3 55 89 e5 57 31 ff 85 c9 74 0e 89 Message from syslogd@UnRaid at Fri Oct 26 12:30:21 2012 ... UnRaid kernel: Call Trace:

Drive contents (at a file level) are perfect - mounted independently and did a diff against my production server (which is an rsync mirror master). reiserfsck also yielded no issues. Disk is fine. Got the disk back online - no big problem. Very likely a crappy SATA cable issue. However, that's not REALLY the issue - regardless of whether I have a redball disk, I should not be getting a kernel panic every time mover runs. For giggles, I got my test array back online - all green - checked parity, etc. Then I updated to rc5 and turned mover back on. Same panic. Now.... my test server is not exactly going to replace a Google datacenter anytime soon - it's not much more than a pile of parts that were on their way to a dumpster (like... disks, a PS, and a MB laying in the corner of an unused bench - excellent thermal performance when you don't bother using a case). But having said that, my uptime with b13 running CP/SB/Sabnzbd/CrashPlan/PMS/Hamachi was pretty darn huge. More to follow.

Ok - I'm behind. However, b13 system was EXTREMELY stable for me until a few days ago when I added the latest CouchPotato_v2 plugin and updated Plex media server to the latest (0.9.6.6). Got a red ball on a known good disk. Ran SMART tests to confirm. Tried trust array procedure which resulted in mdcmd invalid argument error already covered in another thread. Started the array anyway (unprotected) and disabled anything that would perform writes - other than mover. I did not realize there was still data on the cache to sync. Nonetheless, scenario should not have caused a panic. I'm updating to latest rc hoping that mdcmd trust array fix is in place. Aug 12 19:00:01 UnRaid logger: mover started Aug 12 19:00:01 UnRaid logger: skipping Apps/ Aug 12 19:00:01 UnRaid logger: moving Media/ Aug 12 19:00:01 UnRaid logger: moving Television/ Aug 12 19:00:14 UnRaid logger: ./Television/The Daily Show with Jon Stewart/The Daily Show with Jon Stewart - s17e134 - Jessica Biel.mkv Aug 12 19:00:14 UnRaid logger: rsync: get_xattr_data: lgetxattr(".","user.org.netatalk.supports-eas.aK2YbM",0) failed: Exec format error ( Aug 12 19:00:14 UnRaid logger: rsync: get_xattr_data: lgetxattr("Television","user.org.netatalk.supports-eas.PvAD6m",0) failed: Exec format error ( Aug 12 19:00:14 UnRaid logger: >f......... Television/The Daily Show with Jon Stewart/The Daily Show with Jon Stewart - s17e134 - Jessica Biel.mkv Aug 12 19:00:35 UnRaid kernel: BUG: unable to handle kernel NULL pointer dereference at 00000056 Aug 12 19:00:35 UnRaid kernel: IP: [<c107e2b4>] __kmalloc_track_caller+0xb7/0xef Aug 12 19:00:35 UnRaid kernel: *pdpt = 000000003746e001 *pde = 0000000000000000 Aug 12 19:00:35 UnRaid kernel: Oops: 0000 [#1] SMP Aug 12 19:00:35 UnRaid kernel: Modules linked in: md_mod xor asus_atk0110 pata_jmicron hwmon sata_promise jmicron ata_piix i2c_i801 i2c_core r8169 [last unloaded: md_mod] Aug 12 19:00:35 UnRaid kernel: Aug 12 19:00:35 UnRaid kernel: Pid: 21135, comm: shfs Tainted: G W 3.1.0-unRAID #2 System manufacturer System Product Name/P5QL/EPU Aug 12 19:00:35 UnRaid kernel: EIP: 0060:[<c107e2b4>] EFLAGS: 00010206 CPU: 0 Aug 12 19:00:35 UnRaid kernel: EIP is at __kmalloc_track_caller+0xb7/0xef Aug 12 19:00:35 UnRaid kernel: EAX: 00000000 EBX: f46ce900 ECX: 00004201 EDX: 00004201 Aug 12 19:00:35 UnRaid kernel: ESI: f1c02580 EDI: 00000056 EBP: f1c0ff08 ESP: f1c0feec Aug 12 19:00:35 UnRaid kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Aug 12 19:00:35 UnRaid kernel: Process shfs (pid: 21135, ti=f1c0e000 task=f1d86f40 task.ti=f4764000) Aug 12 19:00:35 UnRaid kernel: Stack: Aug 12 19:00:35 UnRaid kernel: c128f717 00004201 00000020 00000056 f46ce900 00000020 000001fa f1c0ff28 Aug 12 19:00:35 UnRaid kernel: c128e9ce 00000000 f1c02180 000001c0 f158f000 f75d8000 00000000 f1c0ff38 Aug 12 19:00:35 UnRaid kernel: c128f717 ffffffff f158f000 f1c0ff80 f8427c6c f158f40c f158f400 f158fabc Aug 12 19:00:35 UnRaid kernel: Call Trace: Aug 12 19:00:35 UnRaid kernel: [<c128f717>] ? __netdev_alloc_skb+0x16/0x31 Aug 12 19:00:35 UnRaid kernel: [<c128e9ce>] __alloc_skb+0x53/0xf1 Aug 12 19:00:35 UnRaid kernel: [<c128f717>] __netdev_alloc_skb+0x16/0x31 Aug 12 19:00:35 UnRaid kernel: [<f8427c6c>] rtl8169_rx_interrupt+0x19c/0x2c4 [r8169] Aug 12 19:00:35 UnRaid kernel: [<f8427dbd>] rtl8169_poll+0x29/0x126 [r8169] Aug 12 19:00:35 UnRaid kernel: [<c12948fb>] net_rx_action+0x59/0x12a Aug 12 19:00:35 UnRaid kernel: [<c102c9da>] __do_softirq+0x6b/0xe5 Aug 12 19:00:35 UnRaid kernel: [<c102c96f>] ? irq_enter+0x3c/0x3c Aug 12 19:00:35 UnRaid kernel: <IRQ> Aug 12 19:00:35 UnRaid kernel: [<c102c82d>] ? irq_exit+0x32/0x53 Aug 12 19:00:35 UnRaid kernel: [<c10035ab>] ? do_IRQ+0x7c/0x90 Aug 12 19:00:35 UnRaid kernel: [<c130bea9>] ? common_interrupt+0x29/0x30 Aug 12 19:00:35 UnRaid kernel: [<c1300000>] ? quirk_usb_handoff_xhci+0x85/0x1ac Aug 12 19:00:35 UnRaid kernel: Code: 8b 08 85 c9 89 4d f0 75 16 8b 7d e4 8b 55 ec 50 89 f0 89 f9 e8 12 ec ff ff 59 89 45 f0 eb 1f 8b 7d f0 8b 46 14 8b 4d e8 8b 55 e8 <8b> 1c 07 89 f8 41 8b 3e 64 0f c7 0f 0f 94 c0 84 c0 74 b3 83 7d Aug 12 19:00:35 UnRaid kernel: EIP: [<c107e2b4>] __kmalloc_track_caller+0xb7/0xef SS:ESP 0068:f1c0feec Aug 12 19:00:35 UnRaid kernel: CR2: 0000000000000056 Aug 12 19:00:35 UnRaid kernel: ---[ end trace cb01a034686e189d ]--- Aug 12 19:00:35 UnRaid kernel: Kernel panic - not syncing: Fatal exception in interrupt Aug 12 19:00:35 UnRaid kernel: Pid: 21135, comm: shfs Tainted: G D W 3.1.0-unRAID #2 Aug 12 19:00:35 UnRaid kernel: Call Trace: Aug 12 19:00:35 UnRaid kernel: [<c13092db>] panic+0x50/0x143 Aug 12 19:00:35 UnRaid kernel: [<c100484e>] oops_end+0x6e/0x7c Aug 12 19:00:35 UnRaid kernel: [<c101adf1>] no_context+0xac/0xb6 Aug 12 19:00:35 UnRaid kernel: [<c101aee3>] __bad_area_nosemaphore+0xe8/0xf0 Aug 12 19:00:35 UnRaid kernel: [<c101b09a>] ? mm_fault_error+0x129/0x129 Aug 12 19:00:35 UnRaid kernel: [<c101aef8>] bad_area_nosemaphore+0xd/0x10 Aug 12 19:00:35 UnRaid kernel: [<c101b1eb>] do_page_fault+0x151/0x332 Aug 12 19:00:35 UnRaid kernel: [<c128e9a4>] ? __alloc_skb+0x29/0xf1 Aug 12 19:00:35 UnRaid kernel: [<c123033d>] ? ata_scsi_translate+0xbf/0xed Aug 12 19:00:35 UnRaid kernel: [<c101b09a>] ? mm_fault_error+0x12unraid: Host is down