1812 Posted March 20, 2018 Share Posted March 20, 2018 running 6.5 Last night I powered down my main server because we were having massive storms. Even unplugged it from the large ups that I have. It took a bit longer shutting down last night and exported diagnostics, which I thought was odd. This am, the array wouldn't start. It was complaining about Mar 20 06:26:32 Brahms1 kernel: mpt2sas_cm0: _base_diag_reset: failed going to ready state (ioc_state=0xf0000000) Mar 20 06:26:32 Brahms1 kernel: mpt2sas_cm0: diag reset: FAILED Mar 20 06:26:32 Brahms1 kernel: mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:8942/_scsih_probe()! Mar 20 06:40:56 Brahms1 kernel: mpt2sas_cm0: _config_request: failed due to ioc not operational Mar 20 06:40:56 Brahms1 kernel: mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_transport.c:157/_transport_set_identify()! Mar 20 06:40:57 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=1) Mar 20 06:40:58 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=2) Mar 20 06:40:59 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=3) Mar 20 06:41:00 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=4) Mar 20 06:41:01 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=5) Mar 20 06:41:02 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=6) Mar 20 06:41:03 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=7) Mar 20 06:41:04 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=8) Mar 20 06:41:05 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=9) Mar 20 06:41:06 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=10) Mar 20 06:41:07 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=11) Mar 20 06:41:08 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=12) Mar 20 06:41:09 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=13) Mar 20 06:41:10 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=14) Mar 20 06:41:11 Brahms1 kernel: mpt2sas_cm0: _config_request: waiting for operational state(count=15) Mar 20 06:41:11 Brahms1 kernel: mpt2sas_cm0: _config_request: failed due to ioc not operational Mar 20 06:41:11 Brahms1 kernel: mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:5141/_scsih_expander_add()! I swapped out the HBA, the SAS Expander, cabling, and even moved all the disks to the other side of my md1000 which is currently operating in split mode. Regardless nothing worked to bring up the disks which were all listed as missing. I moved the HBA and expander card to a different server and put the drives back in their original side of the md1000. It booted without issue. How did the server die overnight from doing nothing? Not sure where to start to get it back online (even though I'm up and running on a different one currently.) It was solid before then. The end of the diagnosis from last shutdown that it collected are: Mar 19 21:27:32 Brahms1 shutdown[8375]: shutting down for system halt Mar 19 21:27:32 Brahms1 init: Switching to runlevel: 0 Mar 19 21:27:35 Brahms1 emhttpd: req (10): cmdStop=apply&csrf_token=**************** Mar 19 21:27:35 Brahms1 emhttpd: Spinning up all drives... Mar 19 21:27:35 Brahms1 emhttpd: shcmd (16132): /usr/sbin/hdparm -S0 /dev/sdc Mar 19 21:27:35 Brahms1 kernel: mdcmd (297): nocheck Mar 19 21:27:35 Brahms1 kernel: md: nocheck_array: check not active Mar 19 21:27:35 Brahms1 kernel: mdcmd (298): spinup 0 Mar 19 21:27:35 Brahms1 kernel: mdcmd (299): spinup 1 Mar 19 21:27:35 Brahms1 kernel: mdcmd (300): spinup 2 Mar 19 21:27:35 Brahms1 kernel: mdcmd (301): spinup 3 Mar 19 21:27:35 Brahms1 kernel: mdcmd (302): spinup 4 Mar 19 21:27:35 Brahms1 kernel: mdcmd (303): spinup 5 Mar 19 21:27:35 Brahms1 kernel: mdcmd (304): spinup 6 Mar 19 21:27:35 Brahms1 kernel: mdcmd (305): spinup 7 Mar 19 21:27:35 Brahms1 root: Mar 19 21:27:35 Brahms1 root: /dev/sdc: Mar 19 21:27:35 Brahms1 root: setting standby to 0 (off) Mar 19 21:27:35 Brahms1 emhttpd: shcmd (16133): /usr/sbin/hdparm -S0 /dev/sdb Mar 19 21:27:35 Brahms1 root: Mar 19 21:27:35 Brahms1 root: /dev/sdb: Mar 19 21:27:35 Brahms1 root: setting standby to 0 (off) Mar 19 21:27:44 Brahms1 emhttpd: Stopping services... Mar 19 21:27:44 Brahms1 emhttpd: shcmd (16135): /etc/rc.d/rc.libvirt stop Mar 19 21:27:44 Brahms1 root: Waiting on VMs to shutdown Mar 19 21:27:44 Brahms1 root: Stopping libvirtd... Mar 19 21:27:44 Brahms1 dnsmasq[8364]: exiting on receipt of SIGTERM Mar 19 21:27:44 Brahms1 kernel: device virbr0-nic left promiscuous mode Mar 19 21:27:44 Brahms1 kernel: virbr0: port 1(virbr0-nic) entered disabled state Mar 19 21:27:44 Brahms1 avahi-daemon[4518]: Interface virbr0.IPv4 no longer relevant for mDNS. Mar 19 21:27:44 Brahms1 avahi-daemon[4518]: Leaving mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1. Mar 19 21:27:44 Brahms1 avahi-daemon[4518]: Withdrawing address record for 192.168.122.1 on virbr0. Mar 19 21:27:45 Brahms1 root: Network 7b78fb65-4fac-44d5-9754-39482f1311f9 destroyed Mar 19 21:27:45 Brahms1 root: Mar 19 21:27:48 Brahms1 root: Stopping virtlogd... Mar 19 21:27:49 Brahms1 root: Stopping virtlockd... Mar 19 21:27:50 Brahms1 emhttpd: shcmd (16136): umount /etc/libvirt Mar 19 21:27:50 Brahms1 emhttpd: shcmd (16138): /etc/rc.d/rc.docker stop Mar 19 21:27:52 Brahms1 kernel: veth2b1fe4c: renamed from eth0 Mar 19 21:27:52 Brahms1 kernel: docker0: port 1(veth7eb2166) entered disabled state Mar 19 21:27:52 Brahms1 kernel: veth4c990b9: renamed from eth0 Mar 19 21:27:52 Brahms1 kernel: docker0: port 2(veth06eae98) entered disabled state Mar 19 21:27:52 Brahms1 avahi-daemon[4518]: Interface veth7eb2166.IPv6 no longer relevant for mDNS. Mar 19 21:27:52 Brahms1 avahi-daemon[4518]: Leaving mDNS multicast group on interface veth7eb2166.IPv6 with address fe80::3c08:cdff:feab:4a81. Mar 19 21:27:52 Brahms1 kernel: docker0: port 1(veth7eb2166) entered disabled state Mar 19 21:27:52 Brahms1 kernel: device veth7eb2166 left promiscuous mode Mar 19 21:27:52 Brahms1 kernel: docker0: port 1(veth7eb2166) entered disabled state Mar 19 21:27:52 Brahms1 avahi-daemon[4518]: Withdrawing address record for fe80::3c08:cdff:feab:4a81 on veth7eb2166. Mar 19 21:27:52 Brahms1 avahi-daemon[4518]: Interface veth06eae98.IPv6 no longer relevant for mDNS. Mar 19 21:27:52 Brahms1 kernel: docker0: port 2(veth06eae98) entered disabled state Mar 19 21:27:52 Brahms1 avahi-daemon[4518]: Leaving mDNS multicast group on interface veth06eae98.IPv6 with address fe80::b058:d9ff:fe48:5167. Mar 19 21:27:52 Brahms1 kernel: device veth06eae98 left promiscuous mode Mar 19 21:27:52 Brahms1 kernel: docker0: port 2(veth06eae98) entered disabled state Mar 19 21:27:52 Brahms1 avahi-daemon[4518]: Withdrawing address record for fe80::b058:d9ff:fe48:5167 on veth06eae98. Mar 19 21:27:53 Brahms1 kernel: docker0: port 4(veth3e69a59) entered disabled state Mar 19 21:27:53 Brahms1 kernel: veth701618e: renamed from eth0 Mar 19 21:27:53 Brahms1 avahi-daemon[4518]: Interface veth3e69a59.IPv6 no longer relevant for mDNS. Mar 19 21:27:53 Brahms1 avahi-daemon[4518]: Leaving mDNS multicast group on interface veth3e69a59.IPv6 with address fe80::e851:c3ff:fefa:f180. Mar 19 21:27:53 Brahms1 kernel: docker0: port 4(veth3e69a59) entered disabled state Mar 19 21:27:53 Brahms1 kernel: device veth3e69a59 left promiscuous mode Mar 19 21:27:53 Brahms1 kernel: docker0: port 4(veth3e69a59) entered disabled state Mar 19 21:27:53 Brahms1 avahi-daemon[4518]: Withdrawing address record for fe80::e851:c3ff:fefa:f180 on veth3e69a59. Mar 19 21:27:55 Brahms1 kernel: docker0: port 3(vethc7a1561) entered disabled state Mar 19 21:27:55 Brahms1 kernel: vethc5154de: renamed from eth0 Mar 19 21:27:56 Brahms1 avahi-daemon[4518]: Interface vethc7a1561.IPv6 no longer relevant for mDNS. Mar 19 21:27:56 Brahms1 avahi-daemon[4518]: Leaving mDNS multicast group on interface vethc7a1561.IPv6 with address fe80::d014:a1ff:fe16:6fb9. Mar 19 21:27:56 Brahms1 kernel: docker0: port 3(vethc7a1561) entered disabled state Mar 19 21:27:56 Brahms1 kernel: device vethc7a1561 left promiscuous mode Mar 19 21:27:56 Brahms1 kernel: docker0: port 3(vethc7a1561) entered disabled state Mar 19 21:27:56 Brahms1 avahi-daemon[4518]: Withdrawing address record for fe80::d014:a1ff:fe16:6fb9 on vethc7a1561. Mar 19 21:27:56 Brahms1 root: stopping dockerd ... Mar 19 21:27:57 Brahms1 root: waiting for docker to die ... Mar 19 21:27:58 Brahms1 avahi-daemon[4518]: Interface docker0.IPv6 no longer relevant for mDNS. Mar 19 21:27:58 Brahms1 avahi-daemon[4518]: Leaving mDNS multicast group on interface docker0.IPv6 with address fe80::42:73ff:fe30:fd64. Mar 19 21:27:58 Brahms1 avahi-daemon[4518]: Interface docker0.IPv4 no longer relevant for mDNS. Mar 19 21:27:58 Brahms1 avahi-daemon[4518]: Leaving mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1. Mar 19 21:27:58 Brahms1 avahi-daemon[4518]: Withdrawing address record for fe80::42:73ff:fe30:fd64 on docker0. Mar 19 21:27:58 Brahms1 avahi-daemon[4518]: Withdrawing address record for 172.17.0.1 on docker0. Mar 19 21:27:58 Brahms1 emhttpd: shcmd (16139): umount /var/lib/docker Mar 19 21:27:58 Brahms1 unassigned.devices: Unmounting All Devices... Mar 19 21:28:35 Brahms1 root: Status of all loop devices Mar 19 21:28:35 Brahms1 root: /dev/loop1: [2049]:4 (/boot/bzfirmware) Mar 19 21:28:35 Brahms1 root: /dev/loop0: [2049]:3 (/boot/bzmodules) Mar 19 21:28:35 Brahms1 root: Active pids left on /mnt/* Mar 19 21:28:50 Brahms1 unassigned.devices: benchmark: shell_exec(/usr/bin/lsof '' 2>/dev/null|sort -k8|uniq -f7|grep -c -e REG) took 51.057973s. Mar 19 21:28:50 Brahms1 unassigned.devices: Disk with serial 'ST31000528AS_6VP9PRS0', mountpoint 'ST31000528AS_6VP9PRS0' is not mounted and cannot be unmounted... Mar 19 21:29:26 Brahms1 root: USER PID ACCESS COMMAND Mar 19 21:29:26 Brahms1 root: /mnt/cache: root kernel mount /mnt/cache Mar 19 21:29:26 Brahms1 root: /mnt/disk1: root kernel mount /mnt/disk1 Mar 19 21:29:26 Brahms1 root: /mnt/disk2: root kernel mount /mnt/disk2 Mar 19 21:29:26 Brahms1 root: /mnt/disk3: root kernel mount /mnt/disk3 Mar 19 21:29:26 Brahms1 root: /mnt/disk4: root kernel mount /mnt/disk4 Mar 19 21:29:26 Brahms1 root: /mnt/disk5: root kernel mount /mnt/disk5 Mar 19 21:29:26 Brahms1 root: /mnt/disk6: root kernel mount /mnt/disk6 Mar 19 21:29:26 Brahms1 root: /mnt/disk7: root kernel mount /mnt/disk7 Mar 19 21:29:26 Brahms1 root: /mnt/disks: root kernel mount / Mar 19 21:29:26 Brahms1 root: root 1 .rce. init Mar 19 21:29:26 Brahms1 root: root 2 .rc.. kthreadd Mar 19 21:29:26 Brahms1 root: root 4 .rc.. kworker/0:0H Mar 19 21:29:26 Brahms1 root: root 6 .rc.. mm_percpu_wq Mar 19 21:29:26 Brahms1 root: root 7 .rc.. ksoftirqd/0 Mar 19 21:29:26 Brahms1 root: root 8 .rc.. rcu_preempt Mar 19 21:29:26 Brahms1 root: root 9 .rc.. rcu_sched Mar 19 21:29:26 Brahms1 root: root 10 .rc.. rcu_bh Mar 19 21:29:26 Brahms1 root: root 11 .rc.. migration/0 Mar 19 21:29:26 Brahms1 root: root 12 .rc.. cpuhp/0 Mar 19 21:29:26 Brahms1 root: root 13 .rc.. cpuhp/1 Mar 19 21:29:26 Brahms1 root: root 14 .rc.. migration/1 Mar 19 21:29:26 Brahms1 root: root 15 .rc.. ksoftirqd/1 Mar 19 21:29:26 Brahms1 root: root 17 .rc.. kworker/1:0H Mar 19 21:29:26 Brahms1 root: root 18 .rc.. cpuhp/2 Mar 19 21:29:26 Brahms1 root: root 19 .rc.. migration/2 Mar 19 21:29:26 Brahms1 root: root 20 .rc.. ksoftirqd/2 Mar 19 21:29:26 Brahms1 root: root 22 .rc.. kworker/2:0H Mar 19 21:29:26 Brahms1 root: root 23 .rc.. cpuhp/3 Mar 19 21:29:26 Brahms1 root: root 24 .rc.. migration/3 Mar 19 21:29:26 Brahms1 root: root 25 .rc.. ksoftirqd/3 Mar 19 21:29:26 Brahms1 root: root 27 .rc.. kworker/3:0H Mar 19 21:29:26 Brahms1 root: root 28 .rc.. cpuhp/4 Mar 19 21:29:26 Brahms1 root: root 29 .rc.. migration/4 Mar 19 21:29:26 Brahms1 root: root 30 .rc.. ksoftirqd/4 Mar 19 21:29:26 Brahms1 root: root 32 .rc.. kworker/4:0H Mar 19 21:29:26 Brahms1 root: root 33 .rc.. cpuhp/5 Mar 19 21:29:26 Brahms1 root: root 34 .rc.. migration/5 Mar 19 21:29:26 Brahms1 root: root 35 .rc.. ksoftirqd/5 Mar 19 21:29:26 Brahms1 root: root 37 .rc.. kworker/5:0H Mar 19 21:29:26 Brahms1 root: root 38 .rc.. cpuhp/6 Mar 19 21:29:26 Brahms1 root: root 39 .rc.. migration/6 Mar 19 21:29:26 Brahms1 root: root 40 .rc.. ksoftirqd/6 Mar 19 21:29:26 Brahms1 root: root 42 .rc.. kworker/6:0H Mar 19 21:29:26 Brahms1 root: root 43 .rc.. cpuhp/7 Mar 19 21:29:26 Brahms1 root: root 44 .rc.. migration/7 Mar 19 21:29:26 Brahms1 root: root 45 .rc.. ksoftirqd/7 Mar 19 21:29:26 Brahms1 root: root 47 .rc.. kworker/7:0H Mar 19 21:29:26 Brahms1 root: root 48 .rc.. kdevtmpfs Mar 19 21:29:26 Brahms1 root: root 49 .rc.. netns Mar 19 21:29:26 Brahms1 root: root 318 .rc.. oom_reaper Mar 19 21:29:26 Brahms1 root: root 319 .rc.. writeback Mar 19 21:29:26 Brahms1 root: root 321 .rc.. kcompactd0 Mar 19 21:29:26 Brahms1 root: root 322 .rc.. ksmd Mar 19 21:29:26 Brahms1 root: root 323 .rc.. khugepaged Mar 19 21:29:26 Brahms1 root: root 324 .rc.. crypto Mar 19 21:29:26 Brahms1 root: root 325 .rc.. kintegrityd Mar 19 21:29:26 Brahms1 root: root 327 .rc.. kblockd Mar 19 21:29:26 Brahms1 root: root 463 .rc.. ata_sff Mar 19 21:29:26 Brahms1 root: root 483 .rc.. edac-poller Mar 19 21:29:26 Brahms1 root: root 484 .rc.. devfreq_wq Mar 19 21:29:26 Brahms1 root: root 621 .rc.. kswapd0 Mar 19 21:29:26 Brahms1 root: root 731 .rc.. kthrotld Mar 19 21:29:26 Brahms1 root: root 859 .rc.. vfio-irqfd-clea Mar 19 21:29:26 Brahms1 root: root 955 .rc.. ipv6_addrconf Mar 19 21:29:26 Brahms1 root: root 996 .rc.. scsi_eh_0 Mar 19 21:29:26 Brahms1 root: root 997 .rc.. scsi_tmf_0 Mar 19 21:29:26 Brahms1 root: root 998 .rc.. usb-storage Mar 19 21:29:26 Brahms1 root: root 1049 .rc.. kworker/7:1H Mar 19 21:29:26 Brahms1 root: root 1180 frce. udevd Mar 19 21:29:26 Brahms1 root: root 1252 .rc.. loop0 Mar 19 21:29:26 Brahms1 root: root 1253 .rc.. kworker/5:1H Mar 19 21:29:26 Brahms1 root: root 1254 .rc.. kworker/1:1H Mar 19 21:29:26 Brahms1 root: root 1255 .rc.. kworker/4:1H Mar 19 21:29:26 Brahms1 root: root 1256 .rc.. kworker/6:1H Mar 19 21:29:26 Brahms1 root: root 1257 .rc.. loop1 Mar 19 21:29:26 Brahms1 root: root 1267 .rc.. kworker/0:1H Mar 19 21:29:26 Brahms1 root: root 1284 .rc.. kworker/3:1H Mar 19 21:29:26 Brahms1 root: root 1298 .rc.. kworker/2:1H Mar 19 21:29:26 Brahms1 root: root 1317 .rc.. scsi_eh_1 Mar 19 21:29:26 Brahms1 root: root 1318 .rc.. scsi_tmf_1 Mar 19 21:29:26 Brahms1 root: root 1319 .rc.. scsi_eh_2 Mar 19 21:29:26 Brahms1 root: root 1320 .rc.. scsi_tmf_2 Mar 19 21:29:26 Brahms1 root: root 1321 .rc.. scsi_eh_3 Mar 19 21:29:26 Brahms1 root: root 1322 .rc.. scsi_tmf_3 Mar 19 21:29:26 Brahms1 root: root 1323 .rc.. scsi_eh_4 Mar 19 21:29:26 Brahms1 root: root 1324 .rc.. scsi_tmf_4 Mar 19 21:29:26 Brahms1 root: root 1325 .rc.. kipmi0 Mar 19 21:29:26 Brahms1 root: root 1328 .rc.. scsi_eh_5 Mar 19 21:29:26 Brahms1 root: root 1329 .rc.. scsi_tmf_5 Mar 19 21:29:26 Brahms1 root: root 1330 .rc.. fw_event_mpt2sa Mar 19 21:29:26 Brahms1 root: root 1409 .rce. rsyslogd Mar 19 21:29:26 Brahms1 root: root 1445 .rc.. poll_mpt2sas0_s Mar 19 21:29:26 Brahms1 root: root 1488 .rc.. bond0 Mar 19 21:29:26 Brahms1 root: messagebus 1529 Frce. dbus-daemon Mar 19 21:29:26 Brahms1 root: rpc 1538 Frce. rpcbind Mar 19 21:29:26 Brahms1 root: rpc 1542 Frce. rpc.statd Mar 19 21:29:26 Brahms1 root: ntp 1573 .rce. ntpd Mar 19 21:29:26 Brahms1 root: root 1580 Frce. acpid Mar 19 21:29:26 Brahms1 root: root 1589 .rce. crond Mar 19 21:29:26 Brahms1 root: daemon 1591 Frce. atd Mar 19 21:29:26 Brahms1 root: root 1599 Frce. nmbd Mar 19 21:29:26 Brahms1 root: root 1602 Frce. smbd Mar 19 21:29:26 Brahms1 root: root 1605 Frce. smbd-notifyd Mar 19 21:29:26 Brahms1 root: root 1606 Frce. cleanupd Mar 19 21:29:26 Brahms1 root: root 1608 Frce. winbindd Mar 19 21:29:26 Brahms1 root: root 1610 Frce. winbindd Mar 19 21:29:26 Brahms1 root: root 1685 .rc.. kworker/2:3 Mar 19 21:29:26 Brahms1 root: root 2653 Frce. winbindd Mar 19 21:29:26 Brahms1 root: root 4015 .rc.. kworker/1:0 Mar 19 21:29:26 Brahms1 root: root 4215 .rce. ipmiseld Mar 19 21:29:26 Brahms1 root: root 4440 .rce. sshd Mar 19 21:29:26 Brahms1 root: root 4449 Frce. inetd Mar 19 21:29:26 Brahms1 root: root 4456 .rc.. md Mar 19 21:29:26 Brahms1 root: root 4457 .rc.. mdrecoveryd Mar 19 21:29:26 Brahms1 root: root 4461 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4462 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4463 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4464 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4465 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4466 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4467 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4468 .rc.. spinupd Mar 19 21:29:26 Brahms1 root: root 4481 frce. ipmitail Mar 19 21:29:26 Brahms1 root: root 4485 frce. tail Mar 19 21:29:26 Brahms1 root: root 4486 .rce. ipmitail Mar 19 21:29:26 Brahms1 root: root 4502 .rce. netatalk Mar 19 21:29:26 Brahms1 root: avahi 4518 .rce. avahi-daemon Mar 19 21:29:26 Brahms1 root: avahi 4519 .rce. avahi-daemon Mar 19 21:29:26 Brahms1 root: root 4528 Frce. avahi-dnsconfd Mar 19 21:29:26 Brahms1 root: root 4529 .rce. afpd Mar 19 21:29:26 Brahms1 root: root 4530 .rce. cnid_metad Mar 19 21:29:26 Brahms1 root: root 4543 Frce. emhttpd Mar 19 21:29:26 Brahms1 root: root 4632 .rc.. unraidd Mar 19 21:29:26 Brahms1 root: root 4658 Frce. diskload Mar 19 21:29:26 Brahms1 root: root 4762 .rc.. xfsalloc Mar 19 21:29:26 Brahms1 root: root 4763 .rc.. xfs_mru_cache Mar 19 21:29:26 Brahms1 root: root 4764 .rc.. xfs-buf/md1 Mar 19 21:29:26 Brahms1 root: root 4765 .rc.. xfs-data/md1 Mar 19 21:29:26 Brahms1 root: root 4766 .rc.. xfs-conv/md1 Mar 19 21:29:26 Brahms1 root: root 4767 .rc.. xfs-cil/md1 Mar 19 21:29:26 Brahms1 root: root 4768 .rc.. xfs-reclaim/md1 Mar 19 21:29:26 Brahms1 root: root 4769 .rc.. xfs-log/md1 Mar 19 21:29:26 Brahms1 root: root 4770 .rc.. xfs-eofblocks/m Mar 19 21:29:26 Brahms1 root: root 4771 .rc.. xfsaild/md1 Mar 19 21:29:26 Brahms1 root: root 4783 .rc.. xfs-buf/md2 Mar 19 21:29:26 Brahms1 root: root 4784 .rc.. xfs-data/md2 Mar 19 21:29:26 Brahms1 root: root 4785 .rc.. xfs-conv/md2 Mar 19 21:29:26 Brahms1 root: root 4786 .rc.. xfs-cil/md2 Mar 19 21:29:26 Brahms1 root: root 4787 .rc.. xfs-reclaim/md2 Mar 19 21:29:26 Brahms1 root: root 4788 .rc.. xfs-log/md2 Mar 19 21:29:26 Brahms1 root: root 4789 .rc.. xfs-eofblocks/m Mar 19 21:29:26 Brahms1 root: root 4790 .rc.. xfsaild/md2 Mar 19 21:29:26 Brahms1 root: root 4802 .rc.. xfs-buf/md3 Mar 19 21:29:26 Brahms1 root: root 4803 .rc.. xfs-data/md3 Mar 19 21:29:26 Brahms1 root: root 4804 .rc.. xfs-conv/md3 Mar 19 21:29:26 Brahms1 root: root 4805 .rc.. xfs-cil/md3 Mar 19 21:29:26 Brahms1 root: root 4806 .rc.. xfs-reclaim/md3 Mar 19 21:29:26 Brahms1 root: root 4807 .rc.. xfs-log/md3 Mar 19 21:29:26 Brahms1 root: root 4808 .rc.. xfs-eofblocks/m Mar 19 21:29:26 Brahms1 root: root 4809 .rc.. xfsaild/md3 Mar 19 21:29:26 Brahms1 root: root 4821 .rc.. xfs-buf/md4 Mar 19 21:29:26 Brahms1 root: root 4822 .rc.. xfs-data/md4 Mar 19 21:29:26 Brahms1 root: root 4823 .rc.. xfs-conv/md4 Mar 19 21:29:26 Brahms1 root: root 4824 .rc.. xfs-cil/md4 Mar 19 21:29:26 Brahms1 root: root 4825 .rc.. xfs-reclaim/md4 Mar 19 21:29:26 Brahms1 root: root 4826 .rc.. xfs-log/md4 Mar 19 21:29:26 Brahms1 root: root 4827 .rc.. xfs-eofblocks/m Mar 19 21:29:26 Brahms1 root: root 4828 .rc.. xfsaild/md4 Mar 19 21:29:26 Brahms1 root: root 4840 .rc.. xfs-buf/md5 Mar 19 21:29:26 Brahms1 root: root 4841 .rc.. xfs-data/md5 Mar 19 21:29:26 Brahms1 root: root 4842 .rc.. xfs-conv/md5 Mar 19 21:29:26 Brahms1 root: root 4843 .rc.. xfs-cil/md5 Mar 19 21:29:26 Brahms1 root: root 4844 .rc.. xfs-reclaim/md5 Mar 19 21:29:26 Brahms1 root: root 4845 .rc.. xfs-log/md5 Mar 19 21:29:26 Brahms1 root: root 4846 .rc.. xfs-eofblocks/m Mar 19 21:29:26 Brahms1 root: root 4847 .rc.. xfsaild/md5 Mar 19 21:29:26 Brahms1 root: root 4859 .rc.. xfs-buf/md6 Mar 19 21:29:26 Brahms1 root: root 4860 .rc.. xfs-data/md6 Mar 19 21:29:26 Brahms1 root: root 4861 .rc.. xfs-conv/md6 Mar 19 21:29:26 Brahms1 root: root 4862 .rc.. xfs-cil/md6 Mar 19 21:29:26 Brahms1 root: root 4863 .rc.. xfs-reclaim/md6 Mar 19 21:29:26 Brahms1 root: root 4864 .rc.. xfs-log/md6 Mar 19 21:29:26 Brahms1 root: root 4865 .rc.. xfs-eofblocks/m Mar 19 21:29:26 Brahms1 root: root 4866 .rc.. xfsaild/md6 Mar 19 21:29:26 Brahms1 root: root 4878 .rc.. xfs-buf/md7 Mar 19 21:29:26 Brahms1 root: root 4879 .rc.. xfs-data/md7 Mar 19 21:29:26 Brahms1 root: root 4880 .rc.. xfs-conv/md7 Mar 19 21:29:26 Brahms1 root: root 4881 .rc.. xfs-cil/md7 Mar 19 21:29:26 Brahms1 root: root 4882 .rc.. xfs-reclaim/md7 Mar 19 21:29:26 Brahms1 root: root 4883 .rc.. xfs-log/md7 Mar 19 21:29:26 Brahms1 root: root 4884 .rc.. xfs-eofblocks/m Mar 19 21:29:26 Brahms1 root: root 4885 .rc.. xfsaild/md7 Mar 19 21:29:26 Brahms1 root: root 4898 .rc.. btrfs-worker Mar 19 21:29:26 Brahms1 root: root 4899 .rc.. kworker/u17:0 Mar 19 21:29:26 Brahms1 root: root 4900 .rc.. btrfs-worker-hi Mar 19 21:29:26 Brahms1 root: root 4901 .rc.. btrfs-delalloc Mar 19 21:29:26 Brahms1 root: root 4902 .rc.. btrfs-flush_del Mar 19 21:29:26 Brahms1 root: root 4903 .rc.. btrfs-cache Mar 19 21:29:26 Brahms1 root: root 4904 .rc.. btrfs-submit Mar 19 21:29:26 Brahms1 root: root 4905 .rc.. btrfs-fixup Mar 19 21:29:26 Brahms1 root: root 4906 .rc.. btrfs-endio Mar 19 21:29:26 Brahms1 root: root 4907 .rc.. btrfs-endio-met Mar 19 21:29:26 Brahms1 root: root 4908 .rc.. btrfs-endio-met Mar 19 21:29:26 Brahms1 root: root 4909 .rc.. btrfs-endio-rai Mar 19 21:29:26 Brahms1 root: root 4910 .rc.. btrfs-endio-rep Mar 19 21:29:26 Brahms1 root: root 4911 .rc.. btrfs-rmw Mar 19 21:29:26 Brahms1 root: root 4912 .rc.. btrfs-endio-wri Mar 19 21:29:26 Brahms1 root: root 4913 .rc.. btrfs-freespace Mar 19 21:29:26 Brahms1 root: root 4914 .rc.. btrfs-delayed-m Mar 19 21:29:26 Brahms1 root: root 4915 .rc.. btrfs-readahead Mar 19 21:29:26 Brahms1 root: root 4916 .rc.. btrfs-qgroup-re Mar 19 21:29:26 Brahms1 root: root 4917 .rc.. btrfs-extent-re Mar 19 21:29:26 Brahms1 root: root 4918 .rc.. btrfs-cleaner Mar 19 21:29:26 Brahms1 root: root 4919 .rc.. btrfs-transacti Mar 19 21:29:26 Brahms1 root: root 4943 Frce. shfs Mar 19 21:29:26 Brahms1 root: root 4956 Frce. shfs Mar 19 21:29:26 Brahms1 root: root 5139 .rc.. xfs-buf/sdm1 Mar 19 21:29:26 Brahms1 root: root 5140 .rc.. xfs-data/sdm1 Mar 19 21:29:26 Brahms1 root: root 5141 .rc.. xfs-conv/sdm1 Mar 19 21:29:26 Brahms1 root: root 5142 .rc.. xfs-cil/sdm1 Mar 19 21:29:26 Brahms1 root: root 5143 .rc.. xfs-reclaim/sdm Mar 19 21:29:26 Brahms1 root: root 5144 .rc.. xfs-log/sdm1 Mar 19 21:29:26 Brahms1 root: root 5145 .rc.. xfs-eofblocks/s Mar 19 21:29:26 Brahms1 root: root 5146 .rc.. xfsaild/sdm1 Mar 19 21:29:26 Brahms1 root: root 5417 .rc.. cifsiod Mar 19 21:29:26 Brahms1 root: root 5418 .rc.. cifsoplockd Mar 19 21:29:26 Brahms1 root: root 5420 .rc.. cifsd Mar 19 21:29:26 Brahms1 root: root 5597 .rc.. kworker/u16:1 Mar 19 21:29:26 Brahms1 root: root 6107 .rc.. kworker/u16:3 Mar 19 21:29:26 Brahms1 root: root 6135 .rc.. kworker/3:2 Mar 19 21:29:26 Brahms1 root: root 6278 .rc.. kworker/u16:4 Mar 19 21:29:26 Brahms1 root: root 6293 .rc.. kworker/4:1 Mar 19 21:29:26 Brahms1 root: root 6467 .rc.. kworker/6:0 Mar 19 21:29:26 Brahms1 root: root 7468 .rc.. kworker/u17:1 Mar 19 21:29:26 Brahms1 root: root 7859 .rc.. kworker/u16:0 Mar 19 21:29:26 Brahms1 root: root 7860 .rc.. kworker/u16:2 Mar 19 21:29:26 Brahms1 root: root 7861 .rc.. kworker/u16:6 Mar 19 21:29:26 Brahms1 root: root 7862 .rc.. kworker/u16:7 Mar 19 21:29:26 Brahms1 root: root 7863 .rc.. kworker/u16:8 Mar 19 21:29:26 Brahms1 root: root 7864 .rc.. kworker/u16:9 Mar 19 21:29:26 Brahms1 root: root 8380 frce. rc.0 Mar 19 21:29:26 Brahms1 root: root 8389 frce. rc.local_shutdo Mar 19 21:29:26 Brahms1 root: root 8450 .rc.. kworker/1:1 Mar 19 21:29:26 Brahms1 root: root 8637 .rc.. kworker/3:1 Mar 19 21:29:26 Brahms1 root: root 8639 .rc.. kworker/7:2 Mar 19 21:29:26 Brahms1 root: root 8650 .rc.. kworker/u16:10 Mar 19 21:29:26 Brahms1 root: root 8651 .rc.. kworker/u16:11 Mar 19 21:29:26 Brahms1 root: root 8652 .rc.. kworker/u16:12 Mar 19 21:29:26 Brahms1 root: root 8653 .rc.. kworker/u16:13 Mar 19 21:29:26 Brahms1 root: root 8661 .rc.. kworker/u16:14 Mar 19 21:29:26 Brahms1 root: root 8664 .rc.. kworker/u16:15 Mar 19 21:29:26 Brahms1 root: root 8712 .rc.. kworker/0:0 Mar 19 21:29:26 Brahms1 root: root 8736 .rc.. kworker/u16:16 Mar 19 21:29:26 Brahms1 root: root 8737 .rc.. kworker/u16:17 Mar 19 21:29:26 Brahms1 root: root 8738 .rc.. kworker/u16:18 Mar 19 21:29:26 Brahms1 root: root 8757 .rc.. kworker/5:0 Mar 19 21:29:26 Brahms1 root: root 8771 .rc.. kworker/2:0 Mar 19 21:29:26 Brahms1 root: root 8838 .rc.. kworker/6:1 Mar 19 21:29:26 Brahms1 root: root 8923 .rc.. kworker/4:0 Mar 19 21:29:26 Brahms1 root: root 8946 .rc.. kworker/3:3 Mar 19 21:29:26 Brahms1 root: root 8947 .rc.. kworker/7:0 Mar 19 21:29:26 Brahms1 root: root 9069 Frce. emhttp_event Mar 19 21:29:26 Brahms1 root: root 9079 Frce. stopping_svcs Mar 19 21:29:26 Brahms1 root: root 9083 Frce. rc.unassigned Mar 19 21:29:26 Brahms1 root: root 9190 Frce. sh Mar 19 21:29:26 Brahms1 root: root 9191 .rce. lsof Mar 19 21:29:26 Brahms1 root: root 9192 Frce. sort Mar 19 21:29:26 Brahms1 root: root 9193 Frce. uniq Mar 19 21:29:26 Brahms1 root: root 9194 Frce. grep Mar 19 21:29:26 Brahms1 root: root 9195 .rce. lsof Mar 19 21:29:26 Brahms1 root: root 9197 Frce. sh Mar 19 21:29:26 Brahms1 root: root 9199 frce. sadc Mar 19 21:29:26 Brahms1 root: root 9306 Frce. sleep Mar 19 21:29:26 Brahms1 root: root 9313 .rce. logger Mar 19 21:29:26 Brahms1 root: root 11678 .rc.. kworker/0:1 Mar 19 21:29:26 Brahms1 root: root 12330 .rc.. kworker/1:2 Mar 19 21:29:26 Brahms1 root: root 12759 .rc.. kworker/6:3 Mar 19 21:29:26 Brahms1 root: root 14131 .rc.. kworker/0:2 Mar 19 21:29:26 Brahms1 root: root 14335 .rc.. kworker/2:1 Mar 19 21:29:26 Brahms1 root: root 14427 .rc.. kworker/3:0 Mar 19 21:29:26 Brahms1 root: root 14811 .rc.. kworker/7:1 Mar 19 21:29:26 Brahms1 root: root 24908 .rc.. kworker/u16:5 Mar 19 21:29:26 Brahms1 root: root 25790 .rc.. kworker/5:1 Mar 19 21:29:26 Brahms1 root: root 26842 .rc.. kworker/5:2 Mar 19 21:29:26 Brahms1 root: /mnt/user: root kernel mount /mnt/user Mar 19 21:29:26 Brahms1 root: /mnt/user0: root kernel mount /mnt/user0 Mar 19 21:29:26 Brahms1 root: Active pids left on /dev/md* Mar 19 21:29:41 Brahms1 unassigned.devices: benchmark: shell_exec(/usr/bin/lsof '/mnt/disks/WDC_WD10EZEX-75ZF5A0_WD-WMC1S1107366' 2>/dev/null|sort -k8|uniq -f7|grep -c -e REG) took 51.192396s. Mar 19 21:29:41 Brahms1 unassigned.devices: Device '/dev/sdm1' script file not found. 'REMOVE' script not executed. Mar 19 21:29:41 Brahms1 unassigned.devices: Unmounting disk 'WDC_WD10EZEX-75ZF5A0_WD-WMC1S1107366'... Mar 19 21:29:41 Brahms1 unassigned.devices: Unmounting '/dev/sdm1'... Mar 19 21:30:17 Brahms1 root: USER PID ACCESS COMMAND Mar 19 21:30:17 Brahms1 root: /dev/md1: root kernel mount /mnt/disk1 Mar 19 21:30:17 Brahms1 root: /dev/md2: root kernel mount /mnt/disk2 Mar 19 21:30:17 Brahms1 root: /dev/md3: root kernel mount /mnt/disk3 Mar 19 21:30:17 Brahms1 root: /dev/md4: root kernel mount /mnt/disk4 Mar 19 21:30:17 Brahms1 root: /dev/md5: root kernel mount /mnt/disk5 Mar 19 21:30:17 Brahms1 root: /dev/md6: root kernel mount /mnt/disk6 Mar 19 21:30:17 Brahms1 root: /dev/md7: root kernel mount /mnt/disk7 Mar 19 21:30:17 Brahms1 root: Generating diagnostics... Mar 19 21:30:32 Brahms1 kernel: XFS (sdm1): Unmounting Filesystem Mar 19 21:30:32 Brahms1 unassigned.devices: Successfully unmounted '/dev/sdm1' Mar 19 21:30:32 Brahms1 unassigned.devices: Disk with serial 'WDC_WD10EZEX-75ZF5A0_WD-WMC1S1107366', mountpoint 'WDC_WD10EZEX-75ZF5A0_WD-WMC1S1107366' removed successfully. Mar 19 21:30:38 Brahms1 unassigned.devices: Error: SMB/NFS server 'REMOTE' is not responding to a ping and appears to be off-line. Mar 19 21:30:41 Brahms1 unassigned.devices: Error: SMB/NFS server 'REMOTE' is not responding to a ping and appears to be off-line. Mar 19 21:30:43 Brahms1 unassigned.devices: Error: SMB/NFS server 'REMOTE' is not responding to a ping and appears to be off-line. Mar 19 21:30:45 Brahms1 unassigned.devices: Error: SMB/NFS server 'REMOTE' is not responding to a ping and appears to be off-line. Mar 19 21:30:47 Brahms1 unassigned.devices: Error: SMB/NFS server 'REMOTE' is not responding to a ping and appears to be off-line. Mar 19 21:30:47 Brahms1 unassigned.devices: Remote SMB share '//TWINS/domains' is not mounted and cannot be unmounted... Mar 19 21:30:47 Brahms1 unassigned.devices: Remote SMB share '//BACKUP/Backup' is not mounted and cannot be unmounted... Mar 19 21:30:47 Brahms1 unassigned.devices: Removing Remote SMB share '//REMOTE/4TB'... Mar 19 21:30:47 Brahms1 unassigned.devices: Device '//REMOTE/4TB' script file not found. 'REMOVE' script not executed. Mar 19 21:30:47 Brahms1 unassigned.devices: Unmounting Remote SMB Share '//REMOTE/4TB'... Mar 19 21:30:47 Brahms1 unassigned.devices: Unmounting '//REMOTE/4TB'... Mar 19 21:31:38 Brahms1 unassigned.devices: Successfully unmounted '//REMOTE/4TB' Mar 19 21:31:38 Brahms1 unassigned.devices: Removing SMB share file '/etc/samba/unassigned-shares/REMOTE_4TB.conf' Mar 19 21:31:38 Brahms1 unassigned.devices: Removing SMB share definitions from '/boot/config/smb-extra.conf' Mar 19 21:31:38 Brahms1 unassigned.devices: Reloading Samba configuration.. Mar 19 21:31:38 Brahms1 unassigned.devices: Successfully removed SMB share 'REMOTE_4TB'. Mar 19 21:31:38 Brahms1 unassigned.devices: Share '//REMOTE/4TB' unmount successfull. Mar 19 21:31:38 Brahms1 unassigned.devices: Removing Remote SMB share '//REMOTE/750GBa'... Mar 19 21:31:38 Brahms1 unassigned.devices: Device '//REMOTE/750GBa' script file not found. 'REMOVE' script not executed. Mar 19 21:31:38 Brahms1 unassigned.devices: Unmounting Remote SMB Share '//REMOTE/750GBa'... Mar 19 21:31:38 Brahms1 unassigned.devices: Unmounting '//REMOTE/750GBa'... Assistance appreciated! Link to comment
JorgeB Posted March 20, 2018 Share Posted March 20, 2018 Try moving the HBA back to the original server, if it doesn't work again also try a different PCIe slot if available, if still issues possibly a board problem. Link to comment
1812 Posted May 19, 2018 Author Share Posted May 19, 2018 This reared its head again today. Bunch of the same errors and now this mpt2sas_cm0: port enable: FAILED with timeout (timeout=300s) After swapping out the sas expander, hba, server, and all cables inside and to the MD1000 enclosure, I hooked up an MSA70 and it saw all the discs just fine. So the enclosure was going bad and has now failed. Time to find a replacement! Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.