February 3, 20179 yr So, I was glancing at my Unraid v6 and noticed a drive (disk1 xfs) had a red X. It was not there about 5 hours earlier. It had 512 errors and was taken offline. I ran the diagnostics tool and got the Smart report, attached. The drive SMART report looked completely clean. I thought maybe something happened with the Norco controller on that particular drive. So I took the array offline, so I yanked the drive and reseated it. Rebooted. Stopped Array. Disk1 still marked with RedX. Set Disk1 to nothing. Started Array. Stopped Array. Re-assigned the drive to Disk1, to have it rebuild, because I assumed disk1 was actually good and there was a controller hiccup. Disk1 immediately showed 712 errors and it went offline. So, I thought perhaps that slot itself was bad. Stopped Array. Moved drive to an entirely different slot in a different row (different hardware backplane). Checked that Unraid saw the drive. Checked the Smart report, still clean. Assigned the drive to Disk1. Restarted Array. Rebuild. All was well I thought. I looked about 20 minutes later and Disk7 now is showing 17 million errors while Disk1 is being rebuilt. What to do?? I am sure Disk1 data will be trashed. I will have a new replacement drive for Disk1 tomorrow. But, if Disk7 is bad (what are the chances??) then what??? I suspect there is a hardware issue here - but uncertain what to do next. I have two Unraid Arrays on different Norco boxes. But, my other Unraid is maxed out with 24 drives - so I don't think I can make much use of it. help! sos-help.zip
February 3, 20179 yr Author So, I stopped the array rebuild. Powered Down Unraid. Cold booted the Norco box. Unraid came back up and started to rebuild drive 1. 5 minutes in and Disk7 began showing errors again. I am attaching the Smart Report for Disk7, right at reboot - while Drive1 was being rebuilt and before Disk7 began showing errors. It looks clean. persipolis-smart-20170203-0119.zip
February 3, 20179 yr Community Expert Can't open your syslog, also always post the complete diags: tools -> diagnostics
February 3, 20179 yr Author Complete diagnostics file. Thank you persipolis-diagnostics-20170202-2341.zip
February 3, 20179 yr Community Expert USB where the flash drive is constantly resetting, try a different port. Jan 30 04:43:38 Persipolis kernel: usb 1-1: reset high-speed USB device number 2 using ehci-pci Jan 30 04:43:38 Persipolis kernel: sd 3:0:0:0: [sdl] 31258624 512-byte logical blocks: (16.0 GB/14.9 GiB) Jan 30 04:44:01 Persipolis crond[1845]: exit status 127 from user root /usr/lib/sa/sa1 1 1 &> /dev/null Jan 30 04:44:16 Persipolis kernel: usb 1-1: reset high-speed USB device number 2 using ehci-pci Jan 30 04:44:16 Persipolis kernel: sd 3:0:0:0: [sdl] 31258624 512-byte logical blocks: (16.0 GB/14.9 GiB) Jan 30 04:44:52 Persipolis kernel: usb 1-1: reset high-speed USB device number 2 using ehci-pci Jan 30 04:44:52 Persipolis kernel: sd 3:0:0:0: [sdl] 31258624 512-byte logical blocks: (16.0 GB/14.9 GiB) There are 2 disks with pending sectors, they may be a false positives but you should run an extended SMART test to confirm: Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WCAZA1996462 196 Reallocated_Event_Count 0x0032 185 185 000 Old_age Always - 15 197 Current_Pending_Sector 0x0032 200 196 000 Old_age Always - 9 Device Model: WDC WD20EARX-00PASB0 Serial Number: WD-WMAZA8044072 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 6 As for disk1 it looks fine, disk7 as well, syslog doesn't cover the issues with disk7, checking all cables would also be a good idea.
February 3, 20179 yr Author So, when I drop into Unraid via putty and run a smartctl on Disk7 sudo smartctl -c /dev/sdk I get this Smartctl open device: /dev/sdk failed: No such device Is that because Unraid has disabled this drive or is it because the drive has vanished itself from Unraid? I am attaching a new diagnostics report which should contain the syslog for all the errors associated with Drive 7, it comes from the last attempt to build Drive 1. persipolis-diagnostics-20170203-1208.zip
February 3, 20179 yr Community Expert Disk7 probably drooped offline, SMART for it on the previous diags is fine and the error is very similar to disk1, there was a error on the LSI controller and it lost communication with the disk, unfortunately the MPT2SAS driver is not very helpful in the way it reports errors, hence my recommendation to check cables, also check that the controller is well seated.
February 3, 20179 yr Author Okay, so hopefully this is more of a hardware issue and once we can rectify it - then I can rebuild Disk1 and nothing will be lost. I will open that server up and reseat all the cables and so on. I do wonder if the motherboard could be having issues because the USB problem you noticed popped up a few weeks ago. I noticed that /boot was inaccessible. After much tinkering around I ended up moving the USB to a port on the back and I thought that had fixed things, but apparently not. My first thought would be the USB is bad - but with these issues with the disks, I am wondering about the motherboard itself.
February 3, 20179 yr Author I reseated all of the cables and the controller card. I noticed the controller card was, unbelievably, not screwed in and secured to the chassis - so I secured it. I also moved the USB to a different port. Lastly, I moved Disk7 to a different row in the Norco, just in case. Powered up the server. Unraid began rebuilding it. 1 hour and 20 minutes in - 1 million writes to drive 1. No errors so far. Fingers crossed.
February 4, 20179 yr Author @johnnie.black - btw, the USB message you saw appears to be this issue: http://lime-technology.com/forum/index.php?topic=40605.0. Doesn't seem to be a real solution other than reverting to 6.1.9, or perhaps booting off a microSD card + reader. The rebuild is 70% done with no issues since reseating all of the cables - thanks for your help!
February 4, 20179 yr Author Just to close this out - disk1 was succesfully rebuilt. Reseating all the cables seems to have fixed everything. Thank you @johnnie.black!
Archived
This topic is now archived and is closed to further replies.