February 21, 201313 yr 'My cache drive went off line last night. In fact it is a RAID1 volume on a ARC1200 controller. Can you please check the attached syslog to see what went wrong, either the ARC1200 errored or one off the disks. If the controller went bad, would it be possible to move my current raid set over to another controller? Maybe I set it up badly, please give me pointers on how to check this. Update: After rebooting the server came back with the cache drive enabled. But the parity, which is a RAID0 volume on the same ARC1200 controller, detected as new. Currently it is building parity. No wonder off course, both parity and cache volume share the same controller and disks. Either the ARC1200 and X9SCM-F are not a happy couple or the 2 Seagates and the controller don't like each other. First thing is that i am gonna backup the contents of the cache disk. Something else comes to mind: Can it be that the driver for the ARC 1200 is updated in the latest 5.0RCs? The first occurence of this problem was after I upgraded to RC10. It had been running without these problems for over 6 months on beta12a and RC8a. Loaded module sizes are different, don't know if that tell anything. Currently I run unraid baremetal not in a VM. syslog-2013-02-21.zip
February 21, 201313 yr Assuming the driver is OK, it looks like the drive is having difficulty or there is a cabling issue. One of the two drives is having issues. Feb 20 04:48:36 Tower1 kernel: mdcmd (103): spindown 9 Feb 20 09:40:51 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 09:40:51 Tower1 kernel: Info fld=0x0 Feb 20 09:40:51 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 09:40:54 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 09:40:54 Tower1 kernel: Info fld=0x0 Feb 20 09:40:54 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 12:26:11 Tower1 kernel: mdcmd (104): spindown 2 ... Feb 20 16:46:39 Tower1 kernel: mdcmd (133): spindown 15 Feb 20 17:54:30 Tower1 in.telnetd[10130]: connect from 192.168.10.180 (192.168.10.180) Feb 20 17:54:37 Tower1 login[10131]: ROOT LOGIN on '/dev/pts/0' from 'Dirk-ASUSLaptop' Feb 20 17:55:27 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 17:55:27 Tower1 kernel: Info fld=0x0 Feb 20 17:55:27 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 17:55:30 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 17:55:30 Tower1 kernel: Info fld=0x0 Feb 20 17:55:30 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 18:57:33 Tower1 kernel: mdcmd (134): spindown 0 Feb 20 19:03:13 Tower1 kernel: mdcmd (135): spindown 4 Feb 20 19:03:14 Tower1 kernel: mdcmd (136): spindown 5 Feb 20 19:03:34 Tower1 kernel: mdcmd (137): spindown 8 Feb 20 19:46:59 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 19:46:59 Tower1 kernel: Info fld=0x0 Feb 20 19:46:59 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 19:47:02 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 19:47:02 Tower1 kernel: Info fld=0x0 Feb 20 19:47:02 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 20:47:36 Tower1 kernel: mdcmd (138): spindown 1 ... Feb 20 20:55:58 Tower1 kernel: mdcmd (153): spindown 14 Feb 20 21:25:30 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 21:25:30 Tower1 kernel: Info fld=0x0 Feb 20 21:25:30 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 21:25:33 Tower1 kernel: sd 0:0:0:1: [sdb] Sense Key : 0x0 [current] Feb 20 21:25:33 Tower1 kernel: Info fld=0x0 Feb 20 21:25:33 Tower1 kernel: sd 0:0:0:1: [sdb] ASC=0x0 ASCQ=0x0 Feb 20 22:42:09 Tower1 kernel: mdcmd (154): spindown 7 ... Feb 21 00:07:47 Tower1 kernel: arcmsr0: abort device command of scsi id = 0 lun = 1 Feb 21 00:07:51 Tower1 kernel: arcmsr: executing bus reset eh.....num_resets = 0, num_aborts = 1 Feb 21 00:08:31 Tower1 kernel: arcmsr0: wait 'abort all outstanding command' timeout Feb 21 00:08:31 Tower1 kernel: sd 0:0:0:1: Device offlined - not ready after error recovery Feb 21 00:08:31 Tower1 kernel: sd 0:0:0:1: [sdb] Unhandled error code Feb 21 00:08:31 Tower1 kernel: sd 0:0:0:1: [sdb] Result: hostbyte=0x03 driverbyte=0x00 Feb 21 00:08:31 Tower1 kernel: sd 0:0:0:1: [sdb] CDB: cdb[0]=0x2a: 2a 00 0e 76 06 78 00 00 08 00 Feb 21 00:08:31 Tower1 kernel: end_request: I/O error, dev sdb, sector 242615928 Feb 21 00:08:31 Tower1 kernel: Buffer I/O error on device sdb1, logical block 30326983 Feb 21 00:08:31 Tower1 kernel: lost page write due to I/O error on sdb1 Feb 21 00:08:31 Tower1 kernel: REISERFS error (device sdb1): vs-13050 reiserfs_update_sd_size: i/o failure occurred trying to update [62294 62299 0x0 SD] stat data Feb 21 00:08:31 Tower1 kernel: REISERFS (device sdb1): Remounting filesystem read-only Feb 21 00:08:31 Tower1 kernel: sd 0:0:0:1: rejecting I/O to offline device It could also be some timeout value on the driver changed. If it were me I would probably backup the cache drive as soon as possible (which should be done daily anyway). Take down the array. capture a smart log run smart LONG tests on the drive wait and see. capture a smart log and diff with previous. If this cannot be done through the arc-1200, I might connect it to the motherboard or other direct controller to see. From what I remember you cannot access the data directly, but you can always access the smart information. You can also run badblocks in "readonly" mode.
February 21, 201313 yr Author At the moment the system is back again and both RAID volumes are accessed. Running a parity rebuild and backing up the cache disk. When this finishes I will take it apart and run SMART test on both disks. Will the RAID be broken when the disks are taken of the ARC1200?
February 21, 201313 yr At the moment the system is back again and both RAID volumes are accessed. Running a parity rebuild and backing up the cache disk. When this finishes I will take it apart and run SMART test on both disks. Will the RAID be broken when the disks are taken of the ARC1200? Yes the raid will be broken if the controller is powered on. At the very least jump into the BIOS and see what the smart logs say from there. I suppose you could power down the whole array, take one drive, disconnect it, put it in another PC and do the smart LONG test from there. (it will take hours). Then reconnect it and do the same with the other drive.
February 21, 201313 yr Author I break off the parity rebuild now and let the back uo of my cache volume finish. then i take out the two disk and hook them up to my test server and run a long SMART test on that server. When I use "screen" can I then run 2 smart tests together?
February 21, 201313 yr I break off the parity rebuild now and let the back uo of my cache volume finish. then i take out the two disk and hook them up to my test server and run a long SMART test on that server. When I use "screen" can I then run 2 smart tests together? Yes you can run two smart long tests together. Make sure you do not have unraid spinning down drives unexpectedly when doing these long tests. It will take a few hours.
February 21, 201313 yr Author I didn't to see the whole SMART in the ARC1200 BIOS only a few lines. There were no issues detected. Also the event log of the ARC1200 looks clean. Only RAID started and me logging in messages. Can't I configure the disk as pass through and the run the long SMART test on each of them?
February 21, 201313 yr I didn't to see the whole SMART in the ARC1200 BIOS only a few lines. There were no issues detected. Also the event log of the ARC1200 looks clean. Only RAID started and me logging in messages. Can't I configure the disk as pass through and the run the long SMART test on each of them? I believe you can, then if you want to go back as raid0/raid1, you will need to reconfigure the array. You WILL loose your settings for the split hybrid array. (at least I did when I first started messing with the arc-1200 years back).
February 21, 201313 yr Author Sounds logical that you loose the settings for split set-up. But I will check what settings I used and make a hardcopy of it. Thanks to IPMI I can do it without taking it all apart. I will see to it that no data gets written to the array in the meantime. Also the 2 disks on the ARC1200 won't be spundown by unraid in this case but I guess I will have to disable this in the ARC1200s BIOS? Probably the best way is to start the array in Maintenance Mode.
February 21, 201313 yr Author Wondering what to use RAID/Passthrough or JBOD? Also how to run a long SMART test?
February 21, 201313 yr Wondering what to use RAID/Passthrough or JBOD? Also how to run a long SMART test? I think I used JBOD. You can try either. SMART long test is according to the drive's size. I've seen 255 minutes. about 4-5 hours which is about right for 2TB.
February 21, 201313 yr Author Wondering what to use RAID/Passthrough or JBOD? Also how to run a long SMART test? I think I used JBOD. You can try either. SMART long test is according to the drive's size. I've seen 255 minutes. about 4-5 hours which is about right for 2TB. When I run smartctl --test=long I get: root@Tower1:~# smartctl --test=long /dev/sda smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net Extended Background Self Test has begun Please wait 0 minutes for test to complete. Estimated completion time: Thu Feb 21 18:50:32 2013 Use smartctl -X to abort test root@Tower1:~# 0 minutes can't be right, can it? Also unraid sees 2 separate 2 TB disks but with those funny ARC1200 names, not the Seagate names. Like: scsi-20004d927fffff800
February 21, 201313 yr Wondering what to use RAID/Passthrough or JBOD? Also how to run a long SMART test? I think I used JBOD. You can try either. SMART long test is according to the drive's size. I've seen 255 minutes. about 4-5 hours which is about right for 2TB. When I run smartctl --test=long I get: root@Tower1:~# smartctl --test=long /dev/sda smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net Extended Background Self Test has begun Please wait 0 minutes for test to complete. Estimated completion time: Thu Feb 21 18:50:32 2013 Use smartctl -X to abort test root@Tower1:~# 0 minutes can't be right, can it? Also unraid sees 2 separate 2 TB disks but with those funny ARC1200 names, not the Seagate names. Like: scsi-20004d927fffff800 0 minutes is wrong, but that doesn't mean it's not running. do a smart -a /dev/sda and check log if it is running. It's been so long since I worked with my arc-1200, I don't remember if the names were accurate or not. Did you use jbod or pass through?
February 21, 201313 yr Author I now use JBOD root@Tower1:~# smartctl -a /dev/sda smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net Device: Seagate ST2000DM001-1CH1 Version: R001 Serial number: W1E15L2Z Device type: disk Transport protocol: Fibre channel (FCP-2) Local Time is: Thu Feb 21 19:27:10 2013 CET Device supports SMART and is Enabled Temperature Warning Disabled or Not Supported SMART Health Status: OK Current Drive Temperature: 30 C Drive Trip Temperature: 25 C Manufactured in week 30 of year 2002 Specified cycle count over device lifetime: 4278190080 Accumulated start-stop cycles: 256 Elements in grown defect list: 0 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 0 0 0 0 0.000 0 write: 0 0 0 0 0 0.000 0 Non-medium error count: 0 Device does not support Self Test logging
February 21, 201313 yr Ahh, it looks like a SCSI drive instead of an ATA drive. These make me think the drive is OK. > Elements in grown defect list: 0 > SMART Health Status: OK Have you tried pass thru mode? I can't remember if one or the other gave you what looked like a SATA drive.
February 21, 201313 yr Author Ahh, it looks like a SCSI drive instead of an ATA drive. These make me think the drive is OK. > Elements in grown defect list: 0 > SMART Health Status: OK Have you tried pass thru mode? I can't remember if one or the other gave you what looked like a SATA drive. Yes, but gave comparable results. Drive also appeared as /dev/sda. My data drives are attached to M1015 cards and also appear as scsi drive. So, the disks are probably ok, what is next. Possibly a driver issue? As I reported earlier, these errors started when I went to unraid 5.0RC10.
February 21, 201313 yr Ahh, it looks like a SCSI drive instead of an ATA drive. These make me think the drive is OK. > Elements in grown defect list: 0 > SMART Health Status: OK Have you tried pass thru mode? I can't remember if one or the other gave you what looked like a SATA drive. Yes, but gave comparable results. Drive also appeared as /dev/sda. My data drives are attached to M1015 cards and also appear as scsi drive. So, the disks are probably ok, what is next. Possibly a driver issue? As I reported earlier, these errors started when I went to unraid 5.0RC10. How soon after upgrading? Are you virtualized or are you bare metal? If it were me, I woud still do a confidence long test on the drives via some other method. I.e. shutdown unRAID, maybe move a cable from one drive's controller to another so I could access it directly with smartctl to see if there are pending or reallocated sectors. After that I might compare the raw driver source code from one kernel version to another to see what changed. Then possibly drop back a version or so and see what occurs.
February 21, 201313 yr Author Yes, I will do that. I can easily pull the cable of the ARC1200 and put it on the SATA socket on the mainboard. I now run bare metal since the first occurence of this error when I was running virtualized. Will do some more testing tomorrow.
February 22, 201313 yr Author Placed the cables of the controller and placed them on the SATA ports on the mainboard. I pulled a SMART report of both disks and attached them. Hope you can make sense of them. I see some rather high RAW values here and there. Might be the Seagate way. Just started a LONG smart test, does it have any output? Will take over 3 and a half hours, 213/217 minutes. smart.zip
February 22, 201313 yr I don't see anything that stands out. sdb 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 87 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 97 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 99 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 sdc 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 The long test will take a long time. Those values are correct. Post another smart log after they are done. Chances are high the drives are good.
February 22, 201313 yr Author I was just comparing the 2 SMART reports and noticed that one drive has a high Command Timeout: 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 4295032849 while the other doesn't: 188 Command_Timeout 0x0032 100 094 000 Old_age Always - 105 What does that indicate? Loose connection?
February 22, 201313 yr I was just comparing the 2 SMART reports and noticed that one drive has a high Command Timeout: 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 4295032849 while the other doesn't: 188 Command_Timeout 0x0032 100 094 000 Old_age Always - 105 What does that indicate? Loose connection? Good catch, Frankly, I don't know the answer. I would suspect the cable, but I did not see any DMA CRC errors, do a google search and see what it returns.
February 22, 201313 yr Second set of SMART reports after LONG SMART test attached. These tests look good, check out the command time out on google and let us know what you find.
Archived
This topic is now archived and is closed to further replies.