June 10, 201313 yr Author A bit late to this thread but I had a similar problem in the same case and same hardware with the same Seagate 3TB drives. The drives would drop from the array, usually within a few hours after I had put them in. This would happen every time I tried to swap them in with the unraid VM shutdown...until I powered off the entire server and then loaded the drives in with it powered off. Since then I've not had a single problem with the drives. I chalked it up to some kind of firmware issue with hot swapping on the drives and/or that Norco backplanes. That may be what's happening here; I shut down the hardware when I was adding the drives and no issues so far; I have 3 of the 5 back in, parity completed fine, and I am still writing 1.5G to each drive. I will leave it up for 24 hours after the writes finish to see if any of the drives go bad. If they don't I'll add in the other 2, run a parity, put data on them and let the 5 of them soak. If that works I'll move the server to the VM and soak the system.
June 12, 201313 yr Author Ok, all 5 Seagates in, with about 13 more Reds and no issue. Last time I saw the errors while I was preclearing some Reds so I have the array up while I preclear the last 4 Reds I have.
June 15, 201313 yr Author So I precleared the 4 drives, put them in the array, no issues. I decide to run a parity for the hell of it and immediately one of the seagates redballs on me. I can't see anything in the log though that tells me why. Suggestions on what to look for related to a redball? It's Disk 3, /dev/sdm, ST3000DM001-1CH166_Z1F295KQ. Syslog attached. syslog.zip
June 15, 201313 yr Author Results of smartctl on the drive. root@ShepStor1:~# smartctl --all /dev/sdm smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: ST3000DM001-1CH166 Serial Number: Z1F295KQ Firmware Version: CC24 User Capacity: 3,000,592,982,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Sat Jun 15 11:22:11 2013 CDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 600) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 46330328 3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 45 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 053 049 030 Pre-fail Always - 64427976659 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 562 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 27 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 098 098 000 Old_age Always - 2 190 Airflow_Temperature_Cel 0x0022 070 055 045 Old_age Always - 30 (Min/Max 28/30) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 29 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 100 194 Temperature_Celsius 0x0022 030 045 000 Old_age Always - 30 (0 21 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 60580513710231 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 20571466168 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 79185038344 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@ShepStor1:~#
June 15, 201313 yr Author Disk reads just fine, able to copy data off it just fine. Also no issues writing data to the disk, still redballed though.
June 15, 201313 yr From UnRAID's perspective, that's a failed disk (as you know). You are NOT actually reading from it; or writing to it. You're reading/writing to the rest of the array, which is emulating the failed disk via the parity computations. You need to replace the disk -- until you do, you're running "at risk" with no parity protection anywhere else ... lose another disk and you'll lose data. You MAY be able to just reseat the cables on the disk (disconnect, then reconnect it) ... it may then rebuild it from parity and all will be fine. If not, you'll need a replacement disk for it. Alternatively, you could just copy all of the data off of it; then do a "New Config" and simply not assign that drive to your new config. The system will then rebuild the parity drive ... and then you'll be ready to go.
June 15, 201313 yr Author Ugh, that sucks. Thanks for the info on the reading though, I forgot about that. Let me see if I can get it to redball again. Thanks again Gary for the help.
June 15, 201313 yr Author So I stopped the array, removed the disk from the array listing (left it in the drive bay), restarted the array, bam, I reproduced the original issue. 3 different disks went nuts (see pic). Will post syslog next to see if people see anything. God this is so frustrating. Note that these drives are in different places than the original config, and Reds in their previous spots are fine.
June 15, 201313 yr So I stopped the array, removed the disk from the array listing (left it in the drive bay), restarted the array, bam, I reproduced the original issue. 3 different disks went nuts (see pic). Will post syslog next to see if people see anything. God this is so frustrating. Note that these drives are in different places than the original config, and Reds in their previous spots are fine. The picture you just posted looks fine. It shows the drive being rebuilt (current status 0.3%). What SHOULD happen is that in ~ 10-12 hours it should turn green. (A bit longer than your parity checks take)
June 15, 201313 yr I'd just be patient at this point -- don't use the array; just update the display from time-to-time and track the progress of the rebuild.
June 15, 201313 yr Author I think the screenshot is too large; if you scroll right you see the issue I am talking about. Attached is the syslog; here is the relevant issue for disk 1. Jun 15 12:08:08 ShepStor1 kernel: sd 0:0:4:0: [sdg] Synchronizing SCSI cache Jun 15 12:08:10 ShepStor1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01) Jun 15 12:08:10 ShepStor1 kernel: sd 0:0:4:0: [sdg] Unhandled error code Jun 15 12:08:10 ShepStor1 kernel: sd 0:0:4:0: [sdg] Result: hostbyte=0x01 driverbyte=0x00 Jun 15 12:08:10 ShepStor1 kernel: sd 0:0:4:0: [sdg] CDB: cdb[0]=0x28: 28 00 00 0b dd 20 00 04 00 00 Jun 15 12:08:10 ShepStor1 kernel: end_request: I/O error, dev sdg, sector 777504 Jun 15 12:08:10 ShepStor1 kernel: md: disk1 read error, sector=777440 Jun 15 12:08:10 ShepStor1 kernel: md: multiple disk errors, sector=777440 Jun 15 12:08:10 ShepStor1 kernel: md: disk1 read error, sector=777448 Jun 15 12:08:10 ShepStor1 kernel: md: multiple disk errors, sector=777448 A bit later on this pops up: Jun 15 12:08:10 ShepStor1 kernel: md: disk1 read error, sector=777544 Jun 15 12:08:10 ShepStor1 kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Jun 15 12:08:10 ShepStor1 kernel: md: multiple disk errors, sector=777544 Jun 15 12:08:10 ShepStor1 kernel: sd 0:0:4:0: [sdg] Unhandled error code Jun 15 12:08:10 ShepStor1 kernel: md: disk1 read error, sector=777552 Jun 15 12:08:10 ShepStor1 kernel: sd 0:0:4:0: [sdg] Result: hostbyte=0x01 driverbyte=0x00 Jun 15 12:08:10 ShepStor1 kernel: md: multiple disk errors, sector=777552 Jun 15 12:08:10 ShepStor1 kernel: sd 0:0:4:0: [sdg] CDB: Jun 15 12:08:10 ShepStor1 kernel: sd 0:0:4:0: [sdg] md: disk1 read error, sector=777560 Jun 15 12:08:10 ShepStor1 kernel: cdb[0]=0x28Result: hostbyte=0x01 driverbyte=0x00 Jun 15 12:08:10 ShepStor1 kernel: :md: multiple disk errors, sector=777560 Jun 15 12:08:10 ShepStor1 kernel: 28 00md: disk1 read error, sector=777568 Jun 15 12:08:10 ShepStor1 kernel: 00 0b e1 20 00md: multiple disk errors, sector=777568 Jun 15 12:08:10 ShepStor1 kernel: 04 00md: disk1 read error, sector=777576 Jun 15 12:08:10 ShepStor1 kernel: 00 Jun 15 12:08:10 ShepStor1 kernel: end_request: I/O error, dev sdg, sector 778528 Jun 15 12:08:10 ShepStor1 kernel: md: multiple disk errors, sector=777576 Jun 15 12:08:10 ShepStor1 kernel: md: disk1 read error, sector=777584 Jun 15 12:08:10 ShepStor1 kernel: BUG: unable to handle kernel md: multiple disk errors, sector=777584 Jun 15 12:08:10 ShepStor1 kernel: md: disk1 read error, sector=777592 Jun 15 12:08:10 ShepStor1 kernel: md: multiple disk errors, sector=777592 Jun 15 12:08:10 ShepStor1 kernel: md: disk1 read error, sector=777600 Jun 15 12:08:10 ShepStor1 kernel: md: multiple disk errors, sector=777600 About a minute later the kernel reports a problem and then disk5 goes belly up Jun 15 12:09:03 ShepStor1 kernel: NULL pointer dereference at 000000a0 Jun 15 12:09:03 ShepStor1 kernel: IP: [<c11a8854>] sg_scsi_ioctl+0xed/0x286 Jun 15 12:09:03 ShepStor1 kernel: *pdpt = 00000000027be001 *pde = 0000000000000000 Jun 15 12:09:03 ShepStor1 kernel: Oops: 0002 [#1] SMP Jun 15 12:09:03 ShepStor1 kernel: Modules linked in: md_mod xor sg ahci libahci coretemp hwmon e1000e i2c_i801 i2c_core igb mpt2sas scsi_transport_sas raid_class [last unloaded: md_mod] Jun 15 12:09:03 ShepStor1 kernel: Jun 15 12:09:03 ShepStor1 kernel: Pid: 22357, comm: smartctl Not tainted 3.4.36-unRAID #1 Supermicro X9SCL-II/X9SCM-II/X9SCL-II/X9SCM-II Jun 15 12:09:03 ShepStor1 kernel: EIP: 0060:[<c11a8854>] EFLAGS: 00210247 CPU: 6 Jun 15 12:09:03 ShepStor1 kernel: EIP is at sg_scsi_ioctl+0xed/0x286 Jun 15 12:09:03 ShepStor1 kernel: EAX: 00000000 EBX: 00000000 ECX: f7752de8 EDX: f7752df4 Jun 15 12:09:03 ShepStor1 kernel: ESI: 00000006 EDI: 00000024 EBP: f6831e04 ESP: f6831d70 Jun 15 12:09:03 ShepStor1 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Jun 15 12:09:03 ShepStor1 kernel: CR0: 8005003b CR2: 000000a0 CR3: 37691000 CR4: 000407f0 Jun 15 12:09:03 ShepStor1 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Jun 15 12:09:03 ShepStor1 kernel: DR6: ffff0ff0 DR7: 00000400 Jun 15 12:09:03 ShepStor1 kernel: Process smartctl (pid: 22357, ti=f6830000 task=f2904020 task.ti=f6830000) Jun 15 12:09:03 ShepStor1 kernel: Stack: Jun 15 12:09:03 ShepStor1 kernel: 00000054 f6831df4 0000005d f6828800 f7752c00 00000024 ece00840 00000000 Jun 15 12:09:03 ShepStor1 kernel: 12d54f00 00000002 00200246 00000006 00000000 00000000 00000051 00000000 Jun 15 12:09:03 ShepStor1 kernel: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Jun 15 12:09:03 ShepStor1 kernel: Call Trace: Jun 15 12:09:03 ShepStor1 kernel: [<c11a8d91>] scsi_cmd_ioctl+0x330/0x368 Jun 15 12:09:03 ShepStor1 kernel: [<c106248d>] ? filemap_fault+0x75/0x306 Jun 15 12:09:03 ShepStor1 kernel: [<c1087fad>] ? __dentry_open+0x16c/0x225 Jun 15 12:09:03 ShepStor1 kernel: [<c1062191>] ? unlock_page+0x40/0x43 Jun 15 12:09:03 ShepStor1 kernel: [<c11a8dfd>] scsi_cmd_blk_ioctl+0x34/0x3b Jun 15 12:09:03 ShepStor1 kernel: [<c124c1c4>] sd_ioctl+0x7c/0x9b Jun 15 12:09:03 ShepStor1 kernel: [<c124c148>] ? sd_check_events+0x15d/0x15d Jun 15 12:09:03 ShepStor1 kernel: [<c11a6021>] __blkdev_driver_ioctl+0x25/0x2e Jun 15 12:09:03 ShepStor1 kernel: [<c11a696e>] blkdev_ioctl+0x69e/0x6d0 Jun 15 12:09:03 ShepStor1 kernel: [<c1076670>] ? handle_mm_fault+0x129/0x138 Jun 15 12:09:03 ShepStor1 kernel: [<c10aad69>] block_ioctl+0x35/0x3b Jun 15 12:09:03 ShepStor1 kernel: [<c10aad34>] ? nr_blockdev_pages+0x3a/0x3a Jun 15 12:09:03 ShepStor1 kernel: [<c109562b>] do_vfs_ioctl+0x1ff/0x217 Jun 15 12:09:03 ShepStor1 kernel: [<c1095676>] sys_ioctl+0x33/0x4b Jun 15 12:09:03 ShepStor1 kernel: [<c134262d>] syscall_call+0x7/0xb Jun 15 12:09:03 ShepStor1 kernel: Code: 85 7c ff ff ff b9 10 00 00 00 83 7d 88 00 88 5d 8f 0f 95 c2 e8 8c 9c ff ff 89 c3 8a 45 8f c0 e8 05 0f b6 c0 0f b6 b0 08 d2 35 c1 <66> 89 b3 a0 00 00 00 8b 55 08 89 f1 83 c2 08 89 55 90 8b 83 9c Jun 15 12:09:03 ShepStor1 kernel: EIP: [<c11a8854>] sg_scsi_ioctl+0xed/0x286 SS:ESP 0068:f6831d70 Jun 15 12:09:03 ShepStor1 kernel: CR2: 00000000000000a0 Jun 15 12:09:03 ShepStor1 kernel: ---[ end trace be7661c4c8e8d5a2 ]--- Jun 15 12:09:03 ShepStor1 kernel: md: disk1 read error, sector=65688 Jun 15 12:09:03 ShepStor1 kernel: Buffer I/O error on device md3, logical block 8211 Jun 15 12:09:03 ShepStor1 kernel: lost page write due to I/O error on md3 Jun 15 12:09:03 ShepStor1 kernel: sd 0:0:10:0: [sdm] Synchronizing SCSI cache Jun 15 12:09:03 ShepStor1 kernel: sd 0:0:10:0: [sdm] Result: hostbyte=0x01 driverbyte=0x00 Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: removing handle(0x0014), sas_addr(0x5001e677b6f5dff2) Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: md: disk1 read error, sector=4406489400 Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: Buffer I/O error on device md1, logical block 550811175 Jun 15 12:09:03 ShepStor1 kernel: lost page write due to I/O error on md1 Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: handle(0x0014), ioc_status(0x0022) Jun 15 12:09:03 ShepStor1 kernel: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:162/_transport_set_identify()! Jun 15 12:09:03 ShepStor1 kernel: mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:5194/_scsih_add_device()! Jun 15 12:09:06 ShepStor1 kernel: scsi 0:0:17:0: Direct-Access ATA ST3000DM001-1CH1 CC24 PQ: 0 ANSI: 6 Jun 15 12:09:06 ShepStor1 kernel: scsi 0:0:17:0: SATA: handle(0x0014), sas_addr(0x5001e677b6f5dff2), phy(18), device_name(0x0000000000000000) Jun 15 12:09:06 ShepStor1 kernel: scsi 0:0:17:0: SATA: enclosure_logical_id(0x5001e677b6f5dfff), slot(18) Jun 15 12:09:06 ShepStor1 kernel: scsi 0:0:17:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) Jun 15 12:09:06 ShepStor1 kernel: scsi 0:0:17:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:17:0: Attached scsi generic sg6 type 0 Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:17:0: [sdx] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB) Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:17:0: [sdx] 4096-byte physical blocks Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:17:0: [sdx] Write Protect is off Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:17:0: [sdx] Mode Sense: 7f 00 10 08 Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:17:0: [sdx] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 15 12:09:06 ShepStor1 kernel: sdx: sdx1 Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:17:0: [sdx] Attached SCSI disk Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:13:0: [sdp] Synchronizing SCSI cache Jun 15 12:09:06 ShepStor1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01) Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:13:0: [sdp] Unhandled error code Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:13:0: [sdp] Result: hostbyte=0x01 driverbyte=0x00 Jun 15 12:09:06 ShepStor1 kernel: sd 0:0:13:0: [sdp] CDB: cdb[0]=0x28: 28 00 00 e4 7c e0 00 04 00 00 Jun 15 12:09:06 ShepStor1 kernel: end_request: I/O error, dev sdp, sector 14974176 Jun 15 12:09:06 ShepStor1 kernel: md: disk1 read error, sector=14974112 Jun 15 12:09:06 ShepStor1 kernel: md: disk5 read error, sector=14974112 syslog.zip
June 15, 201313 yr I think the screenshot is too large; if you scroll right you see the issue I am talking about. Ah ... yes I see the large error counts for Disk #5. I have no idea what to suggest at this point. You COULD wait and see if the rebuild actually completes; or you could copy all of the data off of Disk 3 (emulated) and then off of Disk 5; and then rebuild the config without either of them. But I know you've been experiencing a LOT of issues with this setup -- I skimmed through the thread again, and am running out of ideas ==> short of simply trashing all the Seagate drives and just using WD Reds
June 15, 201313 yr Shouldn't have left that last post like that ... clearly I don't really mean you should throw away all your Seagates. But I WOULD, since you have plenty of free space, copy all of the data off of Disk3; then do a New Config and use ONLY WD Reds ... then see if all is well. If that resolves things, then I'd pre-clear the Seagates again, and then, one-at-a-time, add them back into the array, writing a bunch of data to each one after it's added, and doing a full parity check after each one.
June 15, 201313 yr Author Oh I have thought about junking those seagates , my wife just gives me the evil eye when I say that as they are brand new and cost me $500. I am going to do as you suggest, rebuild the array without the seagates and see how it goes. Thanks for taking another look at the thread. If it goes well I will try jumping to RC15 since I believe it has a new kernal, maybe it will fix what ails me. If that doesn't help then I'll just leave the seagates out and figure out a use for 15T somewhere else.
June 15, 201313 yr Oh I have thought about junking those seagates , my wife just gives me the evil eye when I say that as they are brand new and cost me $500. I am going to do as you suggest, rebuild the array without the seagates and see how it goes. Thanks for taking another look at the thread. If it goes well I will try jumping to RC15 since I believe it has a new kernal, maybe it will fix what ails me. If that doesn't help then I'll just leave the seagates out and figure out a use for 15T somewhere else. If trashing $500 worth of drives sounds terrible (it is, of course), just remember that at least you still have those 15TB worth of drives you could use for other purposes. I'd suggest you use them for backups of your array ... just put them in an eSATA caddy attached to a Windows PC, and start methodically filling them up. I have 15 (and counting) drives stored in WiebeTech DriveBoxes that contain a complete backup of all the data on my media server That's well over $500 worth of drives (at least $2500). ... and when anyone complains about drive costs these days, I remind them that my first hard drive held a whopping 26MB (that M is not a typo) ... and cost $4500 !!
June 17, 201313 yr Author Here's an update. I re-built the array with Reds only, and it's going fine. I started preclearing the seagates again and after a few hours one of the preclear's stopped. I check the syslog and that drive stopped responding like before. What's really interesting is that after a while it was re-added to the disk list as a brand new drive, and then stopped working again. I've pulled the drive out of the system; I am starting to think that maybe this drive is bad and perhaps it was causing collisions with other drives. I did notice that the 2nd from last failure all the drives that went bad were on the same backplane and same M1015 port (although they were on other ports and backplanes when they failed before). The other 4 drives are continuing to preclear; once they are done I will do more tests. Hopefully the other failures I had were due to this drive. Time will tell.
June 17, 201313 yr Glad all's working well with the Reds. I've become a HUGE fan of these units -- very low power; very reliable; and very good performance. Tough to beat that combination. WD has announced both 4TB and 5TB versions that will be shipping towards the end of this year (4TB) and early next year (5TB). I'm already looking forward to building a nifty little 25TB server in a Lian-Li PC-Q25B case with the 5TB units
June 17, 201313 yr I read most of the thread, but not all of it, so don't know if this was already mentioned. What firmware do the Seagates have on them? My server is all Seagate 2TB and 3TB drives and I have never had this issue. I did do an available firmware update to them when I first installed them. I doubt it is the source of your issues, but just throwing it out there anyway.
June 17, 201313 yr Certainly wouldn't hurt to check the firmware revision on the drives and see if there's an update for the specific drives you have. As for "throwing away" the drives -- obviously NOT ... use them for backups !!
June 18, 201313 yr Author I read most of the thread, but not all of it, so don't know if this was already mentioned. What firmware do the Seagates have on them? My server is all Seagate 2TB and 3TB drives and I have never had this issue. I did do an available firmware update to them when I first installed them. I doubt it is the source of your issues, but just throwing it out there anyway. Yup, one of the first things I looked at; turns out the there are 2 versions of the 3T drives and I have the version that doesn't allow firmware upgrades. Thanks for the idea though.
June 23, 201313 yr Author I'm moving this to Solved; once I removed the Seagate drive having issues in preclear none of the other seagates had any issues. I've added them to the array and been using them for several days now with no issues. I'm RMAing the drive.
Archived
This topic is now archived and is closed to further replies.