March 14, 201016 yr I'm wondering why one of my 1.5TBs has half the reads of my other 1.5TBs. Even the one that is 59% empty has more reads. Note: The parity check is still going but it's only on the 2TB drives now, the 1.5TBs have spun down. This happened a couple of months ago, but my syslog was filled with disk errors for that disk. It turns out it was a SATA cable. This time there are 0 syslog errors and it seems the parity check is still completing with 0 errors? http://img718.imageshack.us/img718/5736/parity.jpg
March 14, 201016 yr I'm wondering why one of my 1.5TBs has half the reads of my other 1.5TBs. Even the one that is 59% empty has more reads. Note: The parity check is still going but it's only on the 2TB drives now, the 1.5TBs have spun down. This happened a couple of months ago, but my syslog was filled with disk errors for that disk. It turns out it was a SATA cable. This time there are 0 syslog errors and it seems the parity check is still completing with 0 errors? http://img718.imageshack.us/img718/5736/parity.jpg Actually, your screen print shows 16 parity errors... (unless they were there before you started the parity check and no subsequent errors have occurred on this check) The number of "reads" is apparently the number of physical "reads" to the disk. Different disk controllers buffer differently, and the buffer cache supplies some of the data needed, so the numbers will never match. (The disks may be on different disk controllers, with different drivers) The read/write statistics are interesting to show you which drives are active, but not terribly useful for specific numbers. Joe L.
March 14, 201016 yr Author I'm wondering why one of my 1.5TBs has half the reads of my other 1.5TBs. Even the one that is 59% empty has more reads. Note: The parity check is still going but it's only on the 2TB drives now, the 1.5TBs have spun down. This happened a couple of months ago, but my syslog was filled with disk errors for that disk. It turns out it was a SATA cable. This time there are 0 syslog errors and it seems the parity check is still completing with 0 errors? http://img718.imageshack.us/img718/5736/parity.jpg Actually, your screen print shows 16 parity errors... (unless they were there before you started the parity check and no subsequent errors have occurred on this check) I'm not sure how I missed that... Any idea why I have 16 sync errors? My last parity check was only 13 days ago with 0 errors. I've added a good 3TB of data since then and 3 hard drives. The drive is on the same controller as most of the others, and it has the same firmware and model #.
March 14, 201016 yr I'm not sure how I missed that... Any idea why I have 16 sync errors? No, since you did not include a syslog for analysis. My last parity check was only 13 days ago with 0 errors. That is good. Did you cleanly stop the array since then... or, rather a better question, did you have any non-clean shutdowns (loss of power) Any of the new drives (or old) could have sectors that are unreadable. You would need to grab a SMART report fron each to learn more. Or you could have a drive that is returning random bits occasionally, or memory that is flaky. Have you performed a memory test? (preferably overnight) is the memory voltage, timing, and clock speed set properly for your specific memory strips in your BIOS? Joe L.
March 14, 201016 yr Author My BIOS doesn't allow me to adjust the memory settings at all, so I can only assume they are correct. They pass 32 hours of memtest. All my drives show 0 pending/reallocted sectors. I've had 0 non-clean shutdowns. I just redid a parity sync and got some syslog errors: Mar 14 11:50:42 Server kernel: ata15.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Mar 14 11:50:42 Server kernel: ata15.00: failed command: READ DMA EXT Mar 14 11:50:42 Server kernel: ata15.00: cmd 25/00:78:97:d6:88/00:00:46:00:00/e0 tag 0 dma 61440 in Mar 14 11:50:42 Server kernel: res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Mar 14 11:50:42 Server kernel: ata15.00: status: { DRDY } Mar 14 11:50:42 Server kernel: ata15: hard resetting link Mar 14 11:50:48 Server kernel: ata15: link is slow to respond, please be patient (ready=0) Mar 14 11:50:52 Server kernel: ata15: SRST failed (errno=-16) Mar 14 11:50:52 Server kernel: ata15: hard resetting link Mar 14 11:50:58 Server kernel: ata15: link is slow to respond, please be patient (ready=0) Mar 14 11:51:02 Server kernel: ata15: SRST failed (errno=-16) Mar 14 11:51:02 Server kernel: ata15: hard resetting link Mar 14 11:51:08 Server kernel: ata15: link is slow to respond, please be patient (ready=0) Mar 14 11:51:37 Server kernel: ata15: SRST failed (errno=-16) Mar 14 11:51:37 Server kernel: ata15: limiting SATA link speed to 1.5 Gbps Mar 14 11:51:37 Server kernel: ata15: hard resetting link Mar 14 11:51:42 Server kernel: ata15: SRST failed (errno=-16) Mar 14 11:51:42 Server kernel: ata15: reset failed, giving up Mar 14 11:51:42 Server kernel: ata15.00: disabled Mar 14 11:51:42 Server kernel: ata15.00: device reported invalid CHS sector 0 Mar 14 11:51:42 Server kernel: ata15: EH complete Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] Unhandled error code Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] Result: hostbyte=0x04 driverbyte=0x00 Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] CDB: cdb[0]=0x28: 28 00 46 88 d6 97 00 00 78 00 Mar 14 11:51:42 Server kernel: end_request: I/O error, dev sdh, sector 1183372951 Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] Unhandled error code Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] Result: hostbyte=0x04 driverbyte=0x00 Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] CDB: cdb[0]=0x28: 28 00 46 88 d7 0f 00 03 90 00 Mar 14 11:51:42 Server kernel: end_request: I/O error, dev sdh, sector 1183373071 Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] Unhandled error code Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] Result: hostbyte=0x04 driverbyte=0x00 Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] CDB: cdb[0]=0x28: 28 00 46 88 da 9f 00 04 00 00 Mar 14 11:51:42 Server kernel: end_request: I/O error, dev sdh, sector 1183373983 Mar 14 11:51:42 Server kernel: sd 15:0:0:0: [sdh] Unhandled error code I'm getting tired of this crap. I've been replacing cables thinking that these errors were caused by them, but it might just be that some sata controller is randomly messing up. 7 sync errors this time and unRAID disabled disk7. I have no idea what to do. This is fustrating, i'm tired of rewiring my case every 2 weeks because of errors, and now im thinking that the cables weren't the problem and that it's just a random error that sometimes happens. The errors are not disk errors, because i've seen that same error across 6 different discs in the past 2 months (on both onboard sata controller and my sata cards). Replacing the SATA cables fixed it (or so I thought, unless I have 7 bad sata cables in 2 months). I also want to add that no matter how many times I restarted it would not redect disk7. I had to shut the server down, pull it out of the hotswap bay and then plug it back in. I'm not sure if that's what caused it to be redected as a new disc, or the cold boot. I had to use "restore" since the disk was fine and now i'm forced to do another parity check. These warnings also started to appear about a week ago, everytime the server is started: Mar 14 15:05:38 Server kernel: ACPI: EC: Look up EC in DSDT Mar 14 15:05:38 Server kernel: ACPI Warning for \_SB_._OSC: Return type mismatch - found Integer, expected Buffer (20090903/nspredef-1006) Mar 14 15:05:38 Server kernel: \_SB_:_OSC evaluation returned wrong type Mar 14 15:05:38 Server kernel: _OSC request data:1 6 Mar 14 15:05:38 Server kernel: ACPI: Interpreter enabled Mar 14 15:05:38 Server kernel: ACPI: (supports S0 S1 S5) Mar 14 15:05:38 Server kernel: ACPI: Using IOAPIC for interrupt routing Mar 14 15:05:38 Server kernel: ACPI: No dock devices found. Mar 14 15:05:38 Server kernel: ACPI Warning for \_SB_.PCI0._OSC: Return type mismatch - found Integer, expected Buffer (20090903/nspredef-1006) Mar 14 15:05:38 Server kernel: _OSC evaluation returned wrong type
March 15, 201016 yr Author Your BIOS may have buggy ACPI. Try booting with acpi=off in your syslinux.cfg I just replaced the SATA cable and disk7 is still doing this whenever I attempt to parity check... it's fine for awhile then it's like it stops detecting the drive. Could ACPI really cause this? I guess i'll give that a shot.
March 15, 201016 yr I would first check a SMART report for that drive, make sure it is healthy. Then I would reconnect it to a different SATA port, preferably on a different disk controller, and test again. Then I would connect it outside of the hotswap bay, perhaps the connectors there are loose or flaky. Edit: that single syslog excerpt does not show a cable error, rather a lost connection. It also could be a power issue ...
March 15, 201016 yr Author I would first check a SMART report for that drive, make sure it is healthy. Then I would reconnect it to a different SATA port, preferably on a different disk controller, and test again. Then I would connect it outside of the hotswap bay, perhaps the connectors there are loose or flaky. Edit: that single syslog excerpt does not show a cable error, rather a lost connection. It also could be a power issue ... Well i've disabled ACPI and so far my "reads" are synced up. It hasn't failed yet so i'm crossing my fingers that the cause was ACPI. Still.. not having ACPI is annoying. I'm not sure what could cause a loss of connection, even restarting the computer doesn't redetect the drive.... I have to cold boot. This has happened before on different sata controllers, with different drives. I'm not actually sure is faulty ACPI could cause all this.
March 15, 201016 yr Well i've disabled ACPI and so far my "reads" are synced up. It hasn't failed yet so i'm crossing my fingers Now it's a good time to get a SMART report on that disk, and post it, and also post the new syslog so we can have a look.
March 15, 201016 yr What about a power supply issue? If the errors happen across different disks on different controllers the only common things I can think of is power issues of memory issues. Could the power supply be having trouble supplying stable power? Seems you have it sorking fine on a cold boot but later it starts acting up. Just a thought.
March 15, 201016 yr Author What about a power supply issue? If the errors happen across different disks on different controllers the only common things I can think of is power issues of memory issues. Could the power supply be having trouble supplying stable power? Seems you have it sorking fine on a cold boot but later it starts acting up. Just a thought. I have a 850W Corsair HX850. One of the best power supplies you can get. This would honestly be the last thing in my system that I would blame. Well i've disabled ACPI and so far my "reads" are synced up. It hasn't failed yet so i'm crossing my fingers Now it's a good time to get a SMART report on that disk, and post it, and also post the new syslog so we can have a look. It's 100% not the disk. This has happened on other drives on other controllers in this system. But here you go: 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 218 178 021 Pre-fail Always - 4066 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 239 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1276 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 87 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 38 193 Load_Cycle_Count 0x0032 184 184 000 Old_age Always - 49456 (This is caused by the green's power saving features - smart misreads it as a load cycle. It's not accurate, some people claim they have 2,000,000 or higher and the drive is still working) 194 Temperature_Celsius 0x0022 114 109 000 Old_age Always - 36 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 40% and no errors so far after disabling ACPI. If it successfully completes i'll do another check just to make sure it finds no errors. Here's the syslog as of now, I restarted the system to disable ACPI so its clean log. Only 'warning' is "Mar 15 08:48:24 Server kernel: xor: automatically using best checksumming function: pIII_sse". Which i've seen in every syslog on this motherboard. No idea what it means and it has been there since day 1. syslog.txt
March 16, 201016 yr Author Still doing it. Out of ideas. Disk7 didn't error this time, but it's sometimes it doesn't. I had 8 sync errors and it doesn't even say which disc they were from. No errors/updates in syslog. Disk8 still has about 40% less reads than the other drives. The original sync goes at 52MB/s... the parity checks after that go at 82MB/s. I would assume thats normal since its reading instead of writing. I'll run it again but I'm sure i'll either get parity errors or disk7 will "disconnect". I'm not sure why disk 8 has less reads... It has 1.35TB of data on it, same as the others. It shouldn't have 40% less reads. I think thats the cause of the parity errors. By the way, can you use your server during the parity check? AKA write new files to it while the check is underway?
March 16, 201016 yr By the way, can you use your server during the parity check? AKA write new files to it while the check is underway? Yes, I have done this many a time while a parity check is running.
March 16, 201016 yr Author By the way, can you use your server during the parity check? AKA write new files to it while the check is underway? Yes, I have done this many a time while a parity check is running. Then i'm out of ideas, because thats the only thing I could think of that could cause sync errors. I have no idea where to go from here, or what to do. I have 17.5TB of unprotected data on here, and I don't know what would of started causing these issues. There's no way to tell what disc is causing this, or what sata controller. I'd have to troubleshoot removing 1 drive at a time, which would take days... even then theres the possibility that it's not a drive and it's a controller. If someone has a better idea than taking the disks out one at a time, and rerun parity each time until I get 0 errors than please let me know... i'm probably going to drop unRAID because it has been nonstop headaches. I have a hard time believing this is hardware related because it ran fine in windows, and as soon as I went to unraid it's been nonstop issues. So either a hardware conflict (theres other users using my exact system!), or an unraid issue. There's also the slight chance that something went faulty the same time I switched to unRAID.
March 16, 201016 yr By the way, can you use your server during the parity check? AKA write new files to it while the check is underway? Yes, I have done this many a time while a parity check is running. Then i'm out of ideas, because thats the only thing I could think of that could cause sync errors. I have no idea where to go from here, or what to do. I have 17.5TB of unprotected data on here, and I don't know what would of started causing these issues. There's no way to tell what disc is causing this, or what sata controller. I'd have to troubleshoot removing 1 drive at a time, which would take days... even then theres the possibility that it's not a drive and it's a controller. If someone has a better idea than taking the disks out one at a time, and rerun parity each time until I get 0 errors than please let me know... i'm probably going to drop unRAID because it has been nonstop headaches. I have a hard time believing this is hardware related because it ran fine in windows, and as soon as I went to unraid it's been nonstop issues. So either a hardware conflict (theres other users using my exact system!), or an unraid issue. There's also the slight chance that something went faulty the same time I switched to unRAID. Pull a smart report for EVERY drive in your system, save it to a text file and upload them here for us to look at. Run smart short tests on all the drives and see what they come back with. Then, turn off disk spin down and run a smart long test on all the drives, save the output to a text file and again post them here. When we get a chance we can look through the reports and see if we can find anything. I started having similar problems with my Abit AB9 Pro board. I had a couple disks drop out for no apparent reason, and it was a pain to get reports from. As soon as I moved them off of the one controller port everything magically worked again with the disk. I connected a known good and working disk that has given me ZERO problems since being in the computer to the suspected bad controller and like that it dropped out. It turned out to be the onboard SIL controller of the Abit AB9 Pro. I disconnected all the disk from it and reset the array to use on the disks on the motherboard SATA, everything magically worked fine again. I ended up buying a new board and giving the old one to a room mate. He knowns fully well that the SIL controller is suspect, but he will never have a need to use them anyway.
March 16, 201016 yr Author By the way, can you use your server during the parity check? AKA write new files to it while the check is underway? Yes, I have done this many a time while a parity check is running. Then i'm out of ideas, because thats the only thing I could think of that could cause sync errors. I have no idea where to go from here, or what to do. I have 17.5TB of unprotected data on here, and I don't know what would of started causing these issues. There's no way to tell what disc is causing this, or what sata controller. I'd have to troubleshoot removing 1 drive at a time, which would take days... even then theres the possibility that it's not a drive and it's a controller. If someone has a better idea than taking the disks out one at a time, and rerun parity each time until I get 0 errors than please let me know... i'm probably going to drop unRAID because it has been nonstop headaches. I have a hard time believing this is hardware related because it ran fine in windows, and as soon as I went to unraid it's been nonstop issues. So either a hardware conflict (theres other users using my exact system!), or an unraid issue. There's also the slight chance that something went faulty the same time I switched to unRAID. Pull a smart report for EVERY drive in your system, save it to a text file and upload them here for us to look at. Run smart short tests on all the drives and see what they come back with. Then, turn off disk spin down and run a smart long test on all the drives, save the output to a text file and again post them here. When we get a chance we can look through the reports and see if we can find anything. I started having similar problems with my Abit AB9 Pro board. I had a couple disks drop out for no apparent reason, and it was a pain to get reports from. As soon as I moved them off of the one controller port everything magically worked again with the disk. I connected a known good and working disk that has given me ZERO problems since being in the computer to the suspected bad controller and like that it dropped out. It turned out to be the onboard SIL controller of the Abit AB9 Pro. I disconnected all the disk from it and reset the array to use on the disks on the motherboard SATA, everything magically worked fine again. I ended up buying a new board and giving the old one to a room mate. He knowns fully well that the SIL controller is suspect, but he will never have a need to use them anyway. All smart reports are fine for all drives, no bad sectors, no pending sectors, no reallocated sectors. I never did a smart test long run on the drives, but I will later. These drives were all precleared and added to the unraid system. The entire system is 2 months old, so unless they went bad in the last 2 months than it's not the drives. I get a different # of errors each time, so I have a hard time believing it's related to bad sectors. Right now I just removed the 4 newest drives from my sytstem.. because I know it worked fine before these were added. I'll see if it correctly checks parity now... but it's going to take awhile because I have to parity sync.. then parity check. If this returns 0 errors than I know its one of the new disks causing this. I then will troubleshoot to find the exact drive, because i'm almost 100% certain the drives are fine - and this is a SATA controller or sata/power cable, or possibly even the hotswap bay. I'm using crappy SATA cables because I had a hard time finding quality SATA cables that fit my on my SATA cards. I can bet its just another bad cable, because i've already had like 5 bad cables. Disk 8 was in those 4, and it was the main reason why I started the thread. It's reads were half of the other drives during parity checks, which doesn't make sense because its on the same SATA controller and has the same amount of DATA. My logic is, disk8 or the sata cable is having issues. I surely hope it is not related to a hotswap bay or a sata card... I will cry if I have to RMA the case, i'd have to tear apart the entire server and it'd cost $40 to ship this thing.
March 16, 201016 yr What about a power supply issue? If the errors happen across different disks on different controllers the only common things I can think of is power issues of memory issues. Could the power supply be having trouble supplying stable power? Seems you have it working fine on a cold boot but later it starts acting up. Just a thought. I have a 850W Corsair HX850. One of the best power supplies you can get. This would honestly be the last thing in my system that I would blame. While that IS a great PSU, I doubt there has ever been a device made that did not have a few returns, DOA's, lemons, "made-on-Friday" units, etc. Good testing programs *should* find them, but the better the quality, the more likely a bean-counter is going to say "Our defect rate is so low that it is cheaper to replace one users device than test 1000 devices in-house". You may be right about that PSU, but you can't rule it out yet, especially since several of your symptoms still point to power as a suspect. The fact that you are getting these problems after adding more drives points to heat or power or new connector issues. The fact that the problems seem more likely when the system is under heavier load (while running parity builds or checks) again points to heat or power issues. The fact that it seems random points to power or memory issues. The fact that the new drives required new cables, connectors, and hot swap bays makes each of those a suspect. And new drives need additional power cabling and possibly splitters. Somewhere, Joe L talks about the problems he had with a power splitter, problems that were similar to yours I believe. A somewhat extreme test, not recommended in normal use, but might be helpful in special cases like this: with the machine running and open, and a tail -f --lines=50 /var/log/syslog running in a console, try playing with the drive power connections, especially any splitters. Try carefully shaking, bending, or stressing them slightly, while monitoring the console. A loose connection *should* result in 'exceptions' reported in the syslog. I still think one or more of the hot-swap bays are also strong suspects, which should be easy to test, by connecting the drives directly, either removed from any hot-swap bay, or moved to a known good bay. And check for vibration issues, could cause disconnections. Cheap cables are never recommended for SATA drives. Your one syslog had no problems at all, and the syslog piece above did not show a normal cable issue (with a BadCRC flag), but they could still have faulty connectors, which could be responsible for a disconnect, perhaps because of thermal expansion or vibrations. Perhaps you have other syslogs that show more errors? Also, nothing I have seen so far explains the sync errors. Only 'warning' is "Mar 15 08:48:24 Server kernel: xor: automatically using best checksumming function: pIII_sse". Which i've seen in every syslog on this motherboard. No idea what it means and it has been there since day 1. This is not a warning or error, just a false positive, probably because of "checksum", which used to only occur in ACPI checksum warning messages only. I confess to being responsible for that one, and at some point, I or someone else will need to update the syslog color-coding, and make some of those more selective. I have a hard time believing this is hardware related because it ran fine in windows, and as soon as I went to unraid it's been nonstop issues. Technically, it is not unRAID that is losing the drive, it is Linux that is reporting that the Linux kernel has lost contact with the drive. As to Windows, you probably did not have this many drives, cables, bays, and controllers in use. I do agree that it no longer sounds like bad drives. (but not impossible, the last batch with that 'load cycle' issue *could* be defective) On the original question, I don't see how the differing reads could be related to anything else here, at all, but that is just my own opinion. The rest of us often see very different numbers too, that do not appear to me to be completely explainable as yet.
March 16, 201016 yr This is not a warning or error, just a false positive, probably because of "checksum", which used to only occur in ACPI checksum warning messages only. I confess to being responsible for that one, and at some point, I or someone else will need to update the syslog color-coding, and make some of those more selective. RobJ I just added a line to code that as "black" in the set of files I'll be posting in a few minutes as unMENU 1.3.1. The new rule is near the top of the syslog_match.conf file so it will color the line before the other actual error rule would make it show red. First match wins, so the line will stay black. match_case||" checksumming"||black Joe L.
March 17, 201016 yr Author After much troubleshooting, I decided to remove just disk7 from the system. I have now parity synced, and then parity checked to verify it with 0 errors. I'm now 100% sure it's related to disk7, or the SATA cable/port that is hooked up to it. I will have to troubleshoot more to find out exactly what it is. I'm hoping that it's not the PCI-E card, or the hotswap bay. If anything I hope it's just a bad hard drive even though SMART claims it's fine. If the hotswap bay is bad than my entire case will have to be RMAed. I replaced the SATA cable and it still happened. My case uses 10 standard molex power connectors to power 20 hotswap bays, so i'd think that disk 8 would also be suffering issues if it's related to the power cable. I'm going to try the drive in another hotswap bay and see if it works, that will rule out the drive.
Archived
This topic is now archived and is closed to further replies.