[Solved] Can't transfer a big file to one of my shares.

[RobJ]

Anyhow, while waiting for my other warranty replacements, I ran preclear on the 3TB drive again. This time connected to the port (SATA3) that it will use after it is in the correct physical slot. In this case I moved it into the 3rd slot to replace disk2. It completed the 2nd preclear in 34 hours and 16 minutes. However, after running a SMART status report it shows there is 1 Current_Pending_Sector.

 

SMART Attributes Data Structure revision number: 16
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1

 

There were 0 pending sectors after the 1st preclear, which I think was run from a different port (SATA5 or SATA7). I'm going to try running preclear again. If it doesn't show 0 Current_Pending_Sectors after this preclear, what should I do?

 

This is VERY unusual!  A healthy drive, brand new, that has just successfully been precleared, should NEVER turn up a fresh bad sector.  In light of the other drive issues you have had, this seems way too coincidental.  With a healthy system, this is statistically ridiculous.  Which is making me think you may have bad power issues, which can cause all sorts of weird problems.  I have to suspect that either you have terrible power with numerous electrical spikes on the line, or a bad power supply.  I was tempted to suggest that the third onboard SATA port you connected to might be defective, but I cannot think of any mechanism by which that could cause a sector to go bad.  I also thought about over-heated southbridge chipset, which could lead to drive controller issues, but that also can't cause bad sectors.  The only possible choices are power line issues to the drive, or someone is gleefully sending you bad drives, or you are dropping the server while it is working!  Or perhaps kicking it or hammering on it?   

 

Then later you attached that 20MB syslog 'piece' (the all-important first part wasn't there!), with constant ATA bus issues, which certainly could also be caused by power problems.  That 20MB was unfortunately all the same, a series of exception handling lines concerning the drive sde, and these lines appear to be AFTER the drive has been disabled, and therefore completely useless.  The only info is that the SATA link speed and ATA access speed had been slowed to their minimums, also consistent with the system giving up on the drive.  Only the SATA link was active and working, but that seems to be the lower layer of drive communications.

 

You mentioned you have a new power supply, do you know if the various drive problems were happening before this new PSU?  I recommend replacing this one, at least for testing purposes.  A 500Watt power supply should be enough if it is a good quality one, and Corsair makes good ones.  But like all products, there are occasional lemons.  It would be a real shame for a new Corsair to be bad, but it is not impossible.  Also, do you have a decent UPS or surge protection for your server?

 

This shows what appear to be a problem with the "sde" drive. This is the 3TB WD30EZRX and it is connected to the SATA3 connection on my motherboard. Does "I/O error, dev sde, sector 3069147056" mean it is having problems writing to that sector and will result in a 'Current_Pending_Sector' added to my count?

 

I'm not sure what ata5.00 is. I'm assuming it means the SATA5 connection, which would be the connection for the new Hitachi drive. Or, maybe it is for Port05, which would be the connection for one of my older 1 TB drives that is currently my disk5 on the SATA6 connection.

 

In this case, I don't believe there's a direct relation with the I/O error and another Current Pending sector.  The system keeps asking for various sectors, but it doesn't matter what it asks for, since communication with the drive is down.

 

The symbol ata5 is assigned at boot time to the next ATA drive the system identifies, in this case it happens to be sde.  It has no relation to a particular SATA port or drive or SCSI symbol, could be different every boot.

[lionelhutz]

I can't recall if you made any hardware changes but the continuous problems you are having points to a hardware issue besides the drives. I have seen a few threads where a poor power supply caused bad sectors to appear.

[GreggP]

This is VERY unusual!  A healthy drive, brand new, that has just successfully been precleared, should NEVER turn up a fresh bad sector.  In light of the other drive issues you have had, this seems way too coincidental.  With a healthy system, this is statistically ridiculous.  Which is making me think you may have bad power issues, which can cause all sorts of weird problems.  I have to suspect that either you have terrible power with numerous electrical spikes on the line, or a bad power supply.  I was tempted to suggest that the third onboard SATA port you connected to might be defective, but I cannot think of any mechanism by which that could cause a sector to go bad.  I also thought about over-heated southbridge chipset, which could lead to drive controller issues, but that also can't cause bad sectors.  The only possible choices are power line issues to the drive, or someone is gleefully sending you bad drives, or you are dropping the server while it is working!  Or perhaps kicking it or hammering on it?   

 

Then later you attached that 20MB syslog 'piece' (the all-important first part wasn't there!), with constant ATA bus issues, which certainly could also be caused by power problems.  That 20MB was unfortunately all the same, a series of exception handling lines concerning the drive sde, and these lines appear to be AFTER the drive has been disabled, and therefore completely useless.  The only info is that the SATA link speed and ATA access speed had been slowed to their minimums, also consistent with the system giving up on the drive.  Only the SATA link was active and working, but that seems to be the lower layer of drive communications.

 

You mentioned you have a new power supply, do you know if the various drive problems were happening before this new PSU?  I recommend replacing this one, at least for testing purposes.  A 500Watt power supply should be enough if it is a good quality one, and Corsair makes good ones.  But like all products, there are occasional lemons.  It would be a real shame for a new Corsair to be bad, but it is not impossible.  Also, do you have a decent UPS or surge protection for your server?

 

The performance problems, which eventually led to discovering the pending sectors, started last Fall. When it reoccurred and more pending sectors were discovered on multiple drives, it was suggested that there could be a power issue and I was advised to replace my power supply, which I did. So the answer to your question is yes, the drive problems were happening before this new PSU. My server is not plugged into a circuit with a sophisticated power conditioner or UPS. The power cord from my server is plugged into a Tripp Lite Isobar Model 1B-8 (or it could be IB-8, can never tell whether the straight vertical line is an i or a 1). The Isobar says it has both a noise filter and surge suppressor. The Isobar is connected to an Electra Guard Electronic Surge Suppressor, which is plugged into and covers my duplex receptacle. I've had both of these for many years. I haven't really had any problems with power surges before, but you never know. My house was built in the 1920s. The original panel was replaced with a 100 Amp service over 30 years ago. I think the server is on a 15A circuit and there's not a lot of other things on that circuit except and couple 60W lights, my cable modem, router, hdhomerrun prime, hdhomerun dual and a couple netgear 8 port gigabit switches. At this time I also have a 15" flat panel LCD monitor connected to my server so I can more easily monitor what's happening during this problem.

[GreggP]

Both of my new 2TB drives, the Western Digital WD20EZRX and the HGST (Hitachi) Deskstar 5K2000 have finished preclearing. I ran the SMART status report and neither show any pending sectors. Hooray!

[GreggP]

The preclear for the 3TB WD30EZRX finished. However, according to the usage notes from unMenu's "myMain" screen, it says "Preclear Not Successful".

 

The SMART status report (attached) for this drive now shows 0 (zero) Current_Pending_Sectors.

 

Also, and I think this is pretty strange, the syslog is now empty. I haven't done anything to this computer since the last syslog I've posted (other than making sure the SATA and power connections were solid). I haven't restarted the server. It has been running unRAID and preclearing my 3 drives for the past couple days. So, why is my syslog empty?

 

I also attached copies of the preclear reports for this drive.

 

Why does it report preclear not successful?

 

This message:

Disk /dev/sde has NOT been successfully precleared==

Postread detected un-expected non-zero bytes on disk

is written in the preclear report.

 

What does "un-expected non-zero bytes on disk" mean?

 

Will I need to run another preclear?

Or, since the SMART report shows 0 pending sectors, is this 3TB WD30EZRX good to go?

 

Thanks for your help.

smart-disk2_sde_2013-03-09.txt

preclear_start__WD-WMC1T1316428_2013-03-09.txt

preclear_rpt__WD-WMC1T1316428_2013-03-09.txt

preclear_finish__WD-WMC1T1316428_2013-03-09.txt

[BobPhoenix]

The preclear report is telling you the same thing that myMain did.  The disk was NOT precleared successfully.  Maybe Joe L can tell you how to proceed now better but you need to run at least ONE more preclear that succeeds - in my opinion.

 

Here is where the preclear report tells you it didn't work:

========================================================================1.13
== invoked as: ./preclear_disk.sh -A /dev/sde
==
== Disk /dev/sde has NOT been successfully precleared
== Postread detected un-expected non-zero bytes on disk==
== Ran 1 cycle

So it looks like you have some bytes on disk that didn't get cleared successfully.  It is possible that was because of the pending sector.  Now that that is cleared up running another cycle might work.  But as I said Joe L could probably give you better advice.

[Joe L.]

The preclear report is telling you the same thing that myMain did.  The disk was NOT precleared successfully.  Maybe Joe L can tell you how to proceed now better but you need to run at least ONE more preclear that succeeds - in my opinion.

 

Here is where the preclear report tells you it didn't work:

========================================================================1.13
== invoked as: ./preclear_disk.sh -A /dev/sde
==
== Disk /dev/sde has NOT been successfully precleared
== Postread detected un-expected non-zero bytes on disk==
== Ran 1 cycle

So it looks like you have some bytes on disk that didn't get cleared successfully.  It is possible that was because of the pending sector.  Now that that is cleared up running another cycle might work.  But as I said Joe L could probably give you better advice.

Start with a memory test.  (if you've not performed one recently)

 

If the drive is returning non-zero bytes after being cleared, then something (disk, disk controller, power supply) is not correct and if you put that disk in service you'll get random parity errors.    Your choice.  Use it, or not. 

[GreggP]

First, I'll try running memtest again. Will have to reboot the server with the old 128MB usb flash drive because I still haven't been able to get memtest to run from either my 16GB or 4GB usb flash drives.

 

UPDATE: Now I can't get memtest to run from the 128MB usb flash drive either. I tried reformatting it in Windows 7, then I copied a backup of my image back onto the drive. Then the drive was no longer bootable. So I reformatted again, and then copied a new image of unRAID Server with the 'make_bootable.bat' file and ran the batch file. The 128MB usb flash drive is bootable, but still can't run memtest. It's frustrating because just a week or so ago I ran memtest from this drive and it worked. Memtest ran for over 24 hours and didn't report any memory errors or problems.

 

Since I can't run memtest on any of my flash drives, I'm still wondering if there's a problem with my USB controller or something.

 

Anyway, I decided to reboot my server with the 4GB flasn, which has my license key for the Pro version of unRAID. It also is running v5RC11, which someone recommended I upgrade to in an earlier post.

 

I'm running preclear again on the 3TB WD30EZRX.

 

Any ideas on why my syslog was suddenly empty?

[GreggP]

I'm considering a couple of other options:

 

1. Try to get an RMA for my new Corsair power supply and replace it with another one.

 

2. I've offered to sell my motherboard, CPU, heatsink and RAM on Craigslist for $50. Sell it and purchase a new mobo/CPU/RAM

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16813157329

http://www.newegg.com/Product/Product.aspx?Item=N82E16819115077

http://www.newegg.com/Product/Product.aspx?Item=N82E16820148539

 

I hate to keep spending money on this and I have limited funds, so it is not an appealing option. However, maybe that is the only way I can get a stable server that I can be confident will work as a reliable place to store my media. As it is, I'm afraid I'll be dealing with another round of performance issues and hard drive failures in the future, even if I get this 3TB drive cleared properly.

 

What would you guys do in this situation?

[GreggP]

The preclear of the 3TB WD30EZRX finished:

 

Preclear Successful

... Total time 33:41:37

... Pre-Read time 8:07:45 (102 MB/s)

... Zeroing time 7:10:17 (116 MB/s)

... Post-Read time 18:22:35 (45 MB/s)

 

It also reports 0 (zero) pending sectors.

 

I've attached a bunch of reports.

 

What next, how do I rebuild after this?

 

Do I assign the drives to the disks that were replaced? In my situation, I lost both disk2 and disk4. Then I run 'initconfig'?

 

Or do I run initconfig and then assign the drives, what comes first?

syslog-2013-03-11.txt

smart_disk2_SDE-2013-03-11.txt

preclear_rpt__WD-WMC1T1316428_2013-03-11.txt

preclear_finish__WD-WMC1T1316428_2013-03-11.txt

[GreggP]

After a search of the support forum and wiki, I think I know what to do next, so I went ahead and assigned my drives. All of the drives that were working fine in my last configuration were assigned to the same disk. The new drives that replaced the 2 bad drives (disk2 and disk4) were assigned to those disks. The new 3TB drive was assigned as my new parity drive, since it is the largest drive.

 

Next, I ran initconfig.

 

Then I picked the button to start the array, but before that would work, it indicated that the 2 new drives would need to be formatted, which I did. Then, I believe it started the array and calculated parity. I'm not exactly sure what happened, because I've been gone for the past 12 hours.

 

Also, during this ordeal it was advised that I upgrade from v4.7 to v5RC11, so I did. However, I'm not familiar with all the new features in v5RC11, so I don't know if I'm following the correct procedures. The new web console is different. After completing the parity calculations it shows there are no errors. However, it also shows that a Parity-Check has not been run. So I am currently running a parity-check.

 

After the parity check completes, I will restore the files that were backed up during this ordeal. Hopefully, performance will be back to normal and then I will edit the thread subject name to indicate that it was solved.

 

Thanks to everyone that provided advice during this long confusing problem. And thanks for being patient with me. I'm still not absolutely sure there still aren't some underlying hardware issues. After replacing several hard drives, my flash drive, the power supply and securing all the connections, I feel a lot better about my server than I did a few days ago. So I'm finally going to use my server for the first time in almost a month.

 

If I have any questions about the transition from v4.7 to v5 and I can't find the answers in the wiki or other support threads, I'll start a new thread, so this one won't get extended any longer.

 

Thanks again!

[mr-hexen]

The preclear of the 3TB WD30EZRX finished:

 

Preclear Successful

... Total time 33:41:37

... Pre-Read time 8:07:45 (102 MB/s)

... Zeroing time 7:10:17 (116 MB/s)

... Post-Read time 18:22:35 (45 MB/s)

 

 

this is in line with what my two 3TB drives did for preclear. just adding.

[GreggP]

I'm not sure if the original problem with file transfer performance is solved, yet. I'm restoring files to my server that were backed up when I started having these drive failures. So I'm copying files across my network from several computers that were used to temporarily store my media files.

 

My syslog (also attached to this message) is now showing the following errors:

 

Mar 13 11:34:08 Media kernel: ata5: SError: { UnrecovData Handshk } (Errors)
Mar 13 11:34:08 Media kernel: ata5.00: failed command: WRITE DMA EXT (Minor Issues)
Mar 13 11:34:08 Media kernel: ata5.00: cmd 35/00:00:60:d5:6a/00:04:37:00:00/e0 tag 0 dma 524288 out (Drive related)
Mar 13 11:34:08 Media kernel: res 50/00:00:5f:d5:6a/00:00:37:00:00/e0 Emask 0x10 (ATA bus error) (Errors)
Mar 13 11:34:08 Media kernel: ata5.00: status: { DRDY } (Drive related)
Mar 13 11:34:08 Media kernel: ata5: hard resetting link (Minor Issues)

 

Are these an indication that I'm still having serious hardware problems? I've run SMART status reports for all of my drives and none of them are showing pending sectors. The file transfers across my network seem to be running at a reasonable speed. Most are in the 30 MB/sec range. However, a file transfer from within the server - from one internal disk to another is pretty slow 4.61 MB/sec.

 

What drive is ata5 referring to?

 

UPDATE: I think I identified the disk that was having communication problems. When I transferred files to my disk2, it was displaying the errors on ata5. Disk2 is the hard drive that was formerly used as my parity drive. It is physically located in the top (or 1st) slot on my server and is connected to my SATA1 port. I've never had any problems with this disk or the connection before, so it is odd that it is causing problems now. Disk2 is not showing any pending sectors in its SMART status report. Anyhow, I shut down the server, replaced the SATA cable for disk2 and again made sure the connections to all my drives were good. After restarting my server and resuming the file transfers, I haven't seen any errors.

syslog-2013-03-13.zip

[RobJ]

In this case, ata5 is referring to the third onboard SATA port (again!), which is connected to your parity drive, sde, the new 3TB WD30EZRX.  What seems significant is that the same errors are occurring, and they seem controller related, not cable or drive related, and this is the third onboard SATA port again, as it was in my previous post.  It seems very possible that at least some of your recurring problems could be related to a bad onboard port.  Try skipping it, connecting any where else you can.

 

Unfortunately, the very first error was also on this drive, but it was different - a media error with UNC flag raised, meaning a bad sector was found on your parity drive.  There is only one occurrence, but you should check the parity drive SMART report (after moving the drive to a different port!)  You must be VERY tired of this!

[GreggP]

Yes, this is getting very tiresome...

 

BTW, after swapping my SATA cable and tightening my connections, I no longer get the constant string of errors you see in my syslog from 3/13/13. However, now I get the occassional error on ata7.

 

Mar 13 12:34:35 Media kernel: ata4.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA (Drive related)
Mar 13 12:34:35 Media kernel: ata7.00: ATA-8: Hitachi HDS5C3020ALA632, ML6OAA10, max UDMA/133 (Drive related)
Mar 13 12:34:35 Media kernel: ata7.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA (Drive related)
Mar 13 12:34:35 Media kernel: ata7.00: failed to IDENTIFY (I/O error, err_mask=0x100) (Errors)
Mar 13 12:34:35 Media kernel: ata7.00: revalidation failed (errno=-5) (Minor Issues)
Mar 13 12:34:35 Media kernel: Probing IDE interface ide1...

 

and

 

Mar 13 12:34:35 Media kernel:  sde: sde1 (Drive related)
Mar 13 12:34:35 Media kernel: sd 4:0:0:0: [sde] Attached SCSI disk (Drive related)
Mar 13 12:34:35 Media kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) (Drive related)
Mar 13 12:34:35 Media kernel: ata7.00: failed to IDENTIFY (I/O error, err_mask=0x100) (Errors)
Mar 13 12:34:35 Media kernel: ata7.00: revalidation failed (errno=-5) (Minor Issues)
Mar 13 12:34:35 Media kernel: ata7: limiting SATA link speed to 1.5 Gbps (Drive related)
Mar 13 12:34:35 Media kernel: ata7.00: limiting speed to UDMA/133:PIO3 (Minor Issues)
Mar 13 12:34:35 Media kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310) (Drive related)

 

What does "failed to IDENTIFY (I/O error, err_mask=0x100)" mean?

 

According to http://lime-technology.com/wiki/index.php/The_Analysis_of_Drive_Issues#Drive_interface_issue_.234:

 

Drive interface issue #4

This is an example of what is probably a loose backplane or cable issue: (could be either the SATA connection or the power connection or both)

ata7.00: exception Emask 0x10 SAct 0x7 SErr 0x990000 action 0xa frozen

ata7.00: irq_stat 0x00400000, PHY RDY changed

ata7: SError: { PHYRdyChg 10B8B Dispar LinkSeq }

ata7.00: cmd 60/48:00:af:1b:97/00:00:10:00:00/40 tag 0 ncq 36864 in

        res 40/00:10:87:5f:96/00:00:10:00:00/40 Emask 0x10 (ATA bus error)

ata7.00: status: { DRDY }

ata7: hard resetting link

ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)

ata7.00: qc timeout (cmd 0xec)

ata7.00: failed to IDENTIFY (I/O error, err_mask=0x4)

ata7.00: revalidation failed (errno=-5)

ata7: failed to recover some devices, retrying in 5 secs

Note: There are no CRC errors here, which normally implicate a bad cable or two.

These problems are often related to a backplane, perhaps loose, perhaps vibration-related, perhaps defective. If the SATA link remains up for awhile, but communications are clearly bad, then the emphasis should probably be on the power connection. The easiest way to test whether it is the fault of the backplane is to reinstall the drive outside of the backplane.

If there is no backplane involved, then the same considerations apply to the cable connections, each end of both the SATA and power cables, including any power cable splitters that may be involved. It is common after opening a computer case, to jostle the cables, and SATA cables are notorious for coming loose, if they aren't the locking type. It is a good habit to check all SATA connections just before closing a case up.

Good quality SATA and power cables and splitters are strongly recommended. Always make certain that they are firmly connected, and not subject to vibration. The same is even more important for backplanes, make sure that drives are firmly and well seated in their trays, and cannot be vibrated loose.

 

I don't have a backplane, so that doesn't apply here. I will double check my connections, again. My case is an Antec Nine Hundred. It has a lot of fans. The drive bay has 3 cages (each cage holds 3 drives) with a 120mm fan in front of each cage. Then there's another 120mm fan out the back and a 200mm out the top of the case. Maybe 5 fans are creating too much vibration, not to mention the fans for the CPU cooler and power supply. At least my drives stay cool  . BTW, I've used this case for my unRAID server since November, 2008 and up until the past few months, I haven't had any problems with bad connections.

 

How did you identify which ata port is assigned to which SATA port? In other words, how did you know ata5 is referring to the 3rd onboard SATA port? What is ata7 referring to?

 

I've attached the latest syslog since my last reboot.

syslog-2013-03-14.txt

[RobJ]

BTW, after swapping my SATA cable and tightening my connections, I no longer get the constant string of errors you see in my syslog from 3/13/13. However, now I get the occasional error on ata7.

 

Mar 13 12:34:35 Media kernel:  sde: sde1 (Drive related)
Mar 13 12:34:35 Media kernel: sd 4:0:0:0: [sde] Attached SCSI disk (Drive related)
Mar 13 12:34:35 Media kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) (Drive related)
Mar 13 12:34:35 Media kernel: ata7.00: failed to IDENTIFY (I/O error, err_mask=0x100) (Errors)
Mar 13 12:34:35 Media kernel: ata7.00: revalidation failed (errno=-5) (Minor Issues)
Mar 13 12:34:35 Media kernel: ata7: limiting SATA link speed to 1.5 Gbps (Drive related)
Mar 13 12:34:35 Media kernel: ata7.00: limiting speed to UDMA/133:PIO3 (Minor Issues)
Mar 13 12:34:35 Media kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310) (Drive related)

Mar 13 12:34:35 Media kernel: ata7.00: configured for UDMA/133

 

What does "failed to IDENTIFY (I/O error, err_mask=0x100)" mean?

 

I added the additional line above, which shows that this time it was able to identify and establish communications at UDMA/133, whereas the first time it failed.  The difference being that it had slowed the SATA link to 1.5 Gbps.  Apparently, this drive (sdg, Disk 4) claimed to support 3.0 Gbps (SATA II), but was unsuccessful until it reverted to SATA I.  The identical drive (sdf, Disk 3) had no problems with SATA II.  Perhaps the cable is not good enough for Disk 4?  Seems strange though, there are no cable errors evident (other than it didn't work at the faster speed!).  Or there could be an interference issue (power cable too close?) that disallowed the faster speed on that cable.

 

I believe "failed to identify" means one side queried "who are you, identify yourself", and the other party did not respond.  That would be a problem!

 

I don't have a backplane, so that doesn't apply here. I will double check my connections, again. My case is an Antec Nine Hundred. It has a lot of fans. The drive bay has 3 cages (each cage holds 3 drives) with a 120mm fan in front of each cage. Then there's another 120mm fan out the back and a 200mm out the top of the case. Maybe 5 fans are creating too much vibration, not to mention the fans for the CPU cooler and power supply. At least my drives stay cool  . BTW, I've used this case for my unRAID server since November, 2008 and up until the past few months, I haven't had any problems with bad connections.

 

At the time I wrote that, there had been a number of backplane issues, so they were on my mind then.  I should have not focused so much on just them.  I'll try to generalize that better.

 

[ran out of time, more later ...]

[RobJ]

How did you identify which ata port is assigned to which SATA port? In other words, how did you know ata5 is referring to the 3rd onboard SATA port? What is ata7 referring to?

 

I've made some small improvements to The Analysis of Drive Issues, but much more still needed.

 

More importantly, I've started a new page Drive Symbols, which is a start to answering your question.  Unfortunately, while it probably *looks* like it is way too much info as an answer, it is only about a quarter of a full answer!  But hopefully, it provides the basics of the process.  The example is actually from your syslog, so more directly answers your question!