Just started up a new unRAID today. I started copying over a bunch of data and about 5 minutes in unRAID stopped responding. I could no longer use the web interface. I've attached my syslog. Based on what I see (write errors) I'm assuming drives 0 and 1 are bad and should be replaced? Ridiculous that two of these drives are already bad. Anyone had experience with SAMSUNG HD103UI 1TB 5400 RPM 32MB Cache SATA 3.0Gb/s Hard Drive drives? Seems like the failure rate is very high.

 

Thanks for any help!

 

EDIT: File is here http://www.2shared.com/file/4977852/4a356a84/syslog.html as size was too large somehow

Actually drive0 seems to be my WD Green 1TB (parity) which I've had for a year or so and haven't had any problems with. Strange?

This was also on my SMART data history:

 

S1MNJ1CQA14386: *ERROR* - ATA_Error_Count has increased from (no data) to 2 since (no data)

S1MNJ1CQA14386: WARNING - UDMA_CRC_Error_Count has increased from (no data) to 4 since (no data)

S1MNJ1NQA25502: WARNING - ATA_Error_Count has increased from (no data) to 1 since (no data)

S1MNJ1NQA25503: WARNING - ATA_Error_Count has increased from (no data) to 1 since (no data)

Y65P690E: *ERROR* - ATA_Error_Count has increased from (no data) to 55364 since (no data)

Y65P690E: *ERROR* - ATA_Error_Count it is now 55364 (error threshold is 175)

Y65P690E: *ERROR* - Reallocated_Sector_Ct has increased from (no data) to 365 since (no data)

Y65P690E: *ERROR* - Reallocated_Sector_Ct it is now 365 (error threshold is 30)

Y65P690E: WARNING - Reallocated_Event_Count has increased from (no data) to 7 since (no data)

Y65P690E: *ERROR* - Offline_Uncorrectable has increased from (no data) to 8 since (no data)

Y65P690E: *ERROR* - Offline_Uncorrectable it is now 8 (error threshold is 5)

S13PJDWQA04076: *ERROR* - Current_Pending_Sector has increased from (no data) to 1 since (no data)

S13PJDWQA04076: WARNING - Current_Pending_Sector it is now 1 (warning threshold is 1)

6 device(s) active, 0 sleeping, 1 did not return SMART data.

 

 

I've just decided to remove Y65P690E from the server. It's an old 250gb drive that is obviously going bad. For some reason it was running at 55C. Nothing on it anyway.

 

The only other error in SMART that looks bad is that one drive is not returning any SMART data at all? What would cause that?

The only other error in SMART that looks bad is that one drive is not returning any SMART data at all? What would cause that?

 

From my understanding that is your unRAID USB key, it cannot and will not return SMART info.  I believe everyone that has the smarthistory program will have at least 1 device that will not return data (anything usb).

 

Cheers,

Matt

 

The drive issues reported in the syslog have no relation to the data in the SMART history report, so be careful in drawing conclusions from the SMART report.  That 250GB may have a number of remapped sectors, but there is no evidence as to when they occurred, so that info may be very old.  There is nothing here to indicate removing the 250GB, unless you really want to.  Also, since it is providing the initial read of the SMART values, BubbaQ may want to rework the formatting of these messages, perhaps to indicate the current or initial value, not comparing with a 'no data' set.  None of them are 'increases', just the initial display of the current numbers of note.  I suspect that BubbaQ may also want to rework the threshold displays, since they have no direct relation to the RAW_VALUEs they immediately follow (and ATA Error Count does not have a threshold).  Thresholds are related to VALUE, WORST, and the initial value of VALUE, usually 100 or 200 or 253 (vendor and model specific).

 

Now to the real problems, as indicated in the syslog.  Only the parity drive and Disk 1 are reporting problems, and it does not look like actual drive problems, but 'system' problems, possibly related to SWNCQ.  This is a hard one to be definitive about, but I don't see any sector related or mechanical issues, just timeouts and SWNCQ issues.  Prior to unRAID v4.4, I needed the swncq=0 boot option to avoid this same kind of error sequences in my system, but I was able to drop it, after upgrading to v4.4 and v4.4.2.  However, your problems look like you need to add the swncq=0 boot option, even though you have v4.4.2.  See the Boot Codes wiki page.  Since SWNCQ only applies to the onboard nVidia SATA ports, you might also consider reconnecting those drives to your other unused SATA ports.

The drive issues reported in the syslog have no relation to the data in the SMART history report, so be careful in drawing conclusions from the SMART report.  That 250GB may have a number of remapped sectors, but there is no evidence as to when they occurred, so that info may be very old.  There is nothing here to indicate removing the 250GB, unless you really want to.  Also, since it is providing the initial read of the SMART values, BubbaQ may want to rework the formatting of these messages, perhaps to indicate the current or initial value, not comparing with a 'no data' set.  None of them are 'increases', just the initial display of the current numbers of note.  I suspect that BubbaQ may also want to rework the threshold displays, since they have no direct relation to the RAW_VALUEs they immediately follow (and ATA Error Count does not have a threshold).  Thresholds are related to VALUE, WORST, and the initial value of VALUE, usually 100 or 200 or 253 (vendor and model specific).

 

Now to the real problems, as indicated in the syslog.  Only the parity drive and Disk 1 are reporting problems, and it does not look like actual drive problems, but 'system' problems, possibly related to SWNCQ.  This is a hard one to be definitive about, but I don't see any sector related or mechanical issues, just timeouts and SWNCQ issues.  Prior to unRAID v4.4, I needed the swncq=0 boot option to avoid this same kind of error sequences in my system, but I was able to drop it, after upgrading to v4.4 and v4.4.2.  However, your problems look like you need to add the swncq=0 boot option, even though you have v4.4.2.  See the Boot Codes wiki page.  Since SWNCQ only applies to the onboard nVidia SATA ports, you might also consider reconnecting those drives to your other unused SATA ports.

 

Thanks for your help! Currently 4 of my drives are connected to my onboard nvidia ports and the other 2 on a RocketRaid 1820A controller. After this post: http://lime-technology.com/forum/index.php?topic=3381.0 I was a little worried that it was the RocketRaid causing the issues so I actually disabled the two drives on that controller. I've managed to copy over ~500GB of data just fine (with just the 4 drives on the onboard controller) but it has finally hung after I tried to download a torrent onto it (mounting the drive in Windows + using uTorrent).

 

The onboard Nvidia is the nForce 650i (old Asus P5N-E) which I thought should be fine based on the Wiki? Anyway, I will try adding that boot code now. Would I be better off just connecting all of the drives to the RocketRaid? It has 8 ports so it wouldn't be an issue.

 

Again, thanks for the help!

Alright, I added the boot code and brought the two drives on the RocketRaid back online. Parity is rebuilidng. I'll report back if I experience any more issues.

Hoping the boot option will take care of it.  I only mentioned moving the drives off the nForce ports as a MUCH less desirable backup plan.  Another option, also less desirable, is to revert to unRAID v4.3.3, because the SWNCQ issues began with the kernel included in the first v4.4 betas.  Then wait for a future kernel release, with fixed support.  They do appear to be working on it, with progress being made.

Nope, no such luck. Server was fine when I get home but there hadn't been any activity on it all day. I started up a couple torrents and left for dinner. When I got back, server was frozen. I couldn't even telnet in to copy over the syslog. Will try to get a syslog when the problem occurs again.

I am sure that crashes like you are experiencing can happen for a variety of reasons.  The only one I know of first hand is that if a drive loses connection with the controller.  A hard crash is frequently the result.  The lost connection can be due to a bad data cable / connection, or because of a bad power connection.  Data cables can be bad and do GO bad over time.  I've had 2 fail on me in the past couple of years.  I've never had a power cable go bad, but other users have (see below).

 

The insidious problem with these types of connection problems is that they are intermittent.  You may have a bad cable that gets twisted just right and works just fine for months, and then when you mount a new drive jiggle it and it can start to give problems.  (Of course the new drive will get blamed!)  Other times users like you, which copying large files, find that the cable fails sometime in the billions of data transmissions over the cable.  On reboot it looks just fine again.  Many a drive gets replaced due to a bad connection.  The funny thing is sometime changing a drive FIXES the problem, but only because a different cable is used or the existing cable just happens to get a good connection this time, and is waiting to cause its mischief in the future.

 

Some users find that backplanes / docks, which offer a tremendous convenience factor, give problems.  If the problem is bad enough, the drive just won't get seen.  This is far better than a marginal connection which causes crashes and other nasty symptoms.  Backplanes introduce multiple cables segments and connections, any one of which can get lose and cause problems.

 

You will likely do a quick check and rule this out, but I would urge you to be thorough.  This is the most frequent cause of these types of problems.  One senior member of the forum (Joe L.) found a faulty power splitter that caused crashes.  He spent well over a week meticulously testing all his drives and replacing data cables and even his PSU before he found this unlikely culprit.  He was about ready to change out his motherboard.  I found a lose motherboard SATA port that required a locking data cable AND also a bad data cable.  It took me over a week as well, to find these two unrelated problems.  In the mean time I cleaned up all the power cabling inside my case to rule them out as a problem.  (In the month prior to my discovery of this problem, i replaced / rebuilt TWO apparently failed drives which were later found to be just fine - all because of my cabling problems).

 

I see no particular negative to moving back to 4.3.3 (unless their are hardware compatibility issues with your motherboard).  You might try that first but if your problems follow you, you have some marginal connection somewhere, and finding it may be tricky.

 

People reading this might ask how to avoid these types of problems.  Although sometimes they are unavoidable, neatness helps.  Use cable zip ties to route the cables.  Avoid rats nests of power splitters and data cab;es.  Try not to bend cables at sharp angles especially near the connectors. Also, don't be too quick to claim success after an array builds parity and seems to be working okay.  Do frequent parity checks for the first few weeks after a new build or substantial upgrade.

^^ Thanks for the help, I will try checking data cables tonight when I get home from work. I left the server running last night while copying a few files over (no heavy data transfer/torrents/etc).

 

When I awoke, 06:48:53 up  9:10,  1 user,  load average: 60.28, 60.08, 60.08. Not sure why the load is so high? The web interface takes forever to refresh but for whatever reason, the parity drive is powered down and flashing red (drive one is online).

 

Syslog is here: http://www.2shared.com/file/4994363/b7cff7bf/syslog.html

 

Lots of stuff like this:

 

Mar  3 06:52:54 unRAID kernel: md: disk1 write error

Mar  3 06:52:54 unRAID kernel: handle_stripe write error: 30594176/1,

Mar  3 00:16:24 unRAID kernel: md: disk0 read error

Mar  3 00:16:24 unRAID kernel: handle_stripe read error: 703097464/0, count: 1

Mar  3 00:16:24 unRAID kernel: md: disk1 read error

Mar  3 00:16:24 unRAID kernel: handle_stripe read error: 703097464/1,

 

Seems like it's not specific to any drive as really only drive0/1 are getting used and there are errors on both. Controller problems still?

So, I just brought it online and now it's saying that a new parity disk was installed?!? I still have the WD Green 1tb as the parity.

So, I just brought it online and now it's saying that a new parity disk was installed?!? I still have the WD Green 1tb as the parity.

 

That is almost certainly because it had 'dropped' the drive after all of the problems.  I am rather surprised it did not drop both Disk 1 and the parity drive.  I searched but could not find syslog lines indicating that, but that is the most likely explanation.

 

It appears that the swncq=0 changed absolutely nothing for you, as you have all of the same problems still, with the same drives.

 

In addition, there is a new problem, Disk 4 (sdg) had some media errors just before the other 2 drives began spitting out errors, so it needs a SMART report and test.

 

Since the drive errors on ata2 and ata3 (parity and Disk 1) still do not appear to be physical drive errors, and still appear related to NCQ, I have another suggestion, turn off NCQ for all of the drives.  Please see this post for a little qd tool, and extract it to your flash drive, and add qd off to the bottom of your go script.  That should effectively remove NCQ from the equation.  Try your same tests.  By the way, that was a good idea to return to stock unRAID for further testing.

 

One small coincidence, in both syslogs the drive problems began very close to 2 hours after booting, probably not noteworthy in itself, unless something else turns up that correlates with that.

So, I just brought it online and now it's saying that a new parity disk was installed?!? I still have the WD Green 1tb as the parity.

 

That is almost certainly because it had 'dropped' the drive after all of the problems.  I am rather surprised it did not drop both Disk 1 and the parity drive.  I searched but could not find syslog lines indicating that, but that is the most likely explanation.

 

It appears that the swncq=0 changed absolutely nothing for you, as you have all of the same problems still, with the same drives.

 

In addition, there is a new problem, Disk 4 (sdg) had some media errors just before the other 2 drives began spitting out errors, so it needs a SMART report and test.

 

Since the drive errors on ata2 and ata3 (parity and Disk 1) still do not appear to be physical drive errors, and still appear related to NCQ, I have another suggestion, turn off NCQ for all of the drives.  Please see this post for a little qd tool, and extract it to your flash drive, and add qd off to the bottom of your go script.  That should effectively remove NCQ from the equation.  Try your same tests.  By the way, that was a good idea to return to stock unRAID for further testing.

 

One small coincidence, in both syslogs the drive problems began very close to 2 hours after booting, probably not noteworthy in itself, unless something else turns up that correlates with that.

 

Thanks Rob. I will try adding the qd tool when I get home this evening and will also run the smartchk on drive 4.

 

I noticed the ~2 hr thing as well. It seems to function fine for a few hours after starting up and then suddenly the errors occur. Not sure what type of hardware issue would cause sudden errors like this. The only things I can think of are power/heat which I'm 99% sure aren't the problem. I believe I have my drives set to spin-down after 2 hours idle, but I doubt this would be a problem?

Disco- your syslog shares many similarities with mine:

 

http://pastebin.com/m25078f40

 

I too have tried with and without swncq = 0.  That boot code worked great, but 4.4.2 just isn't cooperating with my setup.  RobJ had recommended I do some tests pertaining to NCQ, so I will try the qd tool tonight as I have some time to test this finally.

 

Worst case is I'll revert to 4.3.3.  Good luck!

For some reason when I try to get the smart status of a disk I get this error:

 

root@unRAID:~# smartctl -a -d ata /dev/sdg
smartctl: error while loading shared libraries: libstdc++.so.6: cannot open                   ed object file: No such file or directory

 

Anyway, after running a parity check today there were ~270 errors or something on disk 4. I'm assuming this is trouble? To remove, I just remove it from the array and do a restore, correct? There's currently nothing on the drive.

 

I've put the qd script on my flash drive added added the qd off line to my go script. Will report back after I try copying over some files. The server was fine for the last ~8 hours, but there was no data transfer to it which probably explains that.

 

 

For some reason when I try to get the smart status of a disk I get this error:

 

root@unRAID:~# smartctl -a -d ata /dev/sdg
smartctl: error while loading shared libraries: libstdc++.so.6: cannot open                   ed object file: No such file or directory

 

Anyway, after running a parity check today there were ~270 errors or something on disk 4. I'm assuming this is trouble? To remove, I just remove it from the array and do a restore, correct? There's currently nothing on the drive.

 

I've put the qd script on my flash drive added added the qd off line to my go script. Will report back after I try copying over some files. The server was fine for the last ~8 hours, but there was no data transfer to it which probably explains that.

 

 

You are assuming wrong.  The disk could be fine. 

 

The current version of unRAID did not include a library file needed to run smartctl.  Easiest way to get the library file installed is from within the package manager in unMENU, It can be configured to install itself every time you reboot.  Get the unMENU interface here

 

or... follow instructions here: http://lime-technology.com/forum/index.php?topic=2817.msg23548#msg23548 and install the library file yourself every time you reboot.

 

Joe L.

Anyway, after running a parity check today there were ~270 errors or something on disk 4. I'm assuming this is trouble? To remove, I just remove it from the array and do a restore, correct? There's currently nothing on the drive.

 

Were those parity errors, from the bottom of the status page, or disk errors in the Errors column for Disk 4?

 

Sounds like a lot, but as bad as it is, I think I would ignore Disk 4 for now, until you can get the rest of the array behaving, get the parity drive and Disk 1 to stop spewing errors.  Do NOT remove Disk 4, because the array won't start, without additional problematic steps.  Try whatever process you did before (parity check?) that caused all of those errors, so we can see whether turning off NCQ will work.  Going back to v4.3.3 is the next plan, by copying the bzroot and bzimage from the v4.3.3 downloaded zip file to your flash drive and rebooting.  Then test again.  Once the array is stable, then you can pull Disk 4 or run SMART tests on it, etc.  [ Disclaimer:  I'm getting tired, so the quality of my comments is decreasing, and the chance of errors and oversights is increasing! ]

Thanks for your continued support!!

 

The errors were in the disk column, not below. I've left disk 4 in the array. I've been copying files over for about ~3 hours and so far the array seems great, load is fine and what not. I think I will let a parity check run over night to see how that goes.

 

I'm assuming there's no way to view the syslog without shutting the server down? I was going to check for errors but won't bother if it seems stable for now.

 

Once I determine stability I'll check out disk4.

No such luck, system messed up over night again. Log here: http://www.2shared.com/file/5001132/58d2f3d/syslog.html

 

I decided to just go ahead and downgrade to 4.3.3 via the method you described. Parity check running now. Fingers crossed.

If your experiment with 4.3.3. doesn't go well, you should simplify your setup to the greatest extent possible.  3 drives (parity + 2 data) that NO reallocated or pending reallocated sectors.  Attached them directly to the motherboard (no backplanes or docks) with fresh data cables.  Disconnect all other drives from power and data.  Try to get it to build parity, able to copy big chunks of data (50G+), and able to check parity successfully afterwards.  If and when you get that rock stable for several days to a week of heavy use, then start to make small changes (add a drive, use a backplane, etc.) ONE AT A TIME and running similar tests after each one.  Trying to diagnose a problem with so many variables is too difficult.

 

My experience is that it is far easier to diagnose a problem when moving from a known working configuration to a problematic one, then to try to move backwards from a problematic configuration to a working one.

 

Good luck!

I reverted to 4.3.3 and am able to transfer files fine so far. 

I reverted to 4.3.3 and am able to transfer files fine so far. 

 

Same here....so far. I'm transferring files over and running a parity check. 1.5 hours in, so far, so good.

I reverted to 4.3.3 and am able to transfer files fine so far. 

 

Same here....so far. I'm transferring files over and running a parity check. 1.5 hours in, so far, so good.

 

I've successfully transferred the same files multiple times now with 4.3.3 that result in issues with 4.4.2.  Next step is to verify the effect of the qd tool with 4.4.2, if any.  I'll do that tonight.

Unraid 4.3.3 has fixed my problems! Thanks for all the help  I copied over ~50GB of data and ran a parity check and everything is still online and working well! I've attached a screenshot.

 

Does this mean that I won't be able to upgrade?

 

Also, I still see ~140 errors on drive 4 (see screenshot). I will try running a SMART check tonight. This drive was a replacement for a DOA Samsung 1TB drive--I'm assuming it's refurbished. Are errors common on replacements for warrantied drives?

 

I reverted to 4.3.3 and am able to transfer files fine so far. 

 

Same here....so far. I'm transferring files over and running a parity check. 1.5 hours in, so far, so good.

 

I've successfully transferred the same files multiple times now with 4.3.3 that result in issues with 4.4.2.  Next step is to verify the effect of the qd tool with 4.4.2, if any.  I'll do that tonight.

 

What board/chipset are you on? The qd tool didn't work for me. I'm on an Asus P5N-E with the Nvidia nForce 650i chipset. Glad you got yours working as well!