20 minutes ago, Marino said:

But I can not see a difference between a damaged drive or a bad cable, right?

 

It's the old saying. Lack of proof isn't proof of nonexistence of proof. Without SMART errors, it's impossible to know if there was anything wrong. But seeing no SMART errors while having 3 drives with problems at the same time is a very good indication that it was a connection issue.

 

3 minutes ago, Marino said:

The data on the drive should be okay even when they dont match the parity which is why it is disabled.

 

All the "write error" lines in my previous post indicates 4 kB of data writes that never reached the drive. So somewhere on the drive, you have one or more missing files or files that are too short or have invalid content. If this had been an incident involving just one or two drives, then the result of the system would have still been able to track these writes and allowed unRAID to emulate disk 7 with perfect content just as if there hadn't been any errors. But since this was a three-disk incident, updates of parity also stopped. So right now, disk 7 has some invalid content. And emulation of disk 7 also have some invalid content.

 

In the end, there isn't any safe way to say what is more correct - the content of disk 7 or the emulation of disk 7 after disk 6 has been reconnected.

1 hour ago, pwm said:

But since this was a three-disk incident, updates of parity also stopped. So right now, disk 7 has some invalid content. And emulation of disk 7 also have some invalid content.

 

This is a good point, and makes a good case for rebuilding to a different disk. That way you get 2 tries at recovering the data, from the rebuild, and from the original disk.

 

After the reboot if the emulated disk is mountable that is a good sign. Have you tried to look at any of the emulated files? You just read from the disk as if it is really there, which it is in this case but unRAID will not use it since it needs to be rebuilt. Instead it will use all the other disks and the parity calculation to provide the data. And, whatever is in the emulated data is exactly what would be rebuilt.

@ pwm

I transfered a large file (>50gb). Chances are high that this is the bad file and the other ones are okay. But no one knows it this is the only one.

 

@ trurl

Thank you for the link (swapping parity).

Yesterday I was able to see the files, but I haven't tryed to download or open it because of using the complete array for this.

 

So I think the best way is to cut the ties and change at least this two SATA-Cables. For that I would buy some new ones. 

 

Replace the parity drive with the 4tb one would be an option. But for this I'd buy a SATA-Card first, because all of the 8 ports are used. And bulding new parity having the "old" parity drive safe is not an option, because disk7 is missing. So I only could swap it and then use the "old parity drive" to recover disk7. At the end I've 9 ports an should think about dual parity.

 

Buyers list:

- New SATA-Cables

- New bigger HDD's for the other Server, so I get 1 to 2 4TB drives

- SATA-Card for 1 or two disks to extend to dual parity. I only got 1 x16 and 3x x1 PCIe and the x16 is beeing used by an graphic card only for the system being able to boot. Any recommendations on that?

 

For now I rethink which option is best for Upgrading the other server and restore the data. Eventually with a new precleared drive or with the ones of the server being upgraded. In this time the server stays off.

 

Before I open a new thread for this, but when this is to complicated in a short answer, I could open a new thread. I'd like to buy HGST drives with 8 or 10TB. As far as I know, unraid is supporting 4Kn Drives. Are there advantages over 512e drives which are worth the extra charge (around +18%)? Should be a little faster and should be a little bit more rubust error correction (noticable or worth?). And is it right that every disk in the array has to use 4Kn? Are there Hardware limitations? Because of error in this unraid-server here, I replace almost every disk and beginning from scratch (Gigabyte Z87X-UD5H).

 

So, I have to tet to work. It is 8 in the morning here. 

 

 

I appreciate your help guys, you are awesome. 

As for now I have to consider the best hardware options and what to buy. I keep you up to date when doing something to this server, i promise.

3 hours ago, Marino said:

 

- SATA-Card for 1 or two disks to extend to dual parity. I only got 1 x16 and 3x x1 PCIe and the x16 is beeing used by an graphic card only for the system being able to boot. Any recommendations on that?

 

Get something like this. It will fit in a PCIe x1 slot and it will control two disks. It uses the ASMedia 1061 controller, which is stable and compatible. It uses the standard AHCI driver. It's plug and play with unRAID.

3 hours ago, Marino said:

And is it right that every disk in the array has to use 4Kn?

 

unRAID supports 4Kn disks and you can mix them freely with 512e disks. I have a WD Gold 4Kn parity disk alongside WD Red 512e data disks in a server and it works fine. I don't think there's any difference in performance due to the format. Any differences are due to the fact that it spins faster. I suppose that having 4 kiB physical sectors and 4 kiB logical sectors means that there's no translation being done but there's no difference in performance. I got my disk as a straight one for one swap with a friend who thought he was ordering a 6 TB WD Red. I wouldn't pay extra for one.

 

On the subject of HGST 10 TB disks, just make sure you don't buy any of the specialist Ha10 Archive disks. They are host-managed SMR disks and they won't work with unRAID.

56 minutes ago, John_M said:

I don't think there's any difference in performance due to the format.

 

unRAID does all disk accesses in 4kB blocks, so either eight 512-byte sectors or one 4096-byte sector. So it's just a difference in what address to use to seek n*4kB into a partition.

 

So the emulation of 512-byte sectors is just to make a modern disk compatible with software that assumes the blocks are addresses in 512-byte steps. But there isn't any performance difference - unless a disk with emulated 512-byte sectors is used with unaligned partitions. Unaligned partitions would be very, very, very bad.

@ John_M

Perfect. I'll order 2 of them in order to get a few ports extra.

 

When there are no benefits of 4Kn over 512e in safety or better failure management, then it's clearly not worth to pay more for them. For now I wanted to use HE10 Drives an no Archive. Then I can get up to 3 4TB WD Red out of the other Server and can do a Parity Swap and a rebuild on an other disk as the disk7 itself. After Rebuild, I can use dual parity, because of min. one more Drive and the ability to connect it to the SATA-Card (On Mainboard all 8 connectors are used). So for now, I upgrade the other Server and will use the left overs to restore the data. I just have to clarify which modell: HE10 8TB or 10TB and SE, ISE (with or without P3) or SED?!?

Edited August 21, 20187 yr by Marino

Just to clarify. 4Kn disks aren't necessarily more expensive than comparable 512e disks. They are more expensive because they tend to be enterprise disks. I don't know of any 4Kn consumer disks. So in my example above I was comparing a WD Gold 4Kn disk with a WD Red 512e disk when I said I wouldn't pay extra for one. If you compare a Gold 4Kn with a Gold 512e of the same capacity the difference in price should be minimal.

4 hours ago, John_M said:

If you compare a Gold 4Kn with a Gold 512e of the same capacity the difference in price should be minimal.

Exactly - it's basically a difference in firmware in the drive if it should emulate 512 byte sectors or if it should use the native addressing and only support n*4kB reads/writes.

 

But a 512e drive does have 4 kB sectors internally. And a write of a 512 byte virtual sector means the drive needs to locate and read the 4 kB sector. Patch in the 512 byte change and then on next rotation write back the modified 4kB sector data. Which is why it's panic slow to create unaligned partitions on a 512e drive.

 

The reason for moves to 4 kB sectors isn't to make the drives faster, but to fit more data on the track by having fewer sector gaps. And because 8 times as much error correction data covering 8 times as much data gives higher reliability. This is required since the storage capacity keeps increasing and the probability of undetected errors during a full disk readout started to be too high.

The larger disks have 4K already. So the advantage over 512byte is already "build in". Okay, for that 4k is better, but is there an advantage to use this sectors natively instead of emulated?

 

Emulation is normally not that good as native. But is there any reason to buy 4Kn when 512e also uses 4k Sectors? Or are there only compability issues to be aware of when using 4kn without any benefit for my installation? The extra charge when using HE12 12TB is about 10€. Better than 10TB (70€). 

 

EDIT:

Today I ordered new SATA-Cables and the recommended SATA-Controller for PCIe x1. Only a few disks are needed to have some leftovers (4TB-Disks) to use for rebuild and parity swap (3>4TB)

Edited August 22, 20187 yr by Marino

I think you're worrying too much. For unRAID is really doesn't matter as it makes no difference whatsoever. 512e is done for backwards compatibility but, as @pwm said, if you get the alignment wrong the performance drops massively. unRAID gets the alignment right so there's no practical difference. Of course, native is "better" in the sense of it being less complicated but the translation process is so quick there's no performance loss. If hard disks were just being invented they would all be 4Kn but old applications don't support them, hence the need for 512e. unRAID only added support for 4Kn as recently as version 6.4 (I remember because I made the initial feature request) so before that people needed to use 512e.

 

It would have to be a really bad OCD day to make me feel compelled to pay an additional €10. As for an extra €70 - forget it.

 

Incidentally, when comparing prices of hard disks I always find it useful to write down the cost per terabyte as well as the total cost.

46 minutes ago, Marino said:

Emulation is normally not that good as native.

 

Correct if you use an OS and/or file system that performs 512-byte writes.

But just about all file systems in existence have been using 4kB cluster size (or larger) for a large number of years.

So slowdown only happens if there is an unaligned paritition - and the partition editors (and unRAID) understands this.

 

So for modern OS, the emulation is just a question of what address the OS should send to access a specific block of data. No speed difference involved - just a question of compatibility.

 

48 minutes ago, Marino said:

But is there any reason to buy 4Kn when 512e also uses 4k Sectors?

 

It really doesn't matter which one you buy, as long as you have an OS that supports 4kB sector size. If using older OS that assumes 512-byte sectors, then you are forced to buy a 512e drive. So x years from now, most 512e drives will vanish from the market, because there isn't a need for them other than as spare parts for ancient machines.

 

It's confusing to users with a HDD that has one broken sector but reports 8 broken sectors in SMART. That's what happens with a 512e drive, since the broken 4kB native sector is presented as 8 virtual 512-byte sectors.

 

 

Of cause I am worrying to much but in this case its about the first really big disks for me  which should be last a long time in the unraid server. And in addition to that the drives cost me around 330-400€ each. And in long term I like to have a second parity drive which makes this not cheaper. So I just want to make sure that I buy the right ones.

 

I'll compare GB-prices of the HE10 8TB, HE10 10TB and HE12 12TB with SE, ISE etc. and will buy the "cheapest". Cheap is not really the right word for that

Short Update:

Cables, Controller and an additional Fan are already here. I just have to Order HDD's but didn't find the time for it, yet. But this has to be done. For now, the server is turned off.

 

UPDATE 2:

I just ordered 3 HE12 12TB HGST drives. After installing in the other Server, I'll transfer 2 or 3 4TB WD red to the unraid-server, which has obviously some problems and can perform a parity swap and will have dual parity.

 

EDIT: 

Worst than I thought. The speed is getting slower any minute. At the beginning "only" 35h, now after 4 hours are 64h left  Ahhhh...

Edited September 4, 20187 yr by Marino

So guys, I have an update, but also a question.

 

As mentioned, I am in the middle of upgrading my other unraid server to get 3 4TB-drives out of it. So I have to make a parity swap (3TB>4TB) to get an 4TB drive for reconstruct while not touching the missing 3TB one in this case. After reconstructing I would add a second parity (the third 4TB drive).

 

While I was following the wiki to make a parity swap, there is described (Wiki) in #13, that there should be a copy button. This button was not there and I've managed to make my parity invalid and don't know how this exactly happened. 

 

So in this case here, I have to make a parity swap for making it possible to use a 4TB drive to reconstruct data. But when this happens again, I am not able to reconstruct, because of invalid parity. Did misunderstood the wiki? Why wasn't there a copy-button an what can I do to definitely retain the parity data valid while swapping?

 

FIY: ATM my other server computes parity. This takes about 50 hours  After that I can add two more drives. This to will be precleard, assigned and 6TB of data has to be copied to the new ones. After that I could use the three disks in this server here.

 

PS: Ordering the 3 disks was involving 2 hours at the police station and a lot of trouble  because of a little mistake... Should I describe why this happened? 

26 minutes ago, Marino said:

PS: Ordering the 3 disks was involving 2 hours at the police station and a lot of trouble  because of a little mistake... Should I describe why this happened?

What happened  ? 😜

Edited September 4, 20187 yr by Benson

I had to make a new account on the internet shop. While typing I switched one character (. instead of _ as all of my other mail addresses). This was the first site ever where i could log in without confirming the mail address. So I ordered 3 12TB HGST to my address and payed with paypal. One day before delivery I've got a partial repayment. So I picked up the phone and called the customer service. Seems that one day before someone called them. The caller said he was me and had never ordered something. The shop checked the data, everything were normal. They don't noticed that the address was wrong and the mail addresses (ordering and paypal) differ in one character. So he decides to go to the police station. 5 service employees could not help me and the last one forbid me to reorder. I noticed the mistake and wrote an email to that address with . instead of _. 

So now I know, there is someone with exact the same name es me who is living closer than 10km from here. He thought some one stole his account data. So I had to go to the police for clarifying. They wrote a protocol and maybe I'll get a citation in a few weeks.

I ordered the disks somewhere else. And this is why it took so long to get them.

Understand, small mistake in mail address cause big trouble. 😭

Yesterday I finally installed the 4TB drive to swap.

 

I am willing to do a parity swap, but this will not work, because of some errors in log (attached).

 

The first Error is shortly after "Sep  5 17:53:46"

 

Here a few of them:

...Sep  5 23:09:30 unQuad kernel: ata6.00: error: { UNC }
...Sep  5 23:09:31 unQuad kernel: print_req_error: I/O error, dev sde, sector 4171739048
...Sep  5 23:09:33 unQuad kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
...Sep  5 23:09:33 unQuad kernel: print_req_error: I/O error, dev sde, sector 4171739048
...Sep  5 23:09:33 unQuad kernel: Buffer I/O error on dev sde1, logical block 4171738984, async page read
...Sep  5 23:09:36 unQuad kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
...Sep  5 23:09:38 unQuad kernel: Buffer I/O error on dev sde1, logical block 4171738986, async page read

 

I checked the box "yes I want to do this" and was copying. In the middle of the night it seems to be done and I checked the box again, but I can not say if there was standing copy or start, I think copy.

 

The 4TB drive is from the other server and disk7 (sde) is the old parity drive, which replaces disk7 which was there before and was offline. This disk is next to me on the table.

I changed the SATA-Cables from 5 of the 8 disks to new good ones and they are not tied together anymore. The other 3 were changed recently.

 

I am just not good in reading errors. I saw there are some, and they seems to be not good, but many of them are cryptical to me

 

UNC = Uncorrectable?

 

I don't know if I can use preclear for "checking" the drive? I am afraid of installing preclear, as I don't know on which patch it will be installed and the it could be possible to get the parity unvalid, when there is something written on a data disk.

 

I run a short SMART-Test on both the drives:

disk7 gives me "Errors occurred - Check SMART report" (attached)

parity (4TB) is without any error.

 

What is the log exactly telling? Uncorrectable error on the old parity drive, so that I have to put that disk away and deal with the lost of parity?

 

I think these are the interesting parts in the SMART-Log:

  40 -- 51 00 06 00 00 f8 a7 af aa e0 00  Error: UNC 6 sectors at LBA = 0xf8a7afaa = 4171739050
  40 -- 51 00 01 00 00 f8 a7 af a9 e0 00  Error: UNC 1 sectors at LBA = 0xf8a7afa9 = 4171739049
  40 -- 51 00 01 00 00 f8 a7 af a8 e0 00  Error: UNC 1 sectors at LBA = 0xf8a7afa8 = 4171739048
  40 -- 51 00 06 00 00 f8 a7 af aa e0 00  Error: UNC 6 sectors at LBA = 0xf8a7afaa = 4171739050
  40 -- 51 00 01 00 00 f8 a7 af a9 e0 00  Error: UNC 1 sectors at LBA = 0xf8a7afa9 = 4171739049
  40 -- 51 00 01 00 00 f8 a7 af a8 e0 00  Error: UNC 1 sectors at LBA = 0xf8a7afa8 = 4171739048
  40 -- 51 04 00 00 00 f8 a7 af a8 e0 00  Error: UNC 1024 sectors at LBA = 0xf8a7afa8 = 4171739048
  40 -- 51 00 06 00 00 f8 a7 af aa e0 00  Error: UNC 6 sectors at LBA = 0xf8a7afaa = 4171739050

Doesn't look good, but what is that exactly telling me?

Am I correct, that SMART is running on the disk itself and when I get SMART-Errors it could not be a bad connection?

 

 

unquad-diagnostics-20180906-0649.zip

WDC_WD30EZRX-00MMMB0_WD-WCAWZ2834262-20180906-0705.txt

Edited September 6, 20187 yr by Marino

That disk is failing and needs to be replaced

But on it is my parity which should be copied to the 4TB. So, there is no chance to recover the parity and the recovery of disk7 which was offline is only possible while computing new parity to the 4TB disk and then copy the data from the offline disk, where maybe some data is corrupt, because it went offline for some reason?

 

Wouldn't I replace that parity drive I would never noticed that my parity drive is damaged? How to prevent this? Doing long SMART tests every ? weeks or ?? months?

In my case the data is on the "old" disk7 but some data are not okay and I don't know exactly which. When the drive was failing to me and not only going offline, the data would be gone because of single parity right? Dual parity would have helped in this case?

 

EDIT:

In other words, failed disk while reconstruct. In this case the parity drive itself.

Edited September 6, 20187 yr by Marino

30 minutes ago, Marino said:

How to prevent this?

Enable system notifications, you'd get a warning about the pending sector.

 

You can still try the parity swap, not sure if it will ignore read errors or abort, would expect ignore, though even if it does there will be some corruption on the rebuilt disk.

I tryed the parity swap two times. I stops while copying the parity data to the 4TB drive. "Stopped. Upgrading disk/swapping parity". 

So even when the the copy would be complete and there are some error I don't know which data is involved. With the offline disk chances are high, that only one file is affected, because it went offline while I copied a 50GB file on it over smb.

 

My system notification is enabled with twice a day and detailed. I've got no information from this. 

If the parity copy aborts on error you can't do it with a failed disk, you could clone the parity disk to a new one (skipping the errors) and then do a new config and use the new one as parity and do the swap, there will still be some corruption, but if disk7 is more damaged it might be worth a try.

I wish I'd Checksums of the files. The most of them are over 20 gigs. Then I could have checked if they are ok.

I think, copy the data of the offline drive would be the better way to do, because I know which data was written while it was going offline. Cloning to another drive give me the defect sectors and there could be any data involved, which is on this disk.

 

Boah, this is why I wanted to add a second parity