All Data Drives Missing: Please Help

[Vr2Io]

7 minutes ago, adminmat said:

Maybe the Easystore enclosure board has an effect?

Pls note WD / Easystore enclosure often can't direct access the disk by system again if disk not format/partitioning under it.

 

7 minutes ago, adminmat said:

Should i assign i to the array?

Pls don't.

 

Sorry, I leave now, may follow later.

Edited February 8, 2020 by Benson

[adminmat]

12 minutes ago, Benson said:

Pls note WD / Easystore enclosure often can't direct read the disk by system again if disk not format/partitioning under it.

 

Pls don't.

 

Sorry, I leave now, may follow later.

ok, I ordered a SATA to USB powered adapter. Will arrive to me on Sunday. 

 

Thanks and have a nice weekend 

[Vr2Io]

47 minutes ago, adminmat said:

ok, I ordered a SATA to USB powered adapter. Will arrive to me on Sunday. 

 

Thanks and have a nice weekend 

If those adapter also pass SMART data, it also got same problem.

 

I have a new idea, if setting some wrong controller parameters in disk setting, then it may cause couldn't read the SMART. If you kwon how to do or test then may try, or I will test this when I come home. But I don't know this could help or not.

Edited February 8, 2020 by Benson

[JorgeB]

2 hours ago, adminmat said:

can't believe 3 nearly new drives would all fail at the same time.

Likely damage from a power spike or similar, I can't see how it can be anything else than bad disks since they have the same issues with multiple controllers/cable/PSUs.

[adminmat]

6 hours ago, johnnie.black said:

Likely damage from a power spike or similar, I can't see how it can be anything else than bad disks since they have the same issues with multiple controllers/cable/PSUs.

I'm not sure about the power spike. The enclosures are powered by a relatively new, quality PSU that's plugged into a relatively new UPS / surge protector. Along with other devices powered by same UPS (server, rasbPi, cameras) and no other devices are damage but the 3 disks. The UPS logs events and shows no issues. 

 

I can't find instances written in other forums where a new PSU fries a computer in its own.

 

At this point I can't just replace the drives without knowing what caused this. HBA? PSU? Enclosures / Backplanes? Cables? Software? UPS?

 

If I'm going to buy $500 in new drives I'll need to replace all of this so it doesn't happen again. So we're talking about a $1000+ hit here. Not including the hours to archive and entire Blu-ray collection all over again. 

Edited February 8, 2020 by adminmat

[adminmat]

6 hours ago, Benson said:

If those adapter also pass SMART data, it also got same problem.

 

I have a new idea, if setting some wrong controller parameters in disk setting, then it may cause couldn't read the SMART. If you kwon how to do or test then may try, or I will test this when I come home. But I don't know this could help or not.

I don't know how to test this but I'd like to try. 

[Vr2Io]

15 hours ago, adminmat said:

I don't know how to test this but I'd like to try. 

The idea was set in "disk setting" -> "controller type" to some wrong setting, i.e. "HP cciss". I try this and reboot also can identify all disks ( no matter LSI or onboard), but no SMART data show anymore.

 

Pls test it in new Unraid and then check the log does no more SMART error, if positive, you may bootup orginal USB and set it and check again.

 

On other hand, does you feel disk was spinup ??

 

Below are ther error which show in your log

Quote

Feb  7 14:58:45 Tower kernel: ata6.00: failed to read native max address (err_mask=0x1)
Feb  7 14:58:45 Tower kernel: ata6.00: HPA support seems broken, skipping HPA handling
Feb  7 14:58:46 Tower kernel: ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Feb  7 14:58:46 Tower kernel: ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Feb  7 14:58:46 Tower kernel: ata5.00: READ LOG DMA EXT failed, trying PIO
Feb  7 14:58:46 Tower kernel: ata6.00: READ LOG DMA EXT failed, trying PIO
Feb  7 14:58:46 Tower kernel: ata5.00: failed to get NCQ Send/Recv Log Emask 0x1
Feb  7 14:58:46 Tower kernel: ata6.00: failed to get NCQ Send/Recv Log Emask 0x1
Feb  7 14:58:46 Tower kernel: ata5.00: failed to get NCQ Non-Data Log Emask 0x1
Feb  7 14:58:46 Tower kernel: ata5.00: ATA-9: WDC WD80EMAZ-00WJTA0, 7HJVD56F, 83.H0A83, max UDMA/133
Feb  7 14:58:46 Tower kernel: ata5.00: 15628053168 sectors, multi 0: LBA48 NCQ (depth 32), AA
Feb  7 14:58:46 Tower kernel: ata6.00: failed to get NCQ Non-Data Log Emask 0x1
Feb  7 14:58:46 Tower kernel: ata6.00: ATA-9: WDC WD80EMAZ-00WJTA0, 7HJVDXYF, 83.H0A83, max UDMA/133
Feb  7 14:58:46 Tower kernel: ata6.00: 15628053168 sectors, multi 0: LBA48 NCQ (depth 32), AA
Feb  7 14:58:46 Tower kernel: ata5.00: failed to get NCQ Send/Recv Log Emask 0x1
Feb  7 14:58:46 Tower kernel: ata6.00: failed to get NCQ Send/Recv Log Emask 0x1
Feb  7 14:58:46 Tower kernel: ata5.00: failed to get NCQ Non-Data Log Emask 0x1
Feb  7 14:58:46 Tower kernel: ata5.00: configured for UDMA/133
Feb  7 14:58:46 Tower kernel: ata6.00: failed to get NCQ Non-Data Log Emask 0x1

 

 

Edited February 9, 2020 by Benson

[adminmat]

On 2/8/2020 at 11:25 AM, Benson said:

The idea was set in "disk setting" -> "controller type" to some wrong setting, i.e. "HP cciss". I try this and reboot also can identify all disks ( no matter LSI or onboard), but no SMART data show anymore.

 

Pls test it in new Unraid and then check the log does no more SMART error, if positive, you may bootup orginal USB and set it and check again.

 

On other hand, does you feel disk was spinup ??

 

Below are ther error which show in your log

 

A few more things I've tried:

 

- I've tried changing the controller type as you recommended. It was already set to "Automatic." I changed it to a few different things from the list with reboots each time, then back to "Automatic" and a reboot and still could not run a SMART Self test. 

 

- I tried connecting the drive with a powered USB to SATA adapter. This one to be exact. It would not show the drive at all in unRAID. It did show a spare SSD that I connected. Maybe this adapter doesn't work on 8TB drives? I did tape the 3.3V pin again.

 

- I removed the PCB that's attached to the HDD to inspect for damage. Did not see any burning or arcing marks. 

 

- I disassembled the backplane of the iStar USA hot swap enclosure to check for shorts or damage. I used my multi-meter to check for continuity shorts between the different voltages and grounds. Did not find any issues. 

 

- I checked all voltages while the backplane was connected to the power supply. Everything seemed normal except SATA power pin 11 had 1.6 volts. I'm not sure if this is normal or not. I read it has something to do with "Staggered Spin-up" although i'm not sure if that is just normal.

 

Someone on Reddit recommended I use a program on windows called HDDSentinal. It's about $30 which I don't mind. I would really like to understand why this happened. So I can prevent it from happening again.

What do you recommend for the next step? I'd like to see if I can save the data that's on Drive 1. Should I try the program Recuva? I've used this before on an SSD with success. (this drive is XFS Encrypted) Then maybe I'll send a disk to a hard drive tech for some forensic work. Spending a few hundred to find out why it happened is less than buying another $500 in drives and it happens all over again. 

 

Pretty bummed. This is my first server and first NAS. What a pain this is. 

 

 

[Vr2Io]

30 minutes ago, adminmat said:

still could not run a SMART Self test.

Target not on SMART self test, just hope unraid could identify those not a bad disk. That's why I need you check the log and see any good sign.

 

30 minutes ago, adminmat said:

Maybe this adapter doesn't work on 8TB drives?

I don't think so, those adapter should be up to date. ( Have you plug in a power adapter ? )

 

30 minutes ago, adminmat said:

Everything seemed normal except SATA power pin 11 had 1.6 volts.

Yes, those pin would act as Staggered Spin-up control during powerup and then provide the activity signal ( should conect to blackplate LED circuit )

 

30 minutes ago, adminmat said:

I would really like to understand why this happened. So I can prevent it from happening again.

I will vote it is enclosure problem because you found it power down. ( assume PSU still on ). Does the enclosure have memories power state for each bay or each bay have independent control on/off ?

 

I have 8+ those WD disks, never got problem, but really don't know why all your disks bad now.

 

30 minutes ago, adminmat said:

What do you recommend for the next step?

Pls check or provide the log after you change the "controller type", if no good sign then I haven't suggestion. I also want to know does you feel the disk was spinup ?

Edited February 11, 2020 by Benson

[adminmat]

3 minutes ago, Benson said:

Target not on SMART self test, just hope unraid could identify those not a bad disk. That's why I need you check the log and see any good sign.

So this means I just run a diagnostics report after changing the controller type and rebooting? 

3 minutes ago, Benson said:

 

I don't think so, those adapter should be up to date.

I tried all the disks and they would not show in unRAID at all. Only an SSD. I don't have another HDD to test 

3 minutes ago, Benson said:

Yes, those pin would act as Staggered Spin-up control during powerup and then provide the activity signal ( should conect to blackplate LED circuit )

 

Hard to confirm. But I will vote it is enclosure problem because you found it power down. ( assume PSU still on ). Does the enclosure have memories power state for each bay or each bay have independent control on/off ?

When I found it the PSU was off. It's a really simple enclosure. It's just on or off. There's not much to it. No individual disk control. Not even a power control for the entire enclosure. 

3 minutes ago, Benson said:

Pls check or provide the log after you change the "controller type", if no good sign then I haven't suggestion. I also want to know does you feel the disk was spinup ?

I'll check the log tomorrow. I definitely feel the disks spin up. They don't stop spinning until disconnected. 

[Vr2Io]

9 minutes ago, adminmat said:

So this means I just run a diagnostics report after changing the controller type and rebooting?

Yes, pls check the log and post it here.

 

9 minutes ago, adminmat said:

I tried all the disks and they would not show in unRAID at all. Only an SSD. I don't have another HDD to test 

For SSD or 2.5" HDD, there are no need to provide extra power, but if 3.5" HDD then you need provide extra 12v to the DC plug hole.

 

9 minutes ago, adminmat said:

When I found it the PSU was off.

Then it should be PSU problem, some fault should trigger the PSU protection and cause all disks failure.

 

9 minutes ago, adminmat said:

I'll check the log tomorrow. I definitely feel the disks spin up. They don't stop spinning until disconnected. 

Note. Fine.

Edited February 11, 2020 by Benson

[adminmat]

16 hours ago, Benson said:

Yes, pls check the log and post it here.

 

hp cciss - tower-diagnostics-20200211-1409.zip

 

 

AUTOMATIC -tower-diagnostics-20200211-1418.zip

 

16 hours ago, Benson said:

For SSD or 2.5" HDD, there are no need to provide extra power, but if 3.5" HDD then you need provide extra 12v to the DC plug hole.

Yes the one i have is powered with a 12V DC PSU. 

16 hours ago, Benson said:

 

Then it should be PSU problem, some fault should trigger the PSU protection and cause all disks failure.

 

Note. Fine.

 

[adminmat]

So what's the safest way at this point to rescue the data from Disk1?

 

1) in the new unraid install (6.8.2) try to add the disk as an unassigned drive and copy to a 2nd disk?

2) boot up an ubuntu VM on my Windows PC (ESXI Workstation), unlock the file system and copy to a 2nd disk?

3) use a recovery program? (recommendations?)

4) just take it to a data recovery guy in my area and get his initial evaluation? (~$200) i'd also like to see if he can sniff out the cause of the problem before I change all my other hardware out. 

 

Note that i have my Cache pool drives intact. I spent a significant amount of time organizing my plex metadata and adding images/posters etc. I would like to keep this.

 

 

 

Edited February 11, 2020 by adminmat

[JorgeB]

4 minutes ago, adminmat said:

1) in the new unraid install (6.8.2) try to add the disk as an unassigned drive and copy to a 2nd disk?

2) boot up an ubuntu VM on my Windows PC (ESXI Workstation), unlock the file system and copy to a 2nd disk?

3) use a recovery program? (recommendations?)

I don't see how any of these will work because the disk isn't initializing correctly, very unlikely you can mount it with any OS, yes at some point it appeared under UD, but if you check no partition was detected, so not possible to mount.

[adminmat]

24 minutes ago, johnnie.black said:

I don't see how any of these will work because the disk isn't initializing correctly, very unlikely you can mount it with any OS, yes at some point it appeared under UD, but if you check no partition was detected, so not possible to mount.

So your saying my only option is a data recovery service?

[JorgeB]

I believe so, but it won't hurt to try other things before, just to make sure. 

[adminmat]

31 minutes ago, johnnie.black said:

I believe so, but it won't hurt to try other things before, just to make sure. 

I put Drive 2 back in it's Easystore enclosure, connected it to my windows PC, ran the WD App and even there the "SMART Status failed"

 

Would I be correct to assume that SMART info has noting to do with the type of file system? 

And is there a chance that if the SMART isn't working for the disk that you can still read from the disk platter somehow? 

Edited February 11, 2020 by adminmat

[JorgeB]

19 minutes ago, adminmat said:

Would I be correct to assume that SMART info has noting to do with the type of file system?

Correct

 

20 minutes ago, adminmat said:

And is there a chance that if the SMART isn't working for the disk that you can still read from the disk platter somehow?

It's not just SMART, disk isn't being detected correctly, so that's why I believe not many options besides using data recovery services.

[Vr2Io]

Regret you can't solve the problem.

[adminmat]

I'm just returning to this issue now. Yesterday I took the drives into a data recovery shop. They told me that since they are WD Helium drives they can not be worked on and their recovery programs PC-3000 (they mentioned some other programs) does not have support for these drives so no data can be recovered.

 

I spoke to another data recovery shop who basically said the same thing. I'd like to get my server back up and running but this time I'm cutting out all the drive enclosures, HBA card.. and just running the drives off the SATA motherboard ports. I wont be messing around again with a DIY external enclosure solution 🙄

 

I already have two new WD 8TB Elements drives and one WD 10 TB Elements drive. I'll need to start archiving my Blu-Rays again. I'm hoping that before I start this can someone recommend a simple, onsite backup? Maybe to a large external USB drive etc?

 

Also, If anyone has any ideas on how I may "un-brick" these three 8 TB drives for future use I'd love to know. It's been 9 months so maybe some new info is floating around.

 

 

[ATEglauer]

I have no idea, really, but when I tried to change server hardware on my system, the new system didn't see any of my array drives - they were 3TB and mostly 4TB drives.  I switched back to my original server hardware and still, no drives were seen by unraid.  They did exist, but couldn't be read.  (I later discovered the backplane in my new hardware couldn't read drives over 2TB.)  It seemed the new hardware changed the partition table for my array drives.  I used GDISK to recreate the GPT (partition table) and then all drives were back to normal.  A big relief for me at the time! 

 

Perhaps try that and see if the partition tables on the drives can be restored.  I am a beginner at this stuff and was pretty nervous with messing with partitions on the drives, but in the end I didn't loose any data - only some sleep until I used GDISK!

 

Armand