They couldn't both be emulated with single parity. I think part of the diagnostics were collected while disk3 was still working and when it got to the smart reports it had disconnected.

 

Is disk5 currently plugged in?

Ok moved the 12 TB to a new cable on the unused second port of my HBA card and its it doesnt appear to be mounting correctly.  This could be either the drive or or there is a chance the HBA port is having issues.  I have attached the latest diagnostics.   Will have to call it a night.  tower-diagnostics-20250406-1108.zip

15 minutes ago, Bartok27 said:

Will have to call it a night

Me too.

15 hours ago, Bartok27 said:

I have attached the latest diagnostics.

These appear to be the same diagnostics that had me confused yesterday.

Just getting back to troubleshooting this today .  So after a few restarts and swapping two cable.  Where it keep giving me read errors and saying the drive was unmountable or had no file system.  The array is now up and drive 3 mounted.  

 

At this point it seems like there are bad sectors on drive 3 and or connection issues.   I could try running another extended smart scan but assume that is going to crash it again.

 

 I have a new 14 TB drive and another on order as i thought the 12 was completely dead and was ready to start again.

 

The one 14 tb is connected as an unassigned device and the old 8 TB drive 5 is also attached.   I think the 8 is ok but cant access it as its unmounted at the moment and currently  being emulated.

 

I'm wondering if the best route is to replace/copy the data from the 12 to 14 and then add the 8 tb and the 14tb back to the array.  If thats possible? Not sure how best to do that.

I currently only have the one extra 14TB drive (and a second 14 on order).  I also have about 10TB of free space on an external if i need a temp location. 

 

I feel like the ideal step would be to rebuild the disk5 thats being emulated but I'm worried if i add the 8 back and try to rebuild it on itself that either the parity is wrong or the disc 3 will crash during the rebuild and then i loose all the data on drive 8.   I'm wondering if its safer to just add it back as is and only loosing any changes that were possibly made while it was being emulated.  I had turned off all my dockers so nothing should be updating on it.

 

At this point i just want the easiest way to get it back up with the least data loss.

 

Let me know what you would do.   Thanks again

 

 

 

 

Note the system date is currently set to April 1 so the log reflects that.  Not sure why it changed but didn't want to mess with it at this  point.

 

tower-diagnostics-20250408-1748.zip

All array data disks mounted with plenty of contents, including emulated/missing disk5, but disk5 is read-only. Maybe it isn't being accurately emulated due to disk3 problems.

 

SMART for disk3 looks OK but haven't been able to complete an extended self-test. Logs show some problems trying to communicate with it.

22 minutes ago, Bartok27 said:

old 8 TB drive 5 is also attached.   I think the 8 is ok but cant access it as its unmounted at the moment and currently  being emulated.

I assume it is the Unassigned Device sdj (7HJZB2TF).

 

Can you mount this disk with Unassigned Devices?

 

18 minutes ago, trurl said:

I assume it is the Unassigned Device sdj (7HJZB2TF).

Yes that is the old disk5 8TB.    I was going to mount it but didnt know if it would effect anything if i wanted to add it back to the array somehow.  But i guess at this point not much to lose.  I will try mounting it

 

Mounted no issues and last modify dates on all folders and the emulated drive are exactly the same.

In that case should i add it back to the array?  and if so what are the steps

 

Maybe original disk5 will allow us to emulate disk3.

 

I have to go eat now

13 minutes ago, Bartok27 said:

what are the steps

 

I will come back with instructions

Thanks.  I have to head out also.  But will check the instructions when i get back.

Do you have a disk to replace disk3? Must be same size or larger, but not larger than parity.

Yes i have the a 14 TB thats connected as unassigned dev2 currrently and my parity is 14 so it will work.  I haven't precleared it yet.   So i assume i should still do that first or do you think i can skip that and just do a rebuild?  Some people seem to be split on if its required

 

 

Edited April 9, 20251 yr by Bartok27

No need to preclear. Keep original (possibly failing) disk3 safe in case we need to try to recover files from it.

 

Go to Tools, then New Config - Retain All - Apply.

 

Assign original disk5 as disk5, assign new disk as disk3, leave all other assignments as they are.

 

Very important! Check parity valid checkbox so New Config won't rebuild parity, AND check maintenance mode checkbox so no disks are mounted.

 

Start the array. At this point, Unraid will accept all the new assignments without doing anything else.

 

Stop the array. Unassign disk3.

 

Start the array in normal (not maintenance) mode. At this point, Unraid will disable disk3 so it will be need to be rebuilt. But don't yet.

 

Then post new diagnostics so we can consider how to proceed.

Just want to update.  I wasn't sure about the preclear and after some reading i started one last night.  Looks like its still in the pre-read stage so i will cancel it if its not required.

 

Also i was incorrect about there not being changes to Drive5.  I was reading some more logs and the cache drive did get copied to the drive since it was emulated.  So there is definitely a difference between the emulated version and whats on actual drive.  So from my understanding adding it back to the array is out as the parity will be off.

So i'm thinking i can backup the files i need on the 8TB to my external drive just in case.
Then try to add the 8tb and see if it can rebuild it?
I'm not sure if that will work or crash with the issues on the Disc3.
If it does rebuild it then i could follow your steps to rebuild disc3 onto the new 14tb.

Do you think there is a chance it can rebuild the 8TB successfully?
Or would you suggest a different route?

 

Thanks again for all the help.  

Edited April 9, 20251 yr by Bartok27

Might as well let preclear continue until we decide what to do. 

 

There is a fundamental reason you are in a difficult situation, with single parity and 2 drives with problems. In order to rebuild any disk, all other disks must be working well. The parity calculation has to read all other disks to calculate the data to write to the rebuilding disk.

 

https://docs.unraid.net/unraid-os/overview/nas/#parity-protected-array

 

Sounds like disk5 is too far out-of-sync to expect it to provide the data for the parity calculation to rebuild disk3.

 

Disk5 is, in fact, the disk Unraid wants to rebuild. 

 

And it isn't really clear that there is anything actually wrong with disk3.

 

If we could get disk3 connections working well that would be ideal. 

Yes i understand the problem I have got myself into and that with 2 failed drives its technically not recoverable and there could or likely will be data loss.

 

It would also be great if i could test and confirm that disk3 was good but I don't think i have any way to confirm that 100%.  Maybe if i had a spare PC or extra hardware to test with but unfortunately i don't have a great way to test that.  I have replaced several cable throughout this troubleshooting and at this time i think Disc3 is as good as i can get it.  

 

Unless you have any suggestion on how i could test it further?  I could try the extended smart test again but dont know if its best to try that first or just attempt to rebuild disc5.

But I am definitely open to suggestions if you have any?

 

Also not to add more complication but i am getting this error in the log now which appears to be it flushing the cache to the emulated drive5.

 

Apr  9 17:00:01 Tower shfs: copy_file: /mnt/cache/Media/Media_Unraid/TV Shows/Land/Season 1/Land - S01E01 - Twist WEBDL-1080p.mkv /mnt/disk5/Media/Media_Unraid/TV      Shows/Land/Season 1/Land - S01E01 - Twist WEBDL-1080p.mkv.partial (117) Structure needs cleaning
Apr  9 17:00:01 Tower kernel: XFS (md5p1): Free Inode BTree record corruption in AG 0 detected at xfs_inobt_check_irec+0x8e/0xa7!
Apr  9 17:00:01 Tower kernel: XFS (md5p1): start inode 0x7af66400, count 0x40, free 0x18 freemask 0x60000000bf8090, holemask 0x0

 

Does this need to be fixed before i try to rebuild drive5 if thats the route to take?  If its only an issue with that folder i am fine just deleting the folder if that will fix it.

 

So i backed up all the media that i really want off of the 8TB so its good to attempt a rebuild if you think is the best plan at this time. 

If i am understanding correctly to rebuild the 8tb drive it would only need to read the first 8tb worth of data off disc3 so depending what is wrong with the drive maybe it lasts that long.   If it rebuilds and looks good  then i would remove the 12 hope it emulates it and repeat the process on the 14.

 

If it fails to rebuild the 8TB what options would i have:   

      Format and mount the 14tb  and copy as much as i can off the 12 onto it?

      Then would i be able to rebuild the array without erasing the drives?

 

I have attached the latest diagnostics.   Note the time reverted to April 1 at some point in the troubleshoot so that is actually yesterday and when i reset it to the 8th. 

 

Let me know what you suggest.

Thanks for sticking with this the help and suggestions are appreciated.   

 

tower-diagnostics-20250409-1752.zip

 

Edited April 9, 20251 yr by Bartok27

55 minutes ago, Bartok27 said:

I have replaced several cable throughout this troubleshooting and at this time i think Disc3 is as good as i can get it.  

Have you tried disk3 on another port?

I have tried several cables.   But since i hadn't done it very systematically i ran through a bunch of boot tests on different cables.   

 

If i connect it to one of the onboard sata ports.  I get a bunch of read errors and its listed as unmountable or no os.    I tried the same cables and ports with other drives and they boot with no issues.

 

If i connect it via the HBA card I get one I/O read error in one of 3 sectors depending on which cable i use and two device reset fails but it will mount the drive and is full acessable.  So i have now tested 3 sata cable, 3 onboard sata ports, 3 HBA cables (4 sata each).  So i am pretty confident the hardware seems to work.   For the I/O read error i'm not entirely sure if its hitting a physically bad sector or if it just cant find the data that should be there.

 

After doing some reading i also saw some people recommending putting a fan on you HBA card.  Currently i don't have one on the HBA card but its in a node 304 case like your backup server so its pretty open with several case fans.  But just to be sure i ordered an extra fan for it to try.

 

So at this point i figure i will add the fan just in case just as an extra percaution.

 

Then i'm open to suggestion on what you would try next.

 

The options i can think of are :   

-check parity to see if it valid before trying a rebuild

-run another extended smart test

-try to rebuild disk5 onto itself (8TB drive) and if successful rebuild disk3 onto the 14tb drive

-copy or clone the 12 to the 14, add the 14 and 8 as is to the array and rebuild parity  (I'm not sure how this is done with data on all the drive drives or if it can be)

 

Let me know if any of these options make sense or if you have a better suggestion on next steps. 

 

Thankstower-diagnostics-20250410-2318.zip

 

 

 

Do you have another 12TB disk you could use to try and clone disk3 with ddrescue? If that work, you could then use the clone to rebuild disk5

Unfortunately I only have two new 14TB available for use.  

Try cloning to a 14TB, but if the results are good, you will need to upgrade to 7.1 first to proceed with the rebuild:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-525075

 

6 hours ago, JorgeB said:

you will need to upgrade to 7.1

I will try cloning it just curious why i need to upgrade to 71. before the rebuild.   

1 hour ago, Bartok27 said:

why i need to upgrade to 71. before the rebuild

Because earlier versions won't play nice with the "special" conditions you have.

9 hours ago, Bartok27 said:

just curious why i need to upgrade to 71. before the rebuild. 

v7.0 and earlier won't accept the clone with a larger capacity, v7.1 will, it just looks at the largest partition, which will still be 12TB, but usable, and you can then rebuild to expand later.

I've been running ddrescue hopping it would speed up but so far it should be done in just under 11 years.    Really hoping this is just the bad sectors and it speeds up at some point.

 

root@Tower:~# ddrescue -f /dev/sdc /dev/sdb /boot/ddrescue.log
GNU ddrescue 1.29.1
Press Ctrl-C to interrupt
     ipos:   40878 MB, non-trimmed:        0 B,   current rate:   32768 B/s
     opos:   40878 MB, non-scraped:        0 B,   average rate:    571 kB/s
non-tried:   11959 GB,  bad-sector:        0 B,     error rate:       0 B/s
  rescued:   40878 MB,   bad areas:          0,       run time: 19h 51m 18s
pct rescued:    0.34%, read errors:          0, remaining time:   4010d  8h
                               time since last successful read:          0s
Copying non-tried blocks... Pass 1 (forwards)