Bartok27

Have a disabled drive with read errors. Should i troubleshoot or just replace drive. I have been away and came back to find my unraid Disk 2 dissabled and being emulated. In the logs there are a bunch of "Power-on or device reset occurred" errors followed by some read errors. From what i read online could be a problem with the cables or drive controller or the drive itself. The drive appears to have been disabled an emulated since May 31. Last time i had a drive fail i made some mistakes troubleshooting it and ended up not being able to rebuild it. So was looking for some advise before i screw anyting up. Since its been emulated since May 31. I assume i need to rebuild the drive either on itself or replace it? I do have a spare 14tb drive i could replace it with. Would you run any tests on the existing drive or just replace it at this point? Are there any tests that can be run to confirm the drive emulation is good before rebuilding the new drive. Since the array is up and running with the emulated drive is there any other info to collect, tests i should run or troubleshooting i should do now before stopping the system to add the new drive. Any help is greatly appreicated, Thanks tower-diagnostics-20250623-0736.zip

I was trying to do some cleanup in my media folder and right clicked on the disc1 beside one of the files thinking it would remove that file from disc1. It deleted the entire disc1 from the media folder. I stopped the array as i didnt want anything else written to the disk. Is there a way to undelete it or recover it. At this point if i pulled the drive as the little bit i could find talks about using recover software on another pc. If i can recover the folder i can then rebuild the parity as it will obviously mess up the parity if i can recover the drive. Any suggestions would be appreciate. I do not have a backup as its all just media i could download again but its 9 TB of media so if I can recover it instead that would be preffered.

Just wanted to close this off. Thanks the parity has been rebuilt and everything is working. Seem the one drive was bad and the rest is ok. I can still mount the drive on my laptop via a usb dock so will do some more research on if i can recovery any of the data. But my unraid server is back up and working thanks everyone.

I really dont want to make anything worse at this point. can i safely shutdown and restart the server to remove the old 12TB. Or should i do the new config and parity rebuild first. Dont know if its better to pull it first or rebuild then shutdown and pull it

Do i need to clear or format the new 14 TB before assigning it to the disk 3 spot. Not sure what state it is in after a partial ddrescue restore.

The remaining time is still bounces between 3000-4200 days remaining. Still not saying any read errors in ddrescue? However the smart info now states "Failing now" in the raw read error rate row. ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate PO-R-- 012 012 016 NOW 4294942212 and ddrescue is showing GNU ddrescue 1.29.1 Press Ctrl-C to interrupt ipos: 174516 MB, non-trimmed: 0 B, current rate: 43690 B/s opos: 174516 MB, non-scraped: 0 B, average rate: 582 kB/s non-tried: 11825 GB, bad-sector: 0 B, error rate: 0 B/s rescued: 174516 MB, bad areas: 0, run time: 3d 11h 12m pct rescued: 1.45%, read errors: 0, remaining time: 3690d 10h time since last successful read: 0s Copying non-tried blocks... Pass 1 (forwards) So i assume the drive is done and probably time to give up on ddrescue? At this point i'm assume the drive is lost but thinking i will pull it and drop it in an external dock and see if i can pull anything off just to see what happens. If we assume the drive is dead and now there is no way to restore the emulated drive 5. Can we assume drive 5 (8TB) is good. It only contains media and some backups so i think i can just loose any changes that were made to the emulation. So can i add the remaining 4 drives to an array without loosing the data and the new 14TB drive to the array and rebuild the parity. If so could you point me to the correct steps to follow so i dont screw up anything else. tower-diagnostics-20250415-1313.zip Also Just for my information if i have any future issue. I wanted to confirm i took the correct steps for running ddrescue. I took the array offline (did not restart the server), set disc3 to no device so that the 12 TB drive showed up under "Unassigned Disk Device" and ran the command below (attached the current stats of the running command) root@Tower:~# ddrescue -f /dev/sdc /dev/sdb /boot/ddrescue.log On thing i did notice that was a bit weird is the drive is label dev2 and the other two devices are labelled Dev 1 and Dev 2. I assume that is just naming and has not impact but looks weird.

I've been running ddrescue hopping it would speed up but so far it should be done in just under 11 years. Really hoping this is just the bad sectors and it speeds up at some point. root@Tower:~# ddrescue -f /dev/sdc /dev/sdb /boot/ddrescue.log GNU ddrescue 1.29.1 Press Ctrl-C to interrupt ipos: 40878 MB, non-trimmed: 0 B, current rate: 32768 B/s opos: 40878 MB, non-scraped: 0 B, average rate: 571 kB/s non-tried: 11959 GB, bad-sector: 0 B, error rate: 0 B/s rescued: 40878 MB, bad areas: 0, run time: 19h 51m 18s pct rescued: 0.34%, read errors: 0, remaining time: 4010d 8h time since last successful read: 0s Copying non-tried blocks... Pass 1 (forwards)

I will try cloning it just curious why i need to upgrade to 71. before the rebuild.

Unfortunately I only have two new 14TB available for use.

I have tried several cables. But since i hadn't done it very systematically i ran through a bunch of boot tests on different cables. If i connect it to one of the onboard sata ports. I get a bunch of read errors and its listed as unmountable or no os. I tried the same cables and ports with other drives and they boot with no issues. If i connect it via the HBA card I get one I/O read error in one of 3 sectors depending on which cable i use and two device reset fails but it will mount the drive and is full acessable. So i have now tested 3 sata cable, 3 onboard sata ports, 3 HBA cables (4 sata each). So i am pretty confident the hardware seems to work. For the I/O read error i'm not entirely sure if its hitting a physically bad sector or if it just cant find the data that should be there. After doing some reading i also saw some people recommending putting a fan on you HBA card. Currently i don't have one on the HBA card but its in a node 304 case like your backup server so its pretty open with several case fans. But just to be sure i ordered an extra fan for it to try. So at this point i figure i will add the fan just in case just as an extra percaution. Then i'm open to suggestion on what you would try next. The options i can think of are : -check parity to see if it valid before trying a rebuild -run another extended smart test -try to rebuild disk5 onto itself (8TB drive) and if successful rebuild disk3 onto the 14tb drive -copy or clone the 12 to the 14, add the 14 and 8 as is to the array and rebuild parity (I'm not sure how this is done with data on all the drive drives or if it can be) Let me know if any of these options make sense or if you have a better suggestion on next steps. Thankstower-diagnostics-20250410-2318.zip

Yes i understand the problem I have got myself into and that with 2 failed drives its technically not recoverable and there could or likely will be data loss. It would also be great if i could test and confirm that disk3 was good but I don't think i have any way to confirm that 100%. Maybe if i had a spare PC or extra hardware to test with but unfortunately i don't have a great way to test that. I have replaced several cable throughout this troubleshooting and at this time i think Disc3 is as good as i can get it. Unless you have any suggestion on how i could test it further? I could try the extended smart test again but dont know if its best to try that first or just attempt to rebuild disc5. But I am definitely open to suggestions if you have any? Also not to add more complication but i am getting this error in the log now which appears to be it flushing the cache to the emulated drive5. Apr 9 17:00:01 Tower shfs: copy_file: /mnt/cache/Media/Media_Unraid/TV Shows/Land/Season 1/Land - S01E01 - Twist WEBDL-1080p.mkv /mnt/disk5/Media/Media_Unraid/TV Shows/Land/Season 1/Land - S01E01 - Twist WEBDL-1080p.mkv.partial (117) Structure needs cleaning Apr 9 17:00:01 Tower kernel: XFS (md5p1): Free Inode BTree record corruption in AG 0 detected at xfs_inobt_check_irec+0x8e/0xa7! Apr 9 17:00:01 Tower kernel: XFS (md5p1): start inode 0x7af66400, count 0x40, free 0x18 freemask 0x60000000bf8090, holemask 0x0 Does this need to be fixed before i try to rebuild drive5 if thats the route to take? If its only an issue with that folder i am fine just deleting the folder if that will fix it. So i backed up all the media that i really want off of the 8TB so its good to attempt a rebuild if you think is the best plan at this time. If i am understanding correctly to rebuild the 8tb drive it would only need to read the first 8tb worth of data off disc3 so depending what is wrong with the drive maybe it lasts that long. If it rebuilds and looks good then i would remove the 12 hope it emulates it and repeat the process on the 14. If it fails to rebuild the 8TB what options would i have: Format and mount the 14tb and copy as much as i can off the 12 onto it? Then would i be able to rebuild the array without erasing the drives? I have attached the latest diagnostics. Note the time reverted to April 1 at some point in the troubleshoot so that is actually yesterday and when i reset it to the 8th. Let me know what you suggest. Thanks for sticking with this the help and suggestions are appreciated. tower-diagnostics-20250409-1752.zip

Just want to update. I wasn't sure about the preclear and after some reading i started one last night. Looks like its still in the pre-read stage so i will cancel it if its not required. Also i was incorrect about there not being changes to Drive5. I was reading some more logs and the cache drive did get copied to the drive since it was emulated. So there is definitely a difference between the emulated version and whats on actual drive. So from my understanding adding it back to the array is out as the parity will be off. So i'm thinking i can backup the files i need on the 8TB to my external drive just in case. Then try to add the 8tb and see if it can rebuild it? I'm not sure if that will work or crash with the issues on the Disc3. If it does rebuild it then i could follow your steps to rebuild disc3 onto the new 14tb. Do you think there is a chance it can rebuild the 8TB successfully? Or would you suggest a different route? Thanks again for all the help.

Yes i have the a 14 TB thats connected as unassigned dev2 currrently and my parity is 14 so it will work. I haven't precleared it yet. So i assume i should still do that first or do you think i can skip that and just do a rebuild? Some people seem to be split on if its required

Thanks. I have to head out also. But will check the instructions when i get back.

Mounted no issues and last modify dates on all folders and the emulated drive are exactly the same. In that case should i add it back to the array? and if so what are the steps

Yes that is the old disk5 8TB. I was going to mount it but didnt know if it would effect anything if i wanted to add it back to the array somehow. But i guess at this point not much to lose. I will try mounting it

Just getting back to troubleshooting this today . So after a few restarts and swapping two cable. Where it keep giving me read errors and saying the drive was unmountable or had no file system. The array is now up and drive 3 mounted. At this point it seems like there are bad sectors on drive 3 and or connection issues. I could try running another extended smart scan but assume that is going to crash it again. I have a new 14 TB drive and another on order as i thought the 12 was completely dead and was ready to start again. The one 14 tb is connected as an unassigned device and the old 8 TB drive 5 is also attached. I think the 8 is ok but cant access it as its unmounted at the moment and currently being emulated. I'm wondering if the best route is to replace/copy the data from the 12 to 14 and then add the 8 tb and the 14tb back to the array. If thats possible? Not sure how best to do that. I currently only have the one extra 14TB drive (and a second 14 on order). I also have about 10TB of free space on an external if i need a temp location. I feel like the ideal step would be to rebuild the disk5 thats being emulated but I'm worried if i add the 8 back and try to rebuild it on itself that either the parity is wrong or the disc 3 will crash during the rebuild and then i loose all the data on drive 8. I'm wondering if its safer to just add it back as is and only loosing any changes that were possibly made while it was being emulated. I had turned off all my dockers so nothing should be updating on it. At this point i just want the easiest way to get it back up with the least data loss. Let me know what you would do. Thanks again Note the system date is currently set to April 1 so the log reflects that. Not sure why it changed but didn't want to mess with it at this point. tower-diagnostics-20250408-1748.zip

Ok moved the 12 TB to a new cable on the unused second port of my HBA card and its it doesnt appear to be mounting correctly. This could be either the drive or or there is a chance the HBA port is having issues. I have attached the latest diagnostics. Will have to call it a night. tower-diagnostics-20250406-1108.zip

I will shutdown and change the cables and post new diagnostics if it comes up. Is there any way to add disk5 back to the array? I think it was kicked out because it had a bad cable which was replaced previously Yes i dont understand them both being listed as emulated.

I stopped the array and Disk3 disappeared and is listed as missing. With the only options defaulting to no device and no unassign option. If i leave it as no device and try to start the array i get the error below in the logs. I get Apr 6 22:21:42 Tower emhttpd: unRAID array: ERROR:TOO_MANY_MISSING_DISKS and the array will not start I could shutdown and restart and see if it comes back? Or i could shutdown change cables again and restart? Let me know what you think

Thank you for the suggestions. I have a few questions about the two suggestions. Basically either continue troubleshooting the connection or clone the drive So you are suggesting to try other cables in case its still a connection issue? If i go the DDrescue route do i have to clone it to a 12 TB drive. I do not have a spare 12 currently but i do have a 14tb. Also we havent look at the disk5 8TB drive at all. Can it be added back? If we were to add it back could i rebuild the 12 onto the new 14TB. Sorry if thats a dumb option at this point we are way past my knowledge so just trying to learn and find the best route forward. I just want to be sure of the options and route forward so i dont make things worse. Thank again taking the time to help out.

I ran a short self test first just to see if it would run. Then i ran the extended test. The extended appears to not have finished. Last i check it was still at 10% after 28 hours. I left t running and today it shows the drive has spun down and 117,700 read errors on the main screen in unraid. I tried spinning it up but it doesnt work. I have attached the latest diagnostics.tower-diagnostics-20250406-1108.zip Is there another way to force the drive to spin up to get the latest smart info? Let me know what you would suggest to try next. Thanks!

here is the current diagnosticstower-diagnostics-20250405-1215.zip

It been about 21 hours since i started the extended smart check and it still says 10% complete. From a bit of reading its unclear if this may just be a display thing and its actually running in the background. The spin down delay on my drives are set to default. Which i from what i can tell is never spin down so dont think that is the issue. Im leaning toward just waiting another day and see what happens but just checking in to see if there is anyway to tell if its actually still running. Thanks

Thanks for the suggestion, i will have a look at that also.