I installed a Broadcom SAS 9300-8I 8-Port 12Gb/S SATA+SAS Pci-Express 3.0 Low Profile Host Bus controller and moved all of the 16tb drives and a couple of cache drives onto the controller. After booting up I now have major problems, the array just shows starting and will not start. I have a complete flash drive backup and backups of my docker containers as well as a screen shot of what the arrary looked like before the problem started.

 

Any help would be much appreciated

Thanks

ultimate-diagnostics-20240219-1041.zip

Multiple errors with this disk:

 

Device Model:     ST16000NM000J-2TW103
Serial Number:    ZR60AVYP

 

Check/replace cables and post new diags after array start.

I just installed the new card with new cables! I can only install the old cables.

You can also just check the connection or swap cables with another disk.

Ok, I moved and used the old cables. I can't change the cables from the broadcom card because they only have sata connectors on the ends of the cables the other end connects to the card with a plug that has 4 cables attached. I put the old cables in and attached them to the mother board but the parity 1 and disk 3 are still showing as disabled.

Here is the new diags.

Thanks

ultimate-diagnostics-20240219-1127.zip

I found a reddit post that shows how to enable the drives:

level 1

Medical_Shame4079

· 2 yr. ago

Stop the array, unassign the disk, start the array in maintenance mode, stop it again, reassign the drive to the same slot. The idea is to start the array temporarily with the drive “missing” so it changes from “disabled” to “emulated” status, then to stop it and “replace” the drive to get it back to “active” status.

 

Now I am trying to shutdown the array to try and fix disk 2 but the array won't stop, disk 10 keeps showing busy and the real time logs show it trying over and over again to un-mount the drive. It has been trying for over 1/2 hour so far.

I managed to get all the drives working again, now running a parity check, but lost some data, not sure how much but at least it's all working again.

Thanks

13 hours ago, vmax5000 said:

Here is the new diags.

No disk errors so far, you need to check filesystem for disk3, run it without -n.

Yes, I did that but had to use -L to fix it. It's all working now

Thanks

23 hours ago, vmax5000 said:

found a reddit post

Probably better to consult the current documentation. There is a link at top and bottom of the forum, and the "manual" link at bottom right of your Unraid webUI.

 

https://docs.unraid.net/unraid-os/manual/storage-management/#rebuilding-a-drive-onto-itself

 

Also, we recommend repairing filesystem before rebuilding, especially when rebuilding onto the same disk.

Thanks I never noticed the link at the bottom right!

7 hours ago, vmax5000 said:

had to use -L to fix it. It's all working now

Do you have a lost+found share now?

Actually I do, as well as some other very strange things. It looks like it's doing a parity check but 1 parity drive is not doing any reads or writes and I have 2 other drives not  doing any r/w. I also have 1 drive that says it's mounted in one place but unmounted in another place. I really don't know what to do at this point. I checked yesterday and all of the data seemed to be there, but today I'm missing lots of data!

I was going to shut down the parity check tomorrow and try and figure out what to do about the 3 drives. 

Here is a dump that I just took, if you can give me any advice that might help me keep my sanity I would really appreciate it!

Thanks again

ultimate-diagnostics-20240220-1842.zip

Well, just looking at the drives now, I can't find the lost & found but it was there at one point, and the non rw drives show invalid path when I try to look at them. Not sure what happened, everything seemed ok yesterday!

Your screenshot has clipped Array Operation section off the bottom so can't see what it thinks it's doing now.

 

It wasn't doing a parity check. It was rebuilding parity and disk3. But connection problems on multiple disks may have aborted that by now. Disk2 disconnected and became an Unassigned Device. It filled your syslog so can't tell what's happening now, but lack of activity also would indicate abort.

 

All array data disks mounted including emulated disk3, and with plenty of data. Not much data on either pool but maybe that is expected.

 

Nothing in lost+found

 

You will have to shutdown, fix whatever is causing these problems, reboot, and post new diagnostics.

 

How are these powered? Any splitters? Too many on one PSU cable?

Ok, I just tried to cancel the parity check but it doesn't seem to be doing anything. Here is the part you couldn't see in the clip.

I just put in a Broadcom SAS 9300-8I 8-Port 12Gb/S SATA+SAS Pci-Express 3.0 Low Profile Host Bus controller and moved all of the 16tb drives and a couple of cache drives onto the controller. I was having lots of disk errors before putting in the broadcom, I think I changed the sata cables on all the drives prior to putting in the broadcom card at least 10 times and the errors continued, after I put in the new card all be it a day ago, I stopped getting drive errors. Any suggestions as to what I should do to try and fix the current problem?

Thanks

I did upgrade the power supply a while ago to an 850 watt, I don't think I have too many drives on on psu cable, I have 3 psu cables off of the power supply with about 4 empty connectors on one.

 

You will have to reboot to clear logs or there won't be any way to see what is happening.

Ok, I have someone on the system right now when they get off I will reboot the system. It's getting late here now so I don't know if I will be able to get much done tonight, but I will get on it tomorrow morning. I will see if I can get the drives working tonight. I just looked on my plex app and I have lost almost my entire library, I do have the actual dvd's but it will take me months to restore everything.

I really appreciate your help, hopefully there might be something of value in the new dump after the reboot that will maybe help with some recovery?!? Anyway, I will post the new dump as soon as I get a reboot so that you can have a look at it before I do anything with it tomorrow.

 

Well, it has gone from bad to worse, I rebooted the system the first time and the array had not started so I started the array and got flooded with read errors on drives 2 and 3, then I powered down and pulled drive 2 and looked at it from another computer and it seemed to be fine so I put it back in and booted again, this time it came up with parity 2 disabled and the array was not started, so I set parity 2 to unassigned and started the array in maint mode, then I stopped the array and selected the same drive that was in that slot and it showed the drive was now enabled but I could not start the array because I am now getting an invalid configuration. So I powered down the system again removed the broadcom card and attached the drives back on the motherboard, but again after boot I still get an invalid configuration and I can't start the array. I don't know what happened or why I was getting flooded with read errors on 2 drives, but now the system is completely unusable. Is there any hope of getting this working again? Would it be better to wipe everything and start over from scratch. The only thing that I really need to keep is the Plex data that has all my users etc and I have all of that data on one of the cache drives, so if I have to start over I should be able to install plex and still have all of the data right? Can all of this trouble be because of the broadcom card?

Let me know how bad this is...

Thanks

ultimate-diagnostics-20240220-2029.zip

Probably your data is OK, at least on the disks that haven't been giving problems.

 

New Config may be the way out of this, but only if you can get all the drives connected reliably.

 

I will check back tomorrow, maybe someone else will have some ideas.

Thanks, I do have a complete Flash drive backup I made a few days ago when everything was working properly.

 

 

Just a quick update, I ran a short self test on the drives that were giving me read errors and the all passed, I was going to run the long test but it would take about 30 hours. I have also been running spinrite on parity drive 1 and it is almost complete and found no errors, so I don't know why I was getting the read errors. The sata cables off of the broadcom card are brand new and had no issues in the beginning. I can connect the drives back to the motherboard but I was getting lots of random errors. Should I use the flash backup on the flash drive? I have never done that before, is it just a matter of extracting it to the flash drive? I plan on running disk 2 on spinrite level 2 to check for errors but it will take 21 hours to complete. If you aren't familiar with spinrite, check it out at grc.com if you want, it's an incredible piece of software.

I can see that the data is still on the drives, so if I can get the array running again with no drive issues how can I recover the data?

Thanks

Post new diagnostics

Ok, I have 2 drives out of the system right now, running the disk scan on them, so this diag dump will be missing those disks.

 

ultimate-diagnostics-20240221-1824.zip