June 11, 20242 yr Really going nuts! Have a 13 drive server with 2 parity drives. All was fine until 3 days ago. Had a toshiba 20TB in my disk 5 slot. Pulled up GUI and showed drive was red X'd and data emulated due to errors. This drive had been problematic since I bought it so I decided I'm done. Put in a good old reliable 20TB WD Red Pro. Same thing has happened! I removed the WD from disk 5, removed partition, cleared it, then reformatted. Got things back up, and it did it again, almost immediately! I swapped out new cables from the LSI 9300 to the backplane cage. Started over, same issue. I changed out the backplane cage from another box that I knew had a good one. Again, same thing. Finally, I bought another LSI 9300-8i. SAME THING.....AAARRGGHH Do y'all think that it's possible the MOBO PCI-e slot is bad causing just one disk to have errors? I can't think of anything in the MOBO bios that could cause it since this all just happened out of nowhere. I'm at wits end. If anyone has any thoughts, I'd appreciate it. The only silver lining it's only one drive I lost so rebuild is easy.....if I can ever get to that point. tower-diagnostics-20240611-1302.zip
June 11, 20242 yr Community Expert Disk dropped offline immediately, how is the backplane powered? Try swapping that disk to a different slot.
June 11, 20242 yr Author Backplane cage powered by 2 x 4-pin connectors. Using a 750W PSU. Power has never been an issue. I'll try swapping drive slots but last time I tried, it showed drive I moved around was missing and still had trouble with the original drive. I'll get back to you JorgeB
June 11, 20242 yr Author BTW Jorge, this all was happening a few weeks ago when you helped me. Same issue and we thought it was the toshiba drive. I swapped it in another slot and it worked,. Put it back in original, it worked until a couple days ago. That's when I started the above gyrations. It's the same exact spot and disk #. Changing cables, backplane cage, and HBA didn't help. I think this goes deeper but don't know what. In fact, when I get the error and the drive stops, I can't shutdown the array on the GUI either. I either must reboot or full shutdown
June 11, 20242 yr Author Nope, swapping to another slot didn't help...now I'm wondering if there is something with these newer 20TB WD and Toshiba drives that the LSI 9300 doesn't like. The other 20TB drives I have, that are my parity drives, are shucked drives and doesn't seem to be an issue...... I'm really baffled now tower-diagnostics-20240611-1356.zip
June 11, 20242 yr Community Expert 15 minutes ago, isrdude said: LSI 9300 doesn't like. It could be a firmware issue, you are on a very old one: mpt3sas_cm0: LSISAS3008: FWVersion(16.00.01.00) Try updating to latest:
June 12, 20242 yr Author Thought I'd try the original disk, the Toshiba, and see what would happen. It got a little farther down the road but same results. I'm starting to really wonder if this is a MOBO/CPU issue....I've done everything else but swap out the MOBO and CPU. tower-diagnostics-20240611-2149.zip
June 12, 20242 yr Community Expert Can you connect that disk to the onboard SATA, even if only temporarily, and retry?
June 12, 20242 yr Author Went to try and see if I could swap and not lose another disk in the process. Now I'm finding that the server can't be accessed via network on the GUI to shutdown. Showing not connected on the network. So now I'm having another issue. I'm really beginning to think this is a MOBO issue/CPU issue I'll get back to you after I get the box back up and on the network and see if I can swap
June 12, 20242 yr Hi, I don't know if my post will really help. Sorry for that but to complete the JorgeB last post: Maybe it will erase the disk/array (so do a full backup if you can) = can you try your actual configuration with a fresh Windows or Linux OS? no HDD move, just test as is, and import the drives on Disk Management. Are your drives OK (Windows will show a popup if a drive is failing)? Maybe a S.M.A.R.T. test with CrystalDiskInfo/CrystalDiskMark, or even the WD DiagTool and/or Toshibas' one. If drives are OK, so maybe there's an issue between the SAS Card and the Cage. Did you encounter a "flash" electric interrupt? The system stops but restart instantly! Maybe your system did not appreciate Maybe a UPS issue too, if you have one (I had sometimes at home, and in 1 room, some electric shutdown for a milisecond... and I wasn't aware it was because of one of my UPS for all these years until the server was instantly down too. Since I changed my UPS (not warrantied yet), no more pbs!) Finally, try HDD firmware upgrades too?! Yours symptoms seems to be a hardware issue, as I had some too. unRAID told me that my drives were not healthy. I installed Windows, cleared all the drives, scan them, then return to unRAID and all the symptoms has vanished. All is going fine for a month now. No alerts. Hope you will fix it Good luck C. EDIT: I just saw your last reply, as I was writing mine. Edited June 12, 20242 yr by cbeitel
June 12, 20242 yr Author Hi cbeitel... I agree something is hardware related and yes, latest issues started when we lost power and then when it came back on, the connection literally surged through, blew out my UPS and killed 8 drives on my backup server. My primary seemed to be okay, until all this started the other day. I'm operating without an UPS until new one arrives. I managed to swap drive slots to a MOBO sata without killing a 2nd drive! The Toshiba 20TB started building and seemed to be fine. I stopped and started over again and tried the 20TB WD Red Pro....again, it started building fine. I put the Toshiba back in and currently monitoring but all seems to be fine. The drive I put into the backplane connection is running fine. Of note, it was an 18TB drive. Soooo, I have to wonder what is really going on? I replaced the backplane cage with one that I knew was 100% working, put on new SFF-8643 cables from the 9300-8i, and put in a new 9300-8i. Issue never disappeared. Both my 20TB parity drives are connected to MOBO Sata. Makes me start to wonder if the LSI 9300-8i can't handle 20TB, or greater, disks? If so, we as a community have a bigger issue as this limits your ability to expanded to larger drives. Latest firmware for the board is 2019. Don't recall anyone having 20TB disks out then. Now the question could become, "which HBAs and expanders can handle 20TB or greater disks?"
June 12, 20242 yr Community Expert 44 minutes ago, isrdude said: Makes me start to wonder if the LSI 9300-8i can't handle 20TB, or greater, disks? Many people using them, so don't think that is the problem, though your issue is kind of strange.
June 12, 20242 yr 39 minutes ago, JorgeB said: Many people using them, so don't think that is the problem, though your issue is kind of strange. I found this information on the Broadcom Tech Website (hope it's the same LSI model as yours): Compatible SATA Drives – WD, WDC (broadcom.com) none of your 2 20TB (WD / Toshiba) is listed... maybe you had a chance until then? And this topic too: Solved: 20TB SAS drives not supported in Unraid - Storage Devices and Controllers - Unraid Good luck C. Edited June 12, 20242 yr by cbeitel adding infos
June 12, 20242 yr Author C. Don't think that is the issue as the WD Red Pro and Toshiba N300 are standard SATA drives. I've looked on the forum and quite a few people are using 20TB plus. But Jorge would know if anyone has had issues with 20TBs and SAS 9300-8i....I don't get it. Still thinking it's MOBO related because now I have a NEW and exciting issue... Build was going great, showed around 18 hours or so to build. Quite suddenly the speed dropped to under 3 mb/sec and is showing 85 days to build!!!! That is crazy. (I did fix the network error of not showing on my network.....tip worked its way out a bit. Looks like a re-tip for the cable is next on the list after getting all this sorted out.) tower-diagnostics-20240612-1026.zip
June 12, 20242 yr Community Expert Disk is showing what look like power/connection issues, so either there's still a problem with that or you had some bad luck and the disks are just bad, I really doubt it's a board problem.
June 12, 20242 yr Author I swapped out the sata connector, cleared and reformatted the drive and restarted. It's doing 60 mb/sec now, which is normal for my rig. I'm monitoring but what are the chances that 2 new 20TB drives, from 2 different companies, could both be bad? I'd think I'd have better odds in the lottery... That said, I'm running a 750W PSU. 15 drives, as LSI 9300-8i, a graphics card, a USB cache and USB uNRAID drive and two backplane cages with fans. I wouldn't think that would max out the power, even with an intel i7-7820 CPU....my estimates are I should have a power reserve. Conservative estimate, I'm using 500w of power total.
June 12, 20242 yr Author rebuild holding at around 134 mb/sec. so were fine now. I think after the rebuild, I'll change MOBO's and CPUs since I have a spare. I've a couple of 12TB disk on the array. I may swap one out with the WD Red Pro and see what happens with a newer MOBO and CPU. Try to isolate the issue and completely rule out the LSI 9300-8i. My contribution to the community if I can rule it out!
June 16, 20242 yr Author As I was doing the build, had another disk drop out...it was the one that I swapped the 20TB Toshiba with that was working fine. Soooo, I had another 18TB that was brand new. Put it in but this time, I connected it to the onboard SATA (only have one slot left now for onboard SATA). Booted everything up and both the 18TB and the Toshiba 20TB rebuild and are up and running (Thank goodness I went to 2 parity disks a long time ago!). This eliminates that the problem was 20TB drives. I included the diagnostics for when I received the failure and after recovery. Maybe y'all can see something I can't. It does have me wondering if for some reason, one lane on the 9300-8i has failed (I understand it's usually all or nothing failures.) or maybe the MOBO PCIe slot has gone bad where it isn't fully powering the 9300-8i. I think I have it narrowed down to one or the other. I'm just not quite sure how I'm going to narrow it down to one or the other short of putting in a new MOBO or when my 24 port HBA shows up for my backup server, put it's 9300-8i in. I previously had that 9300-8i attached and was receiving the same issue. BUT, I hadn't updated the firmware either. If y'all can think of anything else to troubleshoot and narrow it down, I'd be grateful. I'm running out of ideas. Thank you disk attached to HBA.zip disk attached to MOBO SATA port.zip
June 16, 20242 yr I had a similar issue happen to me, what was happening to me was because the controller was getting too hot. I'm running an SAS9305-16i. To solve the issue, I moved the card to a slot where I had case fans blow directly on it. The hdds are fine, because the controller got so hot, it will report erroneous errors. So check your controller temps. You can do that using storcli64, using the following command: Quote ./storcli64 /c0 show all |grep -i temperature You should be able to find the utility on Broadcom's site.
June 17, 20242 yr Author Solution shpitz461...big thank you....I think you confirmed what I discovered this afternoon. I had everything back up and operational...stressed it with some file loads, etc. Then thought, "what the heck, I've a 20TB WD Red Plus, lets swap out a 12TB and put this on. Everything was fine until I was at 75% complete on the data rebuild on the WD drive. Suddenly, disk 5, my Toshiba drive (AGAIN) got 24,000+) UDMA/CRC errors. Since I was already working with a different 9300-8i, decided to swap out the MOBO to eliminate the PCIe slot as a culprit. Thankfully, I purchased a dual 92mm Graphic Card Fan to blow right on that thing! It came in the mail today, so hopefully that is the fix! tower-diagnostics-20240617-1334.zip
June 19, 20242 yr Author Server completely back up and operational after the MOBO swap and adding a big double cooling fan to blow on the HBA. No problems on rebuild, even stressed the system by adding video files while rebuilding. No problems, rebuild speed held steady averaging around 140 mb/s . Looks like cooling did the trick.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.