isrdude

BTW Jorge, this all was happening a few weeks ago when you helped me. Same issue and we thought it was the toshiba drive. I swapped it in another slot and it worked,. Put it back in original, it worked until a couple days ago. That's when I started the above gyrations. It's the same exact spot and disk #. Changing cables, backplane cage, and HBA didn't help. I think this goes deeper but don't know what. In fact, when I get the error and the drive stops, I can't shutdown the array on the GUI either. I either must reboot or full shutdown

Backplane cage powered by 2 x 4-pin connectors. Using a 750W PSU. Power has never been an issue. I'll try swapping drive slots but last time I tried, it showed drive I moved around was missing and still had trouble with the original drive. I'll get back to you JorgeB

Really going nuts! Have a 13 drive server with 2 parity drives. All was fine until 3 days ago. Had a toshiba 20TB in my disk 5 slot. Pulled up GUI and showed drive was red X'd and data emulated due to errors. This drive had been problematic since I bought it so I decided I'm done. Put in a good old reliable 20TB WD Red Pro. Same thing has happened! I removed the WD from disk 5, removed partition, cleared it, then reformatted. Got things back up, and it did it again, almost immediately! I swapped out new cables from the LSI 9300 to the backplane cage. Started over, same issue. I changed out the backplane cage from another box that I knew had a good one. Again, same thing. Finally, I bought another LSI 9300-8i. SAME THING.....AAARRGGHH Do y'all think that it's possible the MOBO PCI-e slot is bad causing just one disk to have errors? I can't think of anything in the MOBO bios that could cause it since this all just happened out of nowhere. I'm at wits end. If anyone has any thoughts, I'd appreciate it. The only silver lining it's only one drive I lost so rebuild is easy.....if I can ever get to that point. tower-diagnostics-20240611-1302.zip

Did some more detective work today. Hooked all 8 drives back into external enclosure, connected to my main server via USB. My main uNRAID server will see each disk as unassigned. I used Krusader to check the contents. Sadly, the contents come up, show <DIR> but nothing underneath. When we had some T-Storms, we had a power outage. When power came on, it had an unusual spike that smoked my UPS too. I think that surge ran up the UPS into the secondary box on my backup server and caused a static surge that wiped the disk contents. So now, my backup server shows all the disk missing, which makes sense since their is no data now. I guess only thing to do now is clear the disk from the backups historical disk, set up remaining disk as a New Config, preserving current data, then add the disk back, one by one, format and then let them rejoin the array.....since the data on the drives appear to be gone, might as well see if they are stable and useful again...... Thanks for all the help folks.

The enclosure connects to the server via USB connection, but internally its a single drive SATA connection. What I did was what you suggested. I held the enclosure in my hand, turned it on and felt the centrifugal force of the spin. That's why I think they were spinning up. I tried powering the secondary box that the 9300-8e routes to with 2 different PSU's. Nothing! I'm getting some or those combined SATA power/data cables. I want to see if I connect the separate cables into those then plug straight into the drives will make a difference. It's possible, everything is so tight that I'm not getting good connections anywhere. Low probability that is happening but I want to eliminate all that I can. When I get those, I'm going to run and will post more diagnostic data.

Hello Jorge.... I am not sure of anything right now. I took each drive out, put them into an USB enclosure, one by one, and attached to my primary server. They show up as unassigned and whether I want to format or not....pretty good indicator they are spinning up without going deeper into them and possibly losing data. So, I tore both boxes apart, put them back to original build. Then I took all the drives, they are shucked, so I reapplied thermal tape on all the 3 pins. Put everything back, nothing. Tried firing them up with just one connected back to the 9300 8E....nothing. Not sure if they are spinning or not as they sounded pretty quiet and I didn't hear anything click into park when I shut down the box. I'm out of ideas. I suspect, I'm going to need to replace all the drives, transfer data from old drives to the new, then set up as a new config. Problem is two fold....a ton of $$ for drives and when you're talking 12TB and 18TB drives, we're talking where will I store the data until I can transfer. Oh the fun!

Well, good news, bad news....Y'all got me thinking about checking all the drives again. Bought a new usb enclosure and did the freezer trick. Guess what, drives were running when I put into enclosure and tested on my primary server. BUT, my backup server still isn't reading the 8 (I mistyped, actually 8 all along, not 7 down). . So, I decided, get another MOBO and see if it wasn't the PCIe slot that was bad. Nope. The bios can see the SAS LSI9300-8i, but Unraid isn't reading them So now I've swapped out the LSI9300, all the cables that connect the controller to the 8 drives in the satellite box, and the mobo. Unraid still shows the 8 as missing with the dreaded red "X". I checked bios on the MOBO and it's latest. Unraid is latest version, too. Interestingly, when I would boot, I could watch the Controller initialize and see all 8 drives, but that isn't coming up now before it goes to the bios screen. Not sure what is going on since, 3 weeks ago, everything was fine. I keep thinking BIOS setting but not sure what since it's doing the same thing on both MOBOs and I've configured them the same. MOBO: ASUS Rog Strix X299-E gaming CPU: Intel i7-7820X Diagnostics attached. tower2-diagnostics-20240529-2132.zip

thanks Kilrah. Was thinking along those lines but wanted to be sure before I messed up the array more than it is.

YES, I'm sure the drives are dead. I did hook them up via a USB enclosure to my working primary server. All Drives DOA. To answer the question of how? Not sure but I think the PSU spiked in the secondary box since the primary was unaffected. Yes, I bought new cables and tried those....yes, I bought a new PSU....as for the controller, I bought a replacement thinking it could be a controller issue. Nothing worked. Which is why I didn't include the diagnostics as these drives are all in the secondary box cabled to the main box running unraid. Secondary box is just drives, PSU , nothing else. Main box has the mobo, SATA Controllers, drives, Unraid program, etc. I've already tore down the connected box to send out the drives. So running diagnostics is just going to show a working primary rig that shows missing drives. That said, I talked to 3 data recovery companies. And all they can promise is possibility of data but not the actual drive itself. So my question is, based upon all of your kind advice, how do I get the data to the array when I put in the new drives that I'm purchasing, not what the recovery company is providing? Just transfer data externally using Krusader or MC to the drives then rerun parity check?

Hi all. I have a server with 20 data drives and 2 parity. 7 drives, that were in a separate box from the main box, connected via LSI 9300 controller, died on me and now show as missing. Haven't included my logs since, I know the drives are paperweights and hopefully data can be recovered. The good news is the primary box with the parity drives and the 13 other drives are rock solid. Looks like my PSU in the secondary box did something wonky and torched the drives. I am sending the drives to a data recovery service. I need help and advice on how I can get back up and running once I get the data back so my parity won't be destroyed during the process. First, I think I need to stop the array and take the parity drives offline so they don't get corrupted. Do I then "X" out the old drives that are shown as missing ? After I get the dead drives out of the systems memory, with parity drives temporarily uninstalled, I assume I can power down and installed the replacement drives? When I put in the new replacement drives, I then format for XFS but then what? Do I: 1) transfer the recovered data (HOW?) then assigned the drives? Or 2) do I assign the drives then transfer the data? After getting the data transferred and then the drives assigned, Do I put the parity drives back online and go into "New Config" ensuring parity drives remain as they were and the 7 new drives come online as data drives? I'm assuming at that point, with new config that Parity would see the data hasn't changed just the new drive assignments, yes? I think I'm close on what I need to do, just not the order or how to transfer the data to the replacement drives. Any help, advice, guidance is gratefully appreciated.

Jorge.....you were correct again. I bought a Toshiba N300 Pro 20TB, put it into the same SAS slot I had trouble with......no problems at all. Its rebuilding data just fine without any issues!!! Thanks again

Hmmmm...I may try different brand. I was upgrading WD disk to another WD disk. Just seems strange it would be a problem all of the sudden. Well, I'll worry about that when the time comes. Thanks again

JorgeB.... Thanks! That tip worked. Why wouldn't work on my SAS though once I went from an 18 TB to 20TB drive? Next question, since it didn't work on the SAS connections to build a parity, will I have the same problem if I try to slot in 20TB data drives? Thanks again for the helpful tip!

I'll try that....thanks.. :)

Time to call in the Cavalry for help! I have 2x18TB parity drives that I want to upgrade to 20TB. I hung the drives on an external dock, precleared, formatted, etc. Everything check good. I only swap out parity drives 1 at a time. So, put the first one in. As soon as it started errors started coming, then I received the drive disabled message and the dreaded red X by the drive name (Parity 1). Tried with the 2nd new drive same thing. Took both of them to my main computer and ran the WD Smart test tool. Both checked out. Hmmmm? Tried again, same problem. Smart guy I am, I checked all connections internally, ensured my BIOS was up to date as well as Unraid. Still same problem. Next, I tried to swap slots thinking if its a bad cable or power connection, this will tell me. Nope, same issue. Put original 18TB drive back in. No problems. Unraid accepted it and is rebuilding the Parity. I should also add, these are "shucked" drives. Never had an issue with shucked drives before either. Thermal tape off the #3 pin and you're golden. Not today! The shucked drives are Elements 20TB...According to WD site, these are CMR drives. I'm stumped. Thanks in advance for the help. In the 12 years I've been running UnRaid, I've never ran into a issue upgrading Parity Drives tower-diagnostics-20240408-2257.zip

I'll try that. Thank you. JorgeB

OK, still dealing with this issue. Everything works on current setup with older Unraid. My box has latest BIOS as stated above. I'm stuck on Unraid 6.11.5. I'm wondering...I've a LSI SAS 9300-8e PCI Express that is connected to another box that is powering 8 hard drives. As I said everything works until I go to latest version and then I get in Bizroot loop. Would there by chance need to be a script edit notation I need to put in with the newer Unraid versions? It's the only thing I can think of since it's working until I update. Any thoughts folks? Please!!! Been dealing with this since June Thanks

That's exactly what I thought and did a bios update. Didn't fix it. Bios I'm running on my ASUS mobo is 2022/07/12 so that's fairly current. Scratching my head on this. What's killing me is I'm using basically the same ASUS mobo for my primary with a slightly better CPU and same amount of memory. That mobo is using the older bios that was on the problem unit and it updated to Unraid 6.12 with no problems. Problem box can't do it with the older or newer bios. The things that make us go "hmmmmm!"

It will only boot after I revert to 6.11.5 If I put 6.12 on my regular boot usb or on an unraid stock usb, reboots after bizroot

Another thread has similar issue. Couldn't find anything common with that one. df -h data shows: Filesystem Size Used Avail Use% Mounted on rootfs 16G 854M 15G 6% / tmpfs 32M 652K 32M 2% /run /dev/sda1 7.5G 1.6G 5.9G 22% /boot overlay 16G 854M 15G 6% /lib/firmware overlay 16G 854M 15G 6% /lib/modules devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 16G 0 16G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 412K 128M 1% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/addons tmpfs 1.0M 0 1.0M 0% /mnt/rootshare /dev/md1 9.1T 7.7T 1.5T 85% /mnt/disk1 /dev/md2 11T 9.4T 1.7T 86% /mnt/disk2 /dev/md3 11T 8.6T 2.4T 79% /mnt/disk3 /dev/md4 9.1T 6.4T 2.8T 71% /mnt/disk4 /dev/md5 11T 8.2T 2.8T 76% /mnt/disk5 /dev/md6 9.1T 6.4T 2.8T 71% /mnt/disk6 /dev/md7 11T 8.2T 2.8T 76% /mnt/disk7 /dev/md8 11T 8.2T 2.8T 76% /mnt/disk8 /dev/md9 7.3T 4.6T 2.8T 63% /mnt/disk9 /dev/md10 11T 8.2T 2.8T 76% /mnt/disk10 /dev/md11 11T 8.2T 2.8T 76% /mnt/disk11 /dev/md12 11T 8.2T 2.8T 76% /mnt/disk12 /dev/md13 11T 8.2T 2.8T 76% /mnt/disk13 /dev/md14 11T 8.2T 2.8T 76% /mnt/disk14 /dev/md15 9.1T 6.4T 2.8T 71% /mnt/disk15 /dev/md16 5.5T 1.7T 3.9T 31% /mnt/disk16 /dev/md17 9.1T 3.7T 5.5T 41% /mnt/disk17 /dev/md18 5.5T 39G 5.5T 1% /mnt/disk18 /dev/md19 9.1T 3.7T 5.5T 41% /mnt/disk19 /dev/md20 7.3T 1.9T 5.5T 26% /mnt/disk20 /dev/md21 11T 5.5T 5.5T 51% /mnt/disk21 /dev/nvme0n1p1 932G 3.7M 932G 1% /mnt/cache_nvme shfs 201T 131T 70T 66% /mnt/user0 shfs 201T 131T 70T 66% /mnt/user /dev/loop2 10G 4.4G 5.1G 47% /var/lib/docker root@Tower2:~# Diagnostices finally downloaded after 3 tries. Attached. The only issue I had prior to this was I had a drive go bad that I replaced, rebuilt. All of that was finished before trying to update the OS Ran update assistant before trying and it gave me OK's across the board and good to update. Tried the stock load OS and ran into same issue, went into reboot after bizroot OK. Makes me think if something in bios needs changed for 6.12 Thanks for any help or ideas tower2-diagnostics-20230619-1453.zip

I compared abhi.ko's df -h output with mine that goes to same reboot from bizroot. Can't see any common threads that would help solve the original problem since my tmpfs isn't at 100%. Other than minor differences, all mirrors up closely. Guess I can't find a common anomaly that can help. Of course I can't download my diagnostics for some reason to compare both. Oh well, I tried.......

I have the same problem. I did try booting from a "stock" Unraid stick and same issue. Gets to bizroot then reboots. Reverted back to ver. 6.11.5. Since I can't get it to go through boot sequence with 6.12, no logs. What else can I send you to help diagnose? BTW, only issue I had was a disk that died. Replaced it and rebuilt it prior to trying the OS upgrade.

Calibre had been working fine without any hicups. Could access remotely and via desktop to get my books. Suddenly today, I get this: error launching startup command: failed to close file descriptor for child process (operation not permitted) What the????? I know Calibre has had a lot of updates over the last week could that be the issue? tower-diagnostics-20220920-1413.zip

Thanks. I'll try that. It sounds like good, sound, solution to the issue.

Ah, thank you! Much appreciated