February 2, 20251 yr So I've got drives that failed again on Unraid. This is the second time in 30 days. This time I will be losing data I'm guessing since I lost both parity drives and a data drive that had about 8TB of stuff on it. What is weird this time is that unlike previous times, where all the drives stayed in their respective slots and seemed fine with an emulated drive, this time, when I shut down the array, all the drives have been removed from their slots and a totally different array drive that no errors were thrown against doesn't even show as having ever been populated in the array. All drives (except cache) are showing this way (2 parity and 10 array). I've never seen this happen before and curious what next steps would be. Start array and see what happens (after putting all the drives back where they belong) or what. Just curious why all the drives would disappear from their slots like this. Also attached the diags. I can only imagine this is an ongoing hardware issue but honestly don't know. The only other change in the environment is going from 6.9.x to 6.12.13 over the last few weeks and then to 6.12.15 last week. On another note, I do think I had turn on the spin down of drives after 2 hours before both instances occurring. But, this last time, spin down was on for 2-3 days before failure. Any thoughts greatly appreciated. sol-diagnostics-20250202-1348.zip
February 3, 20251 yr Community Expert Are you sure these are the correct diagnostics? They don't seem to agree with your description or screenshot. Array started and all drives mounted.
February 3, 20251 yr Author That what the diag gave me. Everything is dated 2/2 and the syslog starts 1/25 but ends 2/2.
February 3, 20251 yr Community Expert Did you create these diagnostics by going to Tools - Diagnostics after you noticed this problem? Or did you get them from the flash drive?
February 3, 20251 yr Author Tool > Diagnostics before I reboot or do anything else. I believe diags were pulled after I shutdown the array though, but not 100% sure.
February 3, 20251 yr Community Expert Those diagnostics show the array started and all disks mounted. Filesystem Size Used Avail Use% Mounted on rootfs 32G 332M 32G 2% / tmpfs 128M 8.0M 121M 7% /run /dev/sda1 29G 1.3G 28G 5% /boot overlay 32G 332M 32G 2% /lib overlay 32G 332M 32G 2% /usr devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 128M 17M 112M 13% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/addons tmpfs 1.0M 0 1.0M 0% /mnt/rootshare /dev/md1p1 11T 7.4T 3.7T 67% /mnt/disk1 /dev/md2p1 11T 7.3T 3.7T 67% /mnt/disk2 /dev/md3p1 11T 7.6T 3.4T 70% /mnt/disk3 /dev/md4p1 7.3T 52G 7.3T 1% /mnt/disk4 /dev/md5p1 9.1T 6.7T 2.5T 74% /mnt/disk5 /dev/md6p1 9.1T 6.2T 3.0T 68% /mnt/disk6 /dev/md7p1 7.3T 5.2T 2.2T 71% /mnt/disk7 /dev/md8p1 11T 8.2T 2.8T 75% /mnt/disk8 /dev/md9p1 7.3T 5.4T 2.0T 73% /mnt/disk9 /dev/md10p1 7.3T 52G 7.3T 1% /mnt/disk10 /dev/sdn1 3.7T 2.4T 1.3T 66% /mnt/cache /dev/sdo1 1.9T 152G 1.7T 9% /mnt/vm shfs 91T 54T 38T 59% /mnt/user0 shfs 91T 54T 38T 59% /mnt/user //10.5.1.10/drive_x 3.7T 1.6T 2.1T 44% /mnt/remotes/10.5.1.10_drive_x /dev/loop2 75G 36G 38G 49% /var/lib/docker
February 3, 20251 yr Community Expert Those diagnostics agree with your description and screenshot. Do you have a current backup of flash?
February 3, 20251 yr Community Expert You can always download a zipped copy of flash from Main - Boot Device - Flash - Flash Backup. Unraid Connect plugin will keep a backup of your flash in Unraid Cloud. Not clear what happened, your array assignments are kept on flash in config/super.dat, and pools in config/pools, but I didn't notice any other symptoms of flash problems. The super.dat in those 2 different diagnostics you posted are not the same though. Here are your disk assignments from those first diagnostics you posted, disk0 is parity, disk29 is parity2: Jan 25 22:18:58 SOL kernel: md: import disk0: (sdb) ST12000VN0008-2YS101_ZRT1TFXR size: 11718885324 Jan 25 22:18:58 SOL kernel: md: import disk1: (sdh) ST12000VN0008-2YS101_ZRT205L5 size: 11718885324 Jan 25 22:18:58 SOL kernel: md: import disk2: (sdk) ST12000VN0008-2YS101_ZRT205P8 size: 11718885324 Jan 25 22:18:58 SOL kernel: md: import disk3: (sde) WDC_WD121KFBX-68EF5N0_5QK0A7SB size: 11718885324 Jan 25 22:18:58 SOL kernel: md: import disk4: (sdf) WDC_WD80EDAZ-11TA3A0_VGGEVKYG size: 7814026532 Jan 25 22:18:58 SOL kernel: md: import disk5: (sdi) WDC_WD101EFBX-68B0AN0_VCHVHXMP size: 9766436812 Jan 25 22:18:58 SOL kernel: md: import disk6: (sdl) WDC_WD101EFBX-68B0AN0_VCHV5SGP size: 9766436812 Jan 25 22:18:58 SOL kernel: md: import disk7: (sdd) WDC_WD80EDAZ-11TA3A0_VGKL9X2G size: 7814026532 Jan 25 22:18:58 SOL kernel: md: import disk8: (sdg) ST12000VN0008-2YS101_ZRT1TEY7 size: 11718885324 Jan 25 22:18:58 SOL kernel: md: import disk9: (sdj) WDC_WD80EDAZ-11TA3A0_VGH98A0G size: 7814026532 Jan 25 22:18:58 SOL kernel: md: import disk10: (sdm) WDC_WD80EDAZ-11TA3A0_VGH8RSXK size: 7814026532 Jan 25 22:18:58 SOL kernel: md: import disk29: (sdc) WDC_WD120EFBX-68B0EN0_D7GD8ZHN size: 11718885324 Jan 25 22:18:58 SOL emhttpd: import 30 cache device: (sdn) Samsung_SSD_870_EVO_4TB_S6BBNS0TA06183Z Jan 25 22:18:58 SOL emhttpd: import 31 cache device: (sdo) Samsung_SSD_870_EVO_2TB_S6PNNJ0W313005A Jan 25 22:21:50 SOL emhttpd: shcmd (238): mount -t xfs -o noatime,nouuid /dev/sdn1 /mnt/cache Jan 25 22:21:50 SOL emhttpd: shcmd (241): mount -t xfs -o noatime,nouuid /dev/sdo1 /mnt/vm
February 3, 20251 yr Author This server isn't connected to Unraid Connect. I've tried linking it multiple times but the web site doesn't like whatever I'm trying to give it. My new 7.0 server is linked just fine, but that license was purchased after Unraid Connect came in to existence. My older license was purchased quite sometime ago.
February 3, 20251 yr Community Expert If you have another server, you might consider storing flash backup for each on the other.
February 3, 20251 yr Author That is the plan. It is a new server still in pieces and being tested with 7 to determine if I go that route along with xfs, zfs potential decisions. Also upgraded to 10g network recently.
February 5, 20251 yr Author Any thoughts on how to handle the fact all the drives got removed from the array? Put them back under a New Config or do a New Config and have them come up as unassigned? After some digging, I'm wondering if there is an issue with spinning up/down disks and the Perc controller. Seems both times I've had this issue, the spin up/down had just been turned on. The syslog did show some megaraid_sas errors about the time some spin up/down was occurring. Wondering if the "failed" array drive has actually failed or it was purely the controller. I just checked the drive from the previous issue a couple of weeks ago and the data appeared in tact. Feb 2 13:32:26 SOL emhttpd: spinning down /dev/sdj Feb 2 13:33:03 SOL emhttpd: read SMART /dev/sdj Feb 2 13:33:04 SOL emhttpd: spinning down /dev/sdm Feb 2 13:33:04 SOL emhttpd: spinning down /dev/sdd Feb 2 13:33:04 SOL emhttpd: spinning down /dev/sdi Feb 2 13:33:43 SOL emhttpd: spinning down /dev/sdl Feb 2 13:33:46 SOL emhttpd: spinning down /dev/sdf Feb 2 13:34:13 SOL emhttpd: read SMART /dev/sdd Feb 2 13:34:26 SOL emhttpd: read SMART /dev/sdf Feb 2 13:34:49 SOL emhttpd: read SMART /dev/sdi Feb 2 13:34:50 SOL kernel: megaraid_sas 0000:02:00.0: 4790 (791836486s/0x0020/DEAD) - Fatal firmware error: Line 977 in ../../dm/src/dm.c Feb 2 13:34:50 SOL kernel: Feb 2 13:34:50 SOL kernel: megaraid_sas 0000:02:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009 Feb 2 13:34:50 SOL kernel: megaraid_sas 0000:02:00.0: FW in FAULT state Fault code:0x10000 subcode:0x0 func:megasas_wait_for_outstanding_fusion Feb 2 13:34:50 SOL kernel: megaraid_sas 0000:02:00.0: resetting fusion adapter scsi1. Feb 2 13:34:50 SOL kernel: megaraid_sas 0000:02:00.0: Outstanding fastpath IOs: 3 Feb 2 13:34:50 SOL kernel: megaraid_sas 0000:02:00.0: Reset not supported, killing adapter scsi1. Feb 2 13:34:50 SOL kernel: sd 1:0:7:0: [sdi] tag#721 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=0s Feb 2 13:34:50 SOL kernel: sd 1:0:7:0: [sdi] tag#721 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00 Feb 2 13:34:50 SOL kernel: sd 1:0:7:0: [sdi] tag#654 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=0s
February 5, 20251 yr Community Expert 5 hours ago, jmbailey2000 said: Reset not supported, killing adapter scsi1. The controller is resetting, don't know why, try a different PCIe slot if possible, but keep in mind that RAID controllers are not recommended.
February 5, 20251 yr Author It is the standard Perc H730p controller for a Dell R730XD but running in HBA mode, not raid mode. It is mounted to the motherboard and not inserted in to a slot. Are you saying that is a problem because I see hundreds of people doing that with raid cards flashed to HBA mode. My old disk self was the same way with an LSI card flashed to HBA mode.
February 5, 20251 yr Community Expert 10 hours ago, jmbailey2000 said: megaraid_sas With this driver, it's not a true HBA.
February 5, 20251 yr Author Well, ouch. Any suggestions on a replacement? Scouring a ton of messages in different places, guessing the H330 Mini would be better?
February 6, 20251 yr Author Good read on the controllers. Thanks! Any thoughts on how to handle the fact all the drives got removed from the array? Put them back under a New Config or do a New Config and have them come up as unassigned? Just trying to find out what might happen/what top expect if I reboot or I should do a New Config before rebooting and reset up the array.
February 6, 20251 yr Community Expert If you are going to replace the controller you will have to reboot. New Config https://docs.unraid.net/unraid-os/manual/storage-management/#reset-the-array-configuration Parity should still be valid if no disks have been removed or added. Parity2 should still be valid if no disks have been removed or added or had their order changed. If you get them reassigned exactly as they were, you can check the Parity valid box before starting the array. You should probably do a parity check anyway.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.