October 12, 20241 yr Pre-Hurricane I lost an M.2 which was part of a Raid 1 cache with my appdata. I didn't open the case to troubleshoot but the bios didn't recognize the drive so I assumed it failed. I restored the Raid 1 using a different M.2 in the box and made a note to investigate later. This was about one week ago. Along came Hurricane Milton. As I was getting my phone out to shutdown my Unraid server, we had a strange power fluctuation that went on for about a minute. Living in FL, we are used to power fluctuations during storms but this one was strange. Seemed to go on forever. I wasn't worried as my server is connected to a UPS (CyberPower CP1500PFCLCD PFC Sinewave UPS 1500VA) which has "sinewave" power protection. When I did go to shutdown the server, I found that it had rebooted during the power fluctuation. I double checked to ensure the server was plugged into one of the Surge+Battery slots and then shut it down and unplugged it for the duration of the storm. The next day I started the server again to find out that ... 1. TWO DEAD M.2s: The Two M.2s that I have in Raid 1 config which held appdata were both listed under "Unmountable disk present". 2. ONE M.2 ROSE FROM THE GRAVE (SORT OF): The M.2 that was not showing up in the bios Pre-Hurricane was now showing up in the bios and in Unraid. However, when I tried to mount the drive it apparently fails (without message). All it does is display "Reboot" next to the drive (see attached image). Rebooting didn't help. 3. All the spinning rust drives and the boot flash appear to be fine. I tried formatting by checking the "Yes, I want to do this (format)" for the two M.2s but when I click format the browser window refreshes a couple of times and then it apparently fails as it returns to the "Format" button being greyed out and both stay listed under "Unmountable disk present". To make things stranger ... 1. We never lost power. None of the clocks need resetting. Additionally, when we do get power surges usually some low voltage stuff like garage door openers get fried first but nothing else in the house is damaged. The server, which again was connected to a UPS, was the sole recipient of the demonic targeted surge. 2. All three of the M.2 drives pass a SMART short self-test even though one of them won't mount and the other two won't format. 3. I restored the appdata by using the "Backup/Restore Appdata" functionality (amazingly easy) but when I logged in today to write this, all of the drives were blank under the "Main" tab (see attached image). I checked that Plex would play a movie and it did. Thus, the drives were working even if they weren't showing. When I rebooted it returned to normal. At this point I am faced with two unlikely possibilities. 1. The surge had nothing to do with the failures and the MB (ASUS TUF Gaming Z790-Plus WiFi LGA 1700) is just bad. This would explain why an M.2 disappeared and came back. But the timing is amazingly coincidental. 2. The surge killed something but somehow didn't impact anything else in the house and sort of restored the missing M.2. The next obvious troubleshooting step would be to try the M.2s in a different computer but getting everything out of the closet is difficult for us old people. I've attached the syslog for reference. Any suggestions would be appreciated. syslog.txt
October 12, 20241 yr Community Expert Please post the complete diagnostics and the output from btrfs fi show
October 13, 20241 yr Author Thanks for the help! Label: none uuid: 90031dc7-05cb-47c5-84e5-c64b59d0f80e Total devices 2 FS bytes used 7.01TiB devid 1 size 9.09TiB used 7.08TiB path /dev/sdi1 devid 2 size 9.09TiB used 7.08TiB path /dev/sdh1 Label: none uuid: fa25ed97-097e-4b62-94de-f31b162ef879 Total devices 1 FS bytes used 4.80GiB devid 1 size 100.00GiB used 8.02GiB path /dev/loop2 Label: none uuid: 4d281c88-5bbf-4b6f-b2d1-55bf9ecc13f2 Total devices 2 FS bytes used 189.06GiB devid 2 size 1.82TiB used 198.03GiB path /dev/nvme1n1p1 devid 3 size 1.82TiB used 198.03GiB path /dev/nvme2n1p1
October 13, 20241 yr Community Expert nvme0 is interfering with the pool, since before it was part of it, you have to wipe it, offline it, or physically disconnect it, to test, you can offline it, since it's the safer/easier option, type echo 1 > /sys/block/nvme0n1/device/delete The restart the array, if the pool still doesn't mount post new diags.
October 14, 20241 yr Author I get bash: "/sys/block/nvme0n1/device/delete: Permission denied". I am logged in as root. An internet search seems to be limited to using chmod to change permissions but when change to the folder there isn't a file called delete. I get this ... root@Kraken:/sys/block/nvme0n1/device# delete bash: delete: command not found I would expect delete to be a command (not a file) but in this line "echo 1 > /sys/block/nvme0n1/device/delete" it is used like a file. I love the idea of testing this without opening the case.
October 14, 20241 yr Community Expert Solution Sorry, I assumed that since that command works for any sdX device, it would also work for nvme devices, but just tested to confirm and it doesn't, alternative is to physically disconnect the device or to wipe it, wiping it is not a big risk, since despite being part of the same filesystem, it's a much older version: Oct 12 14:56:51 Kraken kernel: BTRFS: device fsid 4d281c88-5bbf-4b6f-b2d1-55bf9ecc13f2 devid 1 transid 131136 /dev/nvme0n1p1 scanned by udevd (1105) Oct 12 14:56:51 Kraken kernel: BTRFS: device fsid 4d281c88-5bbf-4b6f-b2d1-55bf9ecc13f2 devid 3 transid 163236 /dev/nvme2n1p1 scanned by udevd (1128) Oct 12 14:56:51 Kraken kernel: BTRFS: device fsid 4d281c88-5bbf-4b6f-b2d1-55bf9ecc13f2 devid 2 transid 163236 /dev/nvme1n1p1 scanned by udevd (1123) fsid is the same for all, means they belong to the same filesystem, but the transid for nvme0n1 is much lower, meaning older data, it will be out of sync with the other ones, and the pool only has two devices, according to the info you posted earlier: Label: none uuid: 4d281c88-5bbf-4b6f-b2d1-55bf9ecc13f2 Total devices 2 FS bytes used 189.06GiB devid 2 size 1.82TiB used 198.03GiB path /dev/nvme1n1p1 devid 3 size 1.82TiB used 198.03GiB path /dev/nvme2n1p1 If you want to wipe you can with: wipefs -a /dev/nvme0n1
October 16, 20241 yr Author Sorry for the delay. This wipefs worked after a reboot. My other two M.2 is in Raid1 came back on reboot without issue. Thanks, you saved me a ton of time! What do you think I should do with the zombie M.2 (the one that came back to life). Remove it, clear it and put it back in service, ??
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.