April 15, 20242 yr Hello, my Unraid server becomes unresponsive (both GUI and SSH) I am facing some trouble investigating what is the issue. I was changing my home routers and configured everything so that old static IPs (DHCP) are exactly the same on new router, as well as port forwarding and IP subnet (192.168.3.*) and what not. Once done, all devices connected as expected both on lan and wlan. First time I logged to the Unraid server everything looked as usual, but it hang at some point. so I assumed there was an issue with IP addresses, but soon realized it wasn't. so since I had this issue, I have turned off docker service to exclude any problem from my containers but it still freezed, and then removed the parity drive as well as I was suspecting it to be the issue, but same result. I had to force shutdown and boot again every time . But I still have the same issue where the system hangs with "starting array" showing, but everything becomes unresponsive. I can sometimes do the ssh login when its hang, but I don't really get through with the ssh procedure, so the terminal doesn't switch to ssh commands (I forgot the default first line that appears after an ssh, but it stays there and doesn't give me new root user command. I can provide more details on this if needed); Then I checked the logs and found some nvme related errors, so I did the following recommended trick in Syslinux Configuration of flash device but still not luck: nvme_core.default_ps_max_latency_us=0 pcie_aspm=off I don't have an enclosure to test the nvme drive on my personal computer, and really hope it is not dead, I have (if I understand correctly) all my containers settings and configurations on that drive and some personal files I would really not like to loose. Final thing I came up with was turn off auto start for the array, and it seems the system stays responsive. So I tried starting the array in maintenance mode as to help me investigate further, so I naturally clicked on the nvme drive and tried a "Check Filesystem Status" with following logs: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_fdblocks 43163533, counted 44043003 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 But nothing happens afterwards, and the status stays "running". After a few minutes, the system becomes unresponsive both GUI and SSH, I even loose ping response at that point . This is my first time posting for help, I understand from all posts I've been using as reference in the past that a diagnostic file is mostly needed, and will attach mine. What can I do at this point ? Is there any hope to recover ? malkymediahub-diagnostics-20240416-0016.zip
April 16, 20242 yr Author ok, i had an attempt without the -n option and got the following log: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. Since (I suppose) mounting the filesystem makes everything hang, I wanted to proceed with the -L option, but I would rather be safe and prudent at this point and see if this is still your recommendation. Shall I proceed ?
April 16, 20242 yr Author Thanks JorgeB, I ran the fs check with -L option. I had the following log during the first 2 hours (as reference for future users): Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_fdblocks 43163533, counted 44043003 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 At some point I thought the system hang again, but I was able to ping it and open/refresh tabs of the GUI. So I kept the check running and when I came back to check it 6 hours later, the tab that had the check running had refreshed and I couldn't get the full resulting log. However, on the main GUI section, the array was not running in maintenance anymore, so I tried my chances assuming the filesystem check had an effect and started the array again in normal mode. The array now starts normally without any hanging (while still having the parity drive unselected) and I now get the following message next to the NVME drive: Unmountable: Unsupported or no file system with the following prompt in the "array operation" section: Format will create a file system in all Unmountable disks. ( ) Yes, I want to do this I also noticedb that docker services are now on, and all containers are stopped by default with a few containers totally missing (sonarr, radar, actual, qbittorent, tautilli, kavita, lidarr, overseerr and probably other containers I don't have in mind right now and I honestly do not have the memory to check which personal files are also missing). I tried starting a few containers, some gets the "Execution error: Server error" popup and others are running ok. Started container: code-server dashdot Dozzle firefox flaresolverr it-tools speedtest-tracker Containers receiving error popup when starting: CloudCommander duplicati homarr plex Portainer-CE youtube-dl-material Containers that won't start but without a popup message: Czkawka MediaInfo My assumption is that some containers had their data folders in unmounted NVMe drive thus cannot start, while other containers were using the other 2TB SSD 2.5" mounted drive I have as another source of their data folder, so able to run normally. Based on that, I figure I have to format the nvme drive, and rebuild the missing containers from scratch, and use the data available from currently available containers but can't start so I can replicate the configuration I made AFTER mounting the NVMe as a new "formatted" drive ? Or is there still another way to recover my configurations ? I am hoping for the latter, but I guess current situation isn't as catostraphic as I thought since I at least have a starting point now. do you please have any recommendation I am unaware of on how to move forward with this ? Quote Also, since I was noob enough to remove parity drive at some point of my troubleshooting, I am unable to use the most useful feature of Unraid to rebuild the damaged drive. A piece of advice for newbies like myself: dont remove the parity drive during your troubleshooting, I think I could've recovered everything at this point if parity was intact. malkymediahub-diagnostics-20240417-0801.zip Edited April 17, 20242 yr by carlos16196 attached new diagnostics
April 17, 20242 yr Community Expert Solution Disk1 is still unmoutable, there may be a device problem, run an extended SMART test on it.
April 17, 20242 yr Author Ok, thanks JorgeB, you're help is highly appreciated. I'm trying the extended SMART test, but as soon as I click on it it starts and stops within less than a second (tried both before and after starting the array in maintenance mode). also the "SMART short self-test" above becomes disabled as soon as I try the extended test. As an update that might be useful, I was able to witness what happens when doing the -L filesystem check (that I tried to repeat), and the check starts with this log: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... totally zeroed log and on the console log I get the following: Apr 17 11:46:21 MalkyMediaHub ool www[14394]: /usr/local/emhttp/plugins/dynamix/scripts/xfs_check 'start' '/dev/md1p1' 'Lexar_SSD_NM620_512GB_MLJ3992009295P1100' '-n' Apr 17 11:46:25 MalkyMediaHub ool www[14481]: /usr/local/emhttp/plugins/dynamix/scripts/xfs_check 'cancel' '/dev/md1p1' Apr 17 11:46:30 MalkyMediaHub ool www[14460]: /usr/local/emhttp/plugins/dynamix/scripts/xfs_check 'start' '/dev/md1p1' 'Lexar_SSD_NM620_512GB_MLJ3992009295P1100' '-L' Apr 17 11:46:51 MalkyMediaHub kernel: nvme nvme0: I/O 320 (I/O Cmd) QID 8 timeout, aborting Apr 17 11:46:51 MalkyMediaHub kernel: nvme nvme0: I/O 321 (I/O Cmd) QID 8 timeout, aborting Apr 17 11:46:51 MalkyMediaHub kernel: nvme nvme0: I/O 322 (I/O Cmd) QID 8 timeout, aborting Apr 17 11:47:21 MalkyMediaHub kernel: nvme nvme0: I/O 0 QID 0 timeout, reset controller Apr 17 11:47:21 MalkyMediaHub kernel: nvme nvme0: I/O 320 QID 8 timeout, reset controller Apr 17 11:50:32 MalkyMediaHub kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 at that point the systems reboots, that should explain the result of my first attempt. Now back to the extended SMART check, I tried researching online how to perform it as ssh to be able to maybe have more logs or info, but apparently if I understand things right, I won't be able to do it as the drive is not mounted, so I do not have the path to use with the smartctl cli. How can I proceed ?
April 17, 20242 yr Author i also have the following in the SMART error log: Num ErrCount SQId CmdId Status PELoc LBA NSID VS 0 380 0 0x0010 0x2002 0x000 0 0 - Which I guess isn't a good sign.
April 17, 20242 yr Community Expert 19 minutes ago, carlos16196 said: I'm trying the extended SMART test, but as soon as I click on it it starts and stops within less than a second Upgrade to v6.12.10 first and try again, IIRC the release you are using doesn't support SMART tests for NVMe devices.
April 17, 20242 yr Author Ok, Unraid upgraded to v6.12.10 and I was able to run the extended SMART check. It reached 20% but got the following in the console log before the system auto rebooted: Apr 17 13:20:18 MalkyMediaHub kernel: nvme nvme0: I/O 17 QID 0 timeout, reset controller Apr 17 13:23:27 MalkyMediaHub kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
April 17, 20242 yr Community Expert Try again, if the same, looks like you will need to replace that device.
April 17, 20242 yr Author It failed again. We can consider the drive is dead. I will now proceed with a "new config" and recover from there. I might need help with the docker ing, but at that point I'll either check existing posts or create a new one to separate both phases. Thank you again very much my friend for all the help.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.