March 15, 20251 yr If anybody has followed the original thread here... ...you can see I am in nothing short of an Odyssey to fix my server. Not being able to fix everything exactly as it was, I am now in the crossroads of having to see how to realistically proceed to a working server. Given are the following: - My two pools, made by two M.2 NVMe, are not accessible any more by my server (motherboard issue, it cannot even see their existence) and seemingly were corrupted too (both at the same time!) and xfs_repair did NOT help, but taken to other computers I CAN see the NVMe, CAN see they are formated xfs, but Linux has issue to mount them (and as I said xfs_repair did not help, run from inside UNRAID on another machine, with array stopped). - ...But in Windows, using DiskInternals Linux Reader, I WAS able to read the partitions and extract the data. I cannot yet verify if the data are ok (some text based files, init etc. seem to be readable), I am especially worried about one of my VM (because it is VITAL) and of course let's not forget, that every security info was lost. Keep this in mind. - So I have now in some Windows disk (NTFS), my appdata/domains and system (without the docker folder which is useless after every move anyway). - One of the M.2 NVMe (that I don't care about the content) I formatted (on another machine), as I don't care about it's contents (where temp files). I want to reuse it in my server. I don't trust to put it on motherboard (where the incident happened), but I am trying to find a PCIe card that I can use (one I tried ALSO did not work - that was working on another machine - on a slot that I know it works). Probably some issue of the Chinese motherboard and PCIe speeds. In any case I am waiting for a couple other PCIe cards to put the NVMe on and HOPING one of them will work. (here comes the important part where I need help) - If any of these work, the disk will reappear with its original ID in UNRAID but will be empty and it will miss the OTHER NVMe (as I have no way to put it back) that (the other disk) used to HAVE appdata/domains/system. - If the above won't work in my motherboard (replaced 10 days ago, used, don't plan to replace again yet), I will just put an extra SATA SSD and use that as my pool, until I find a better solution (which probably is to replace the Chinese motherboard with one by the brands I usually trust). So problem is this... In first case, where a card works and one NVMe is back online with its old ID, I need to somehow put from Windows the data back to the "old" (xfs) pool that didn't have it originally. a1) How do I make that transfer? (which is maybe 100GB) Probably use some USB disk and connect it to UNRAID? a2) How do I make UNRAID eliminate the other pool that will not reappear (as the disk is already missing for ever)? a3) How do I make UNRAID use the data I supposedly transferred to the remaining pool? (#1 above) a4) How can I make sure the attributes (read/write/execute, owner, group) of the files/folders are as they should? In case I have to use the SSD instead (I cannot make NVMe work), then both old pools will remain with missing disks and I need to make a new one OR put the SSD in place of one of those pools, probably. b2) How do I make UNRAID eliminate the other pool (or both pools) as the disks will not reappear? b3) Like a3 above. b4) Like a4 above. Any help appreciated. BONUS question: How is it possible that xfs_repair doesn't fix it, Linux doesn't mount it, but some Windows tool managed to extract data and didn't even whine. This is where Linux's recovery options end and xfs resiliency ends? Not very encouraging... Edited March 15, 20251 yr by NLS
March 17, 20251 yr Author On 3/16/2025 at 5:23 AM, bmartino1 said: i think this will get merrged... we need a daig and syslog... I don;'t think to answer my questions diag and syslog are needed. I can tell you already what they say: They don't even see the devices. It is not that they see them and not mount them or anything. My questions are quite different above and I am not looking into finding what is wrong (most probably something with the motherboard itself), but bypassing the issue. So again: Quote In first case, where a card works and one NVMe is back online with its old ID, I need to somehow put from Windows the data back to the "old" (xfs) pool that didn't have it originally. a1) How do I make that transfer? (which is maybe 100GB) Probably use some USB disk and connect it to UNRAID? a2) How do I make UNRAID eliminate the other pool that will not reappear (as the disk is already missing for ever)? a3) How do I make UNRAID use the data I supposedly transferred to the remaining pool? (#1 above) a4) How can I make sure the attributes (read/write/execute, owner, group) of the files/folders are as they should? In case I have to use the SSD instead (I cannot make NVMe work), then both old pools will remain with missing disks and I need to make a new one OR put the SSD in place of one of those pools, probably. b2) How do I make UNRAID eliminate the other pool (or both pools) as the disks will not reappear? b3) Like a3 above. b4) Like a4 above. Can someone help with this?
March 17, 20251 yr Community Expert You most likley need to re read the motherboards manual... Get into Bios and Fix many settings... Boards, even some low end thread ripper have bios options that may not be set correctly. AMD bifurcation... some options may disable the use of other options that may be what your fighting... Example the need and use of pcie bandwith... SO Quote In first case, where a card works and one NVMe is back online with its old ID, I need to somehow put from Windows the data back to the "old" (xfs) pool that didn't have it originally. when plugged into a pcie slot you may need to enter bios and fix bifurcation exmaple (x4/x4/x4/x4 for a x16 slot for a nvme addin card...)... in a pcie x16 slot telling it that the device is wired for signals this way... and look into the pcie addon cards generation(Even NVME has settings for this!). Enabling other setting may remove sata ports, u3 connection, nvem ports over pcie... since unriad is LINUX! you will need to use a LINUX OS! Windows is garbage for recovery... use windows for window, use linux for linux... Unraid is linux! So for you to use windows and to see and import and use the existing xfs file formatted disk... there are some non-free software you could use and install to read the disks... *IF they are in a muti pool setup ALL THE DISK! need to be read to mound and access the files... this is dependent on how you setup your D1, parity, and pools... All of which I would need to help with structure and grabbing the correct disk verifying it is connected and how its conected and how motherboard/bios has it connected BEFORE any attempt at data recovery can be done! Please post a diagnostic file.. Or seek help on a windows IT forum... as its clear these are not unraid related then... Yes, a Diag will Help amensly, your not getting answer or help as we have to assume to much based on your muti-posted on the forumed post referencing data that meaning less. I require a diagnosis file at a min, this give me quite a bit of info that is needed... As We don't know what you meant when xyz card is working and nvem is xyz... Unless ints in this thread I will not go searching for it... It is pointless unless you have and use a single post with all the data... GO back and read the forum rules! https://unraid.net/policies As, so far all I've seen is garbage and no one will be able to assist you remotely that way... ############### So I would recommend going after a Linux Live boot OS of a Debain/ubuntu system as you will need a Operating system that will be able to see all the disks and can mount the windows File system and the unriad XFS Disk to do a copy in the live boot between them. My Go too would be ubuntu Mate. https://ubuntu-mate.org/ You can live boot a windows like gui to open the linux formated drives on that os and the windows drive on that os to file copy the data over... ########## But givin that your not willing to share or refuse too... and what I'm seeing as I'm trying to catch up to alot of your posts with limited data... Its clear to me you lack some of the advance techniques and knowledge to do some of the stuff your trying to do... So All I can do is recomend other stuff. As they may be easier soultion to assit... This would mean using different os and booting into a liunx live os... However, I will note that 3rd part software exist for windows and you could atempt to leverage Hyperv and WSL to mount linux formated disks... https://learn.microsoft.com/en-us/windows/wsl/wsl2-mount-disk https://superuser.com/questions/1761826/how-to-make-windows-read-ext4-zfs-etc-linux-drives Review: https://recoverit.wondershare.com/linux-recovery/xfs-windows.html what disk are connected an how. what bios settings have you checked to confirm software control over the motherboard and how the ports are being used for your devices. Whats your Motherbards make and model... I will galdy go though and point out where in the manul... As I have done simalr with simlar hardware... What is currently contend to what... nvme slot 1 ? 2? 3? Pcie Device ? slot bottom top? x16, x8 ? biforcated? gen? how many sata ports p1 is ssd? hdd? To much is not known to assit you. More data needed. Otherwise while it started in unradi the suport you need is not here.. https://www.techsupportforum.com/
March 17, 20251 yr Community Expert If using unraid AND ONLY UNRIAD! in unriad there is a option to reset the disk configuration... webui > settings > new config otherwise review: we can also use the unraid web termianl and do this via linux comands and plugins. Unasigned disk will help you here...
March 17, 20251 yr Author Thanks for the (huge) reply. Let see... - I know about bifurcation. Not the case here (as it will become obvious below) but also this stpd motherboard does not have such settings. - I COULD NOT use UNRAID for recovery for two reasons: (remember these two disks are two separate cache pools, NOT part of an array and NOT a single pool - I explain this above already, but someone can get lost with all the info) a) As I said, on my own server, the disks cannot even be seen by BIOS/UEFI and not by UNRAID either (it's like they are not there). b) Taking my UNRAID USB stick and putting it on another machine (and the "problem" disks), I can see the disks (but I don't have my array along, so I cannot even start in maintenance mode, I stay with array stopped). With stopped array but seeing the problem disks, UNRAID once detected a disk as xfs and even showed a correct free/used bar, but failed to show contents of the disk. In another attempt it didn't even show free/used, although it again knew it was xfs format. The disks are not readable. xfs_repair does NOT fix them. I tried from within UNRAID (on another machine). On the other hand, Windows already allowed me to read the files, maybe too easily. Sorry. Point me to a tool that can manage to repair the disks on Linux (most probably I have already tried it). My most important cache disk (with the VM etc.) is stored, until I can be able to recover it even better than what I got from Windows. So, yes, will to do something a certain way is nice, yet, it didn't work. - Again, I have already read the disks and got the files. But of course (since it is Windows) I lost all attributes (user information etc.). - You require a diag file yet I cannot provide you one, I mean, read a and b above on why. - I know the forum rules I believe. You can see my profile and see since when I am registered here. Yet, this is a very weird case where I cannot reach the point of being able to make a diagnostic. - I did boot (on other machine that can see the disks) various Linux live distros. They cannot mount the disks. I could probably supply you with the failure info, which is the only "diag" I could possibly supply. No Linux live distro with recovery tool allowed me to recover the data. Again I told you I did that, before you posted that I should try it. - I tried all on-board slots (able to do NVMe) and all PCIe slots that could fit the cards. The card or bifurcation is not the problem, see below on why. As I said there are no bifurcation settings and I tried to set the PCIe to 1-2-3-4-5 gen. No SATA ports are used on motherboard. I have PCIe SATA card. 12 HDD disks. This is unrelated. They were always there when the system was working. No SSD. - So, why BIOS settings are unrelated? Because I tried another NVMe that was NOT on my server and it worked! Server sees this NVMe. On the PCIe cards. They work. So the cards or BIOS settings are not the issue. The specific 2 "bad" NVMe cannot be seen by server, BUT CAN BE SEEN BY ANY OTHER MACHINE I TRIED! Yet the server CAN see other NVMe. Can you explain this? I did BIOS reset, I did format the disks, deleted partitions etc. No go. Thanks for the help. If you read me, you will see I have already done what you think I should try.
March 17, 20251 yr Community Expert this tells me that a bios setting has disabled the ports the disk are using. Otherwise, the disk are dead... example sata port on the board will be disabled if x8 slot, nvme and u3 is enabled. if not using u3 sas. you need to tell bios to disable u3 to keep the sata ports 5 and 6 active... This is a bifurcation due to how the baord chipset layout the pcie and its use to use a nvem drive vs sata ports and what bandwidth can be used. Bios will tell you. If the disk doesn't show up in bios start the issue not unraid, and not windows... again, need specs and layout... You may need to find your block diagram to make sure your disks configuration can be used... example of unclear garbage... Quote (remember these two disks are two separate cache pools, NOT part of an array and NOT a single pool - I explain this above already, but someone can get lost with all the info) 2 separate disks, that fine. What unraid key version? There are disk limits too even with the unsigned plugin... but the disk would show up... SO not array not pool... I have to assume then is its not clear! disk new empty, broken, plugin atached .. again need data not provided... more of issues of contradiction... if is a cache pool, then its in the array.... otherwise there need to be clarification of what happened... was the disk removed? using plugin UD to mount it? new disk to add or use? but regardless, bios is not seeing the disk so its not a unraid issue... Edited March 17, 20251 yr by bmartino1
March 17, 20251 yr Author 54 minutes ago, bmartino1 said: this tells me that a bios setting has disabled the ports the disk are using. Otherwise, the disk are dead... example sata port on the board will be disabled if x8 slot, nvme and u3 is enabled. if not using u3 sas. you need to tell bios to disable u3 to keep the sata ports 5 and 6 active... This is a bifurcation due to how the baord chipset layout the pcie and its use to use a nvem drive vs sata ports and what bandwidth can be used. Bios will tell you. If the disk doesn't show up in bios start the issue not unraid, and not windows... again, need specs and layout... You may need to find your block diagram to make sure your disks configuration can be used... example of unclear garbage... 2 separate disks, that fine. What unraid key version? There are disk limits too even with the unsigned plugin... but the disk would show up... SO not array not pool... I have to assume then is its not clear! disk new empty, broken, plugin atached .. again need data not provided... more of issues of contradiction... if is a cache pool, then its in the array.... otherwise there need to be clarification of what happened... was the disk removed? using plugin UD to mount it? new disk to add or use? but regardless, bios is not seeing the disk so its not a unraid issue... First, (and I find the need to clarify, so that we don't start from the alphabet) let me tell you I am into computing since I was 9 and I am now 51. Since my early 20ies I am in IT professionally. I've seen motherboards in the hundreds. About your deduction: 1) No this doesn't tell you that a BIOS setting disabled the ports. Because if I put different NVMe on those same PCIe cards, it does see the NVMe. Only the two "problematic" ones don't show. 2) But no, this doesn't tell you that the disks are dead either. Because those same disks SHOW UP on two different other computers on the same PCIe cards (or even on board). In fact one of the two (where I didn't have data I cared) I now use it on another (Windows) computer. Formatted (not quick format) successfully and copied data to it. No diagnostic shows an issue. 3) I already said that the issue is not UNRAID. The only issue with UNRAID (because you said you want to only use UNRAID), is that if I put UNRAID on a computer (not my server) that can see the two problematic disks, the tools provided by Linux/UNRAID (mount commands and xfs_repair) did not fix the partition, neither let me access the contents. This is where the problem with UNRAID ends. 4) I am an UNRAID Pro/Lifetime user for several years. No disk limits. 5) Those disks WERE my two separate cache pools. I am saying they are not in the array in the sense that they are not part of the parity protected pool. They are cache pools. Anyway, to NOT overcomplicate things. I am not trying any more to see those two disks WITHIN UNRAID WITH THIS MOTHERBOARD. It is a lost cause. I have already put a different NVMe (on a PCIe card, I don't trust the motherboard ports anymore) that I plan to temporarily use as a NEW cache pool. Also, I have (maybe) recovered the data, unfortunately on a Windows computer, since no Linux tool helped me. --- CURRENT SITUATION, for anybody TL:DR --- The questions that are current and I would love someone to answer are those and those only (and are all UNRAID related): 1) How can I make UNRAID forget the two missing pools? Should I make "new configuration"? (or forget one and put different disk on the other) I have done new configuration a number of times (for example to re-arrange disks) so I know how it works. But is the process to follow? EDIT: Already done. New config, retain assignments, but not cache. Reconfigured with new cache disk. Docker and VM are disabled for now, until #2 below is done. 2) How can I use appdata/domains/system I have (hopefully) recovered from a Windows system, copied back to UNRAID and get the proper attributes/user/group? I will probably copy the recovered data over the network. But the problem is will they be usable? Is there a default attrib and chown I should use (or some tool to do it for me)? 3) (bonus) If someone can suggest a specific live rescue distro and specific tool within it, that can help me do the recovery work from within Linux (so that attributes possibly stay), I would love it. Nothing I tried in Linux managed to do it, only a tool in Windows. My most important NVMe (with issue) is still available to recover (Windows recovery only "read" from it and gave me my data, didn't fix it). Thanks, one more time. Edited March 17, 20251 yr by NLS
March 18, 20251 yr Community Expert 13 hours ago, NLS said: If someone can suggest a specific live rescue distro and specific tool within it, that can help me do the recovery work from within Linux If Unraid doesn't work to repair the filesystem, it should be the same with any other Linux distro, sorry but this thread is rather long, can you post the exact command you used and the output you see when trying to repair that filesystem using an Unraid trial key?
March 18, 20251 yr Author 2 hours ago, JorgeB said: If Unraid doesn't work to repair the filesystem, it should be the same with any other Linux distro, sorry but this thread is rather long, can you post the exact command you used and the output you see when trying to repair that filesystem using an Unraid trial key? I will try to re-do it but check if the following already covers you: The exact command was xfs_repair -v and the path of the device on that specific computer (/dev/nv-something). The output was all dots (scrolling pages and pages), with sporadic "found candidate superblock" followed by "unable to verify superblock". Which I think was also the final ending of the output. Here is a casual photo I got when I tried originally, a few days ago. I take it that we "fight" to not use the Windows already recovered data because the lost attributes make real use of the data very difficult? (on other news - because this really deserves the "I am going to cry" title of the original thread- my replaced motherboard that did all this, now that I put another M.2 ON a PCIe instead of motherboard, decided to kill the on-board 2.5Gbit NIC... it is like non existent, not detected by UNRAID any more and its port LEDs never light up... so all issues aside, I *AM* going to replace the replaced motherboard... and lose the money as it was bought used and worked like 10 days, so, aside from not very politely letting the seller know, I cannot really request a refund or something... great, ain't it?)
March 18, 20251 yr Community Expert 2 hours ago, NLS said: (/dev/nv-something) By that output looks like you didn't specify the partition, e.g., if the device is nvme0n1, the command needs to be xfs_repair -v /dev/nvme0n1p1 Also, if the command was correct, and it still didn't work, you can be pretty sure it will be the same in any other Linux distro.
March 18, 20251 yr Community Expert Have you checked for nvme firmware update? When I was building a Photoshop Windows PC for my wife a few years ago, I had to update the firmware before I could reliably get both nvme going at the same time.
March 18, 20251 yr Author 15 minutes ago, JorgeB said: By that output looks like you didn't specify the partition, e.g., if the device is nvme0n1, the command needs to be xfs_repair -v /dev/nvme0n1p1 Also, if the command was correct, and it still didn't work, you can be pretty sure it will be the same in any other Linux distro. Yes that was correct. I did specify properly what to scan. I find it very weird that almighty Linux (the light irony is not targeted towards anybody here or even UNRAID - I am not leaving UNRAID, I have promoted it to so many people), cannot recover anything on a supposedly very resilient fs like xfs, while a simple tool I bought on-line in Windows DID (most probably) recover the files and didn't even whine it had an issue or difficulty. Anyway, recovery aside (hoping I recovered most of what I care properly, I could just test some text based files), my remaining questions remain, if you have any idea to help on that (read above your post after the TL:DR)... I am looking on how to "re-import" those data properly on a new cache from where I keep recovered in NTFS based storage (not the physical process of course, I will find a way... I mean maybe resetting attributes and user/group properly, if the default copy to cache doesn't do that correctly). 3 minutes ago, trurl said: Have you checked for nvme firmware update? When I was building a Photoshop Windows PC for my wife a few years ago, I had to update the firmware before I could reliably get both nvme going at the same time. I did. It was the first think I did when I got the used motherboard (the previous owner never flashed it). It was not even easy on this Chinese brand... I had to go to EFI shell. Remember the motherboard worked fine (with all hardware) for about 10 days. Then hell let loose. EDIT: Oh you meant the firmware of the two storage? No I didn't. Remember, they work on other motherboard (and even worked on THIS one for 10 days). Also they are two different models and sizes. It is not that I bumped on an NVMe firmware bug. Edited March 18, 20251 yr by NLS
March 18, 20251 yr Community Expert 3 minutes ago, trurl said: Have you checked for nvme firmware update? 2 minutes ago, NLS said: got the used motherboard (the previous owner never flashed it) I wasn't talking about BIOS update. I was specifically talking about the individual nvme drives firmware.
March 18, 20251 yr Author Just now, trurl said: I wasn't talking about BIOS update. I was specifically talking about the individual nvme drives firmware. Yes I edited my reply while you were typing this. See above.
March 18, 20251 yr Community Expert Mine were the same brand, but different sizes. And they each worked by themselves, but not together until I updated the firmware. 6 minutes ago, NLS said: firmware of the two storage? No I didn't
March 18, 20251 yr Author 1 minute ago, trurl said: Mine were the same brand, but different sizes. And they each worked by themselves, but not together until I updated the firmware. I will try to see if there is updated firmware for them. They don't work on that motherboard even alone, not only together. They are two different Samsung models of M.2. But my priority now is what I say above. How to re-import things from Windows recovered files to a "new" working cache. My appdata/domains and system (except docker folder... I deleted that, I know it doesn't move gracefully around but can be rebuilt).
March 18, 20251 yr Community Expert 2 minutes ago, NLS said: two different Samsung models of M.2 Mine were both 980 Pro, 1TB and 256GB. Possibly relevant thread:
March 18, 20251 yr Author Thanks. Any idea about the current problem of re-importing my important UNRAID folders properly? (for them to be usable by UNRAID) I will probably flash the NVMe, but I am 99% sure the issue is the motherboard.
March 18, 20251 yr Community Expert 28 minutes ago, NLS said: Yes that was correct. I did specify properly what to scan. If that is correct, and the device was connected to an m.2 slot, not using for example a USB enclosure, it's basically certain that you will have the same results with any other Linux distro, with the exception of possibly older or newer xfsprogs making a difference.
March 18, 20251 yr Community Expert For everything except appdata, New Permissions will make owner and permissions as they need to be. I think some containers might even fix their own appdata permissions when they start. Might have to take each container one at a time, maybe compare them to the permissions set on a new install of the container.
March 18, 20251 yr Author 2 minutes ago, JorgeB said: If that is correct, and the device was connected to an m.2 slot, not using for example a USB enclosure, it's basically certain that you will have the same results with any other Linux distro, with the exception of possibly older or newer xfsprogs making a difference. Yes, on board a PCIe card or directly on-board (I don't remember), not USB. Will do it again though (my most "serious" of the two NVMe is just sitting waiting for the best treatment of recovery - the Windows based tool just read it, didn't write anything on it). I did try an older (apparently) xfs_repair using a live distro, and it didn't even attempt to fix the disk, claiming I need a newer version because of I don't remember what. The version that produced the result above (i.e. attempted to fix but failed), was the one within UNRAID 7.0.1. Just now, trurl said: For everything except appdata, New Permissions will make owner and permissions as they need to be. I think some containers might even fix their own appdata permissions when they start. Might have to take each container one at a time, maybe compare them to the permissions set on a new install of the container. Understood. Right now, the one thing that is most important to me is to see if my VM (NethServer8 with my mail of decades) survived (along with libvirt image I guess). For appdata, most of them I can rebuild and the couple that are important I can probably install from stock and manually replace content of configuration files (or check security of new setup and apply it)... one that comes to mind is my technitium based DNS server container. From UNRAID if I could "request" something (which is not even for me), is to make sure in the future, people that setup a new system are aware of the importance of appdata/domains/system and either back it up or set up a protection scheme for them (and UNRAID requires it, or sets it up or anyway pushes the user to implement it).
March 21, 20251 yr Author So, I started closing the threads I opened, which threads handled not a single issue, but a series of issues that happened during my Odyssey (so it makes sense to be broken). So for this thread, a weird thing. I now have managed to boot UNRAID on a new motherboard (details on other thread), so I thought I should install my two NVMe. One large NVMe was formatted, used purely as cache anyway (so no static data I care about). One small NVMe very important to me, as it holds appdata (which I had backup though), domains (of which a single VM mattered much, which I DID NOT have backup of the whole VM, although I do have backup of parts of the VM taken from inside the VM) and system. If you read this thread you will see that I had various issues to recover the data etc. Well in unassigned devices on my "new" server, I did see both NVMe. The big one was expected (as it was re-formatted), the small one... well it showed up properly as XFS, free/used space... so I clicked to see the contents. AND I SAW THE CONTENTS! Not really sure what to think, I made a new configuration (preserved only array, not cache pools), and so I assigned cache to my big NVMe and the second pool to my small NVMe... And started the array (with docker and VM stopped). Array started fine! I checked my shares and it still remembered that appdata/domains/system should be on my second cache pool. So I started docker. It started! I saw my containers and the few auto-start containers DID start! Then I started KVM. It started and started my VM! Apparently Windows was right to believe the xfs disks where ok. They just worked. So after huge complications I am back to a working server! Thanks everybody that tried to help.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.