UPDATE: Issue was resolved and then happened again, please check post:

 

--- ORIGINAL POST ---

So, this needs a bit of history first.

I am a long time UNRAID user (10+ years), I am in IT, but the issue is one of those that in the end need a... second server to debug and cannot really afford that (also timewise).
My UNRAID machine is in latest 6.12.15 version, haven't moved to 7.X yet, waiting for the second public release. Or would wait if my server worked. The machine itself, has been upgraded several times and is a rather modern setup.
Current setup is, my own case (back from the time of 3 x 320GB disks), 12 disks of various sizes 4TB-18TB (11 data + 1 parity), 1 M.2 (cache disk, in zfs format if that matters), 2 PCIe SAS2 2008 compatible controllers where (most) disks are on, 10Gbit PCIe NIC, 48GB RAM (2x16 + 2x8 and yes this was running fine for months)., AMD 5600G (wanted a cheap graphics capable CPU as I have no spare PCIe). The whole thing is on UPS of course.
I have several containers installed but run 6-7 only (the others are stopped on a need-to-use basis) and one running VM.
 

So... I had some crashes in the past, but the rate was not something to worry me really. It could be once every few months.
And started to happen more and more. Again I was thinking "I should start looking into what is wrong" but didn't have the time. After all every time it just rebooted and was back ok in a few minutes. I just realised it rebooted by seeing uptime and that parity was checking.
I did RAM test back then (first thing that comes to mind with reboots), without issues.
And last month, it started being nightmare. 2 times a day, 5 times a day... then cannot run more than 20-30 min without rebooting.
And ok rebooting is not as bad as freezing. Sometimes was freezing and rebooting after a while. A few times just freeze and I needed manually to reboot.
I clearly have to resolve this.
 

First thing I tried, is play a bit with UEFI versions, as I did have some issues in the past with this. Tried older and came up to the latest, but thing is I am forced to boot the stick in legacy mode (that was an old problem to be honest, but was not critical... I always had issue with UNRAID in UEFI mode... many times it never reached the blue boot menu). Anyway, with legacy I was booting.
But freezing/rebooting was not resolved. 10 min, 20, 30 tops... boom.
A few times it crashed even before reaching GUI which led me to believe issue is hardware. But I am not so sure now (see below).

 

Other thing I tried. Safe UEFI settings, even XMP off. Same problem, although (because I closed all auto-CPU overclock that current CPUs do) I got 10 degrees celcius down (was not in unsafe AMD levels before anyway).

 

Then I tried using my RAM half and half. Left the 2 x 16 on it. Still crashed after a while. Left the 2 x 8 on it (on the other two slots), again the same. It is not RAM.

Then I changed my boot USB stick. I did the proper process, booted with the new(er) one, boom after a while. Not the stick.

 

Somewhere between attempts (and since I opened the box a few times for cleanups and resitting of some of the cables, I managed to break one SATA port of a disk (at least it is an older 4TB one). But it works  because the pins are in place and I can fit the stuck plastic. The disk shows ok. I will of course replace when system is stable.
Main issue remains.
BTW with all the reboots, one of my newest (18TB) became "disabled" (although I am almost sure it has no issue). Which makes it vital to keep a stable system for long enough to enable and rebuild.

What else to try...

So my next attempt, I found a working similar PSU, a bit better model (than my already good Corsair gold+), same Wattage (which I remind you was ok for years), hopping a faulty PSU is the problem. It wasn't. Rebooted once before reaching GUI and and one 2-3 minutes after GUI booted. I managed to see my machine running just before that (disks where ok except the disabled one).

 

So what remains from hardware are all the disks (duh!), the two controllers, NIC, motherboard and CPU... I am at a loss!

 

Then I thought of trying a Live OS from my debug stick. I would boot a Linux image, but before that I booted WinPE 11. Well it works. I am now 100 minutes online. It can see all the disks (controllers were detected), but of course doesn't "use" them as they are Linux formatted (and I don't want OS to touch them anyway, because it will crap my parity). But hasn't crashed. This confuses me. If it doesn't crash, tomorrow I will boot a live Linux image. Also half of the time I even made it "stress CPU" from CPU-Z. Which is not really stressing it (one core 100% but ~30% of total), but DOES look a bit like normal UNRAID use (although in UNRAID threads where indeed spread on the cores).
 

Could UNRAID be the issue? Yes I know, it worked fine all this time. Some Linux driver?
The remaining hardware is not easy to replace (esp. without being sure what to pinpoint as issue).

My next steps are these (except the live Linux boot):
- Again stress test RAM (although it 99% is not that, as I tried half and half, but this is... for free)
- Replace my 10Gbit NIC, as I happen to have a second same one. Or even remove it and return to my mobo's own 1Gbit NIC.

- Replace my two SATA controllers with a single 16 port one. This does cost, will take time to try (as ebay takes several days from China), but I wanted to replace those two controllers anyway. Of course then I need to see what to do with the broken port. I will need to... move the broken plastic! But ok.

...but CPU and mobo... REALLY tough to replace (and by then I would have replaced everything else except the disks).

That it actually runs (with the stress test) in WinPE11, for 2 hours (in this edit), is really weird and makes me return to thinking it is UNRAID the issue.
The things that don't run in WinPE11, is the 10Gbit NIC (it hasn't detected it), the disks are detected but practically off (as Windows doesn't recognize the partitions). Everything else is on...

Any ideas? I am at a loss. I need some fresh idea.
It is one of the cases that as IT pros we don't want to face.
Because you need to have a spare for every component to test.
Which at home, is not easy.

EDIT: I have tried no plugins safe mode, somewhere between my other experiments. I don't remember the result, but most probably it froze too, because else I would be positive what is wrong. Will redo that to document it though. Right now after 3 hours of NO crash in WinPE11 (!!!) with CPU-Z stress test (which is not much of a stress), I am doing memtest86. No errors yet.

 

 

Edited March 13, 20251 yr by NLS

Enable the syslog server and post that after a crash, together with the diagnostics.

9 minutes ago, JorgeB said:

Enable the syslog server and post that after a crash, together with the diagnostics.

Will do as requested, but thing is I've done it before (when the crashes were more rare) and it wrote nothing out of the ordinary. It just stopped in its tracks.

In other news, other tests I did since last post:
- WinPE never crashed. It did detect my disks (so controller drivers were installed), but my Marvel 10Gbit was not detected. (I mention this why, read below)
- memtest86, was fine for 4 passes, at least 8 hours tonight. I stopped it.
- Today I booted Hiren's BootCD (which is among my many diagnostic tools). I wanted to try one more Windows boot, before trying various Linux live images. IT JUST CRASHED after a few minutes. Next time it rebooted after a minute. Now it froze again!
Only difference I can think (but could be others) with the simple WinPE I tried before... my Marvel 10Gbit was detected and used.
Hmmm...
 

...and then I tried another Windows live build (MediCat VHD)... Windows 11 and this also saw all my devices including the 10Gbit network card that I was leaning on blaming after the above...
AND IT WORKS! I am now at 100 minutes without crash.

I really don't know what to think.
 

Edited February 23, 20251 yr by NLS

16 hours ago, NLS said:

AMD 5600G

Maybe I missed it somewhere in all that text, but have you seen this?

 

1 hour ago, trurl said:

Maybe I missed it somewhere in all that text, but have you seen this?

 

Actually no, I haven't. Will read now. Thing is, my system was running with this CPU for  maybe a couple of years.

 

...follow up.
Did the setting to protect me from Ryzen bug.

No go. Same issue.

Then I removed the 10Gbit NIC that I thought could be the culprit after all.

And I indeed looked like it was. I booted the server and had enough time switch my UNRAID network to the old on-board 1Gbit NIC.

Seemed to work. I even managed to update my VM (which is NethServer 8). Not my plugins though. Nor I did anything to re-enable my 18TB disk that got disabled, until system indeed showed stable for hours. A wise choice.

Because then... it suddenly rebooted.

So we are officially back to "something else".
Only motherboard/CPU are the old ones. SATA controllers too, although I am waiting for another.
But also could be software as other OS seem to trigger it and other not.

And all this on software and hardware I've used for months to years.

 

This is crazy.
 

EDIT: Pending to re-enable syslog and see if it manager to write anything. If I have time to enable it, that is. Also, waiting for controllers.
Maybe I'll get some mid range intel CPU and mobo. Then nothing really remains, it should work.

 

The only thing I cannot really change (and don't see them as problem) are the actual disks.

 

Edited February 25, 20251 yr by NLS

On 2/23/2025 at 5:32 AM, JorgeB said:

Enable the syslog server and post that after a crash, together with the diagnostics.

 

7 hours ago, NLS said:

Because then... it suddenly rebooted.

This is almost always a hardware problem.

On 2/25/2025 at 9:57 AM, JorgeB said:

This is almost always a hardware problem.

 

Well the only things untouched, are:

- The two SATA controllers. I ordered another to swap both with a single stronger one. Will take a few weeks to get. Also this replaces SATA cables, as it uses different port (card side).
- Motheboard.
- CPU.
- Disks, M.2 cache. I DO NOT plan to change or blame those.

I am close to getting a used mobo + CPU of similar specs but different make. From 5600G to i7 12600.

OK new update. Someone up there really hates me.

Got new (old) mobo and CPU someone was selling as a set. A Colorful Z690 Gaming Pro. Had two PCIe x16 and it includes onboard 2.5Gbit LAN. CPU is intel 12600.

 

After fighting the Chinese mobo to update firmware (it needed update using efi shell, because of some major changes), fighting mobo to see all my RAM, fighting mobo for various things AND AGAIN having to choose booting from legacy mode (which makes me believe the issue with UEFI boot IS UNRAID after all, but I don't care right now)... also loosing the initial boot menu (for some reason Colorful doesn't show at all the initial boot steps (leading to believe it doesn't do anything) and only shows UNRAID booting after the scren mode change.

Anyway, I boot, I reach the GUI, login and immediately see these issue, old and new:
 

1- My disk 1 is disabled (this was from previous attempts to fix my freezing... something caused it to "disable" although I am 99.99% sure the disk is fine.

2- It doesn't see the 2.5Gbit network at all. The leds light up, I see my switch detecting 2.5Gbit, but no traffic. Which is weird because it is intel on-board NIC. Unfortunately this mobo doesn't have another PCIe slot capable to host my 10Gbit PCIe card. So I am left with this (unsupported???) on-board NIC, until my single SATA controller arrives that can replaces the two SATA controllers I use now. Which is weeks from now.

3- My disk 7, is identified ok, understands it is xfs, but shows as "unmountable"!!! I tried reconnecting the cables to it, no change.

 

...at least I had GUI. Notice the tense. HAD.

So since network was not identified, I thought of deleting /config/network.cfg yes? (I renamed it not deleted it)
Well after reboot, I never saw web GUI again. UNRAID boots, I can login the graphical interface (locally on server), but the web page NEVER starts.
System did create new network.cfg (as I see from the console), but no web interface at all and I suspect no network either.

 

So,

 

a) How can I help UNRAID detect this on-board NIC? (which is intel with an older but not too old chipset, which should be supported by the kernel)
b) Will this give my web interface back?

c) What to do to diagnose my disk 7, maybe fix xfs errors?

d) How can I re-enable disk 1 without rebuilding it? (remember disk 1 99.99% sure data are ok AND with disk 7 problem I cannot rebuild from parity anyway)

 

Maybe I should disable parity, "accept" somehow disks with the data they have (and probably repair disk 7) and then rebuild parity from data? (instead of data from parity)

 

Help???

 

EDIT: Reverting to old network.cfg, I can have my web interface back (locally), but no network. At least with the GUI I might help you to help me more... Also I notice that MAC address is empty. So I guess the NIC is not detected at all???

 

EDIT 2: Network works now! After forcing it to update some things by disabling bonding and re-enabling it, seems to have fixed this.

So now we remain with c and d above.
Plus now (let's call it "e") my disk 10 is missing. I will shuffle the cables again to see.
DAMN. I will probably need to wait for new controller (and cables that come with it) and start array from scratch.

 

EDIT 3: OK shuffling the cables (pending full replacement) fixed both "e" and "c". So I MIGHT rebuild disk 1 after all IF server proves stable.

But since I trust the data in disk 1, I wonder if there is a way to enable that disk and NOT rebuild it. Is there a way?

Maybe removing parity from config? Can someone with more experience in this point to the steps to verify?

 

 

Edited February 28, 20251 yr by NLS

Attach Diagnostics to your NEXT post in this thread.

Nevermind, I see you started another thread. Usually better to just continue in the same thread so we have continuity. I am locking this thread. Unlocked as requested by user.

 

Anyone wanting to reply go here:

 

OK waking up this topic.

 

I will also update first post.
 

Server just froze again today. Mouse not moving on local screen.

After maybe a week without freeze and after the following have been done:

 

- Replaced USB stick.

- Replaced PSU (both previous and new are Corsair 850 gold-plus modular).

- Replaced motherboard and CPU and switched from AMD to intel as I read about AMD issues (which I didn't have for many months that I used that AMD).
- Flashed latest firmware.
- Did not replace RAM, but extensively memory checked (run memtest all night) AND tested with 2 vs other 2 sticks and slots (instead of the total 4) and the issue still happened (did this on old motherboard/CPU).
- Replaced my dual PCIe SATA controllers with a single newer one.

- Flashed it with latest firmware.

- Replaced all SATA cables as the new controller used different ones anyway.

- Replaced a single disk in the array (as I broke its connector) which I rebuilt successfully (on "new" system).

- Didn't replace my cache, but added a second one, made new pool and moved appdata, system and domains there.

- All other disks remain (my existing array).

- UNRAID is on version 7.0.1 which I updated after my system was practically replaced and stable.

 

I was happy again because system was not crashing and I moderately upgraded the system in the process of "fixing" it.

AND TODAY NOT ONLY IT FROZE (right in the middle of the day and without any active task that I know happening - in other words nobody was actively using the server, all disks probably sleeping, no backups happening or any maintenance - most maintenance things happen at night), but according to my son, it freezes almost immediately after reaching login screen!
(will verify that final part later today, when home)

 

My UNRAID has a VM (which shouldn't affect it, it is my mail server etc.) and although I have several plugins and containers, I only run 5-6 containers (the rest are stopped unless needed). Really everything points out to UNRAID itself right now.

 

As the original subject says, I AM GONNA CRY!

I practically replaced everything.

If someone asks, I am not even sure if I can supply debug information if the system started freezing so quickly.

 

I am in IT decades (and use UNRAID for more than a decade) and really this is very weird.

Help!?!?

 

Edited March 13, 20251 yr by NLS

If it's freezing very quickly it still points more to hardware, unless is during array start, but see if you can get something in the syslog

OK update.

So... it is not freezing (I mean continuously). My son misidentified that.

 

BUT, the (very serious) issue exists and is this: It suddenly lost BOTH the M.2 NVMe storage it has that are both ON-BOARD ... yes on the motherboard!

(that were my two cache pools)

Remember I was using the server whole week as is and this morning I saw it working, then "lost it", when nobody was doing anything on it.

I go to UNRAID GUI properly, but array stays offline because both pools are missing.

 

In BIOS it doesn't see them either. But I am not sure they ever showed up really. But UNRAID identified them properly.

I did various BIOS tests (on a very weird Chinese "Colorful" branded motherboard), no go. I even disabled SATA (that I don't use at all). The two pool disks never reappeared.

I then quickly got out and got a PCIe dual NVMe card to ignore onboard M.2 controllers and use that instead.

NEVER seen the disks either!

The card is PI6C20400BLE based and supposdly transparently seen by Linux.

So I thought that maybe they somehow died both at the same time (some motherboard/PCU issue), so I took them out and put them on an NVMe-USB device I have and inserted to my Windows machine. Both can be seen (partition shows healthy but cannot mount xfs in Windows). So the M.2 are there.

 

I don't know what happened and why (without anybody touching the server and after a full week of fully working "upgraded" system).

 

I also tried a live boot stick (I just tried Windows 11 PE). Again, no go either from on-board M.2 slot or the new PCIe M.2 card. I could see all my data disks and parity disks, my USB UNRAID boot, but not the two M.2 storage.

I really don't know what is happening.

 

One last test was to remove my PCIe SATA controller (yes ALL disks), just to see if there is some weird lane sharing issue.

Again didn't see my two M.2 NVMe... So this specific system decided to NOT show those two NVMe in any way. Or not?

So I thought of putting one of those NVMe again on my NVMe-USB, I put it on Windows again (my main machine), again it showed up as a 232GB healthy (unknown as it is xfs) partition. So I booted UNRAID on my server and attached this USB device on it.

It shows up as a 250GB NVMe on unassigned devices (but shows the controller's ID instead of the disk). It sees there is an xfs partition in it and the proper used/free information!

But if I clicked to see the contents of the device, nothing happened... I was waiting forever.

Then I refreshed the page. Invalid path.

Seems it doesn't show in unassigned devices any more (!)... I see the USB device's led blinking.

I remove and re-attach, led is now stable on, but since it didn't dismount cleanly before, unassigned devices shows "reboot".

 

My final action (except maybe in the future swapping the motherboard AGAIN), will be to replace the two pools with a single slow extra HDD. I know my PCIe SATA controller works at least. Not fantastic but will do (if it works, because I believe nothing any more).

 

Even if the NVMe are done for some reason, I don't trust this motherboard any more, becuae if they are dead IT killed them.

 

NEW QUESTION now is: If I do that swap above and since my pool contents (at least one of the two) are VITAL, as it hosts appdata, domains, system. How can I copy those two a new pool AND copy them there? Will UNRAID just resume working after that copy?

 

Also, in case xfs trashed in any way, what are my recovery solutions?

 

Edited March 13, 20251 yr by NLS

13 hours ago, NLS said:

If I do that swap above and since my pool contents (at least one of the two) are VITAL, as it hosts appdata, domains, system. How can I copy those two a new pool AND copy them there? Will UNRAID just resume working after that copy?

If the devices are detected with a different board, nothing else you shoudl need to do.

 

13 hours ago, NLS said:

Also, in case xfs trashed in any way, what are my recovery solutions?

If the filesystem is corrupt, you can check filesystem, I would avoid doing that with a USB bridge at this point.

 

 

2 hours ago, JorgeB said:

If the devices are detected with a different board, nothing else you shoudl need to do.

If the filesystem is corrupt, you can check filesystem, I would avoid doing that with a USB bridge at this point.

For the first. No I am talking about extracting the data and putting on new cache disk (maybe not even M.2). Is there a process to do that?

 

For the second, here is my practical problem. My motherboard stopped seeing the two NVMe altogether, even with the use of a PCIe-NVMe card (!!!). On other computers, I could see the NVMe in UEFI, but I had this issue: Linux systems (live), couldn't really mount them. They could see there is an xfs partition, but failed to mount (missing superblock or something).

 

ON THE OTHER HAND, weirdly enough, Windows machine with ability to see Linux fs (including xfs), was able to see the contents and I could even extract them. But being extracted on Windows etc. doesn't warranty the files are valid (esp. my Nethserver VM image is the most important).

 

Also, I tried to get the two NVMe AND UNRAID stick and boot it on other system.

IT DID see the two NVMe but the system cannot start even in maintenance mode as "too many disks are missing" (the whole array is on another server).
I would really want to be able to start maintenance mode even like that! (maybe should be allowed?)

I tried to run xfs_repair -v /dev/nvme1n1p1 (or whatever was the device) from console, but it didn't fix anything because I got sporadic "found candidate secondary superblock", "unable to verify superblock, continuing"...

What should I do? Why Windows could do what this couldn't?

 

TL:DR: Any more sophisticated Linux tools to use to extract data from the NVMe (where xfs_repair fails but Windows "looks like" it can?) and an UNRAID method to use those data on a new pool?

 

Edited March 14, 20251 yr by NLS

1 hour ago, NLS said:

Is there a process to do that?

If the drive is detected with another board, you can just copy the data after booting any Linux OS, including Unraid.

 

1 hour ago, NLS said:

couldn't really mount them.

Post the error you see after a mount attempt.

17 hours ago, NLS said:

since my pool contents (at least one of the two) are VITAL, as it hosts appdata, domains, system

Just in case you haven't already taken care of this, disable Docker and VM Manager in Settings until you get these shares working again on whatever disk they end up on.

@trurl You are right.

I am not even sure I can enable them.

I have the server shut the last couple of days until I decide what to do.

 

@JorgeB Will do, will try in the weekend again.

To be honest, being mostly a Windows person (although I use UNRAID from the... ancient times), I was surprised that xfs proved to be so NON resilient AND the lack of recovery possibilities. The only tool that managed to see the contents of my partitions (of both NVMe), was Windows based. The issue is I don't trust much what they recovered, maybe they are ok, maybe not... AND all metadata information (security attributes etc.) is lost which in the case of containers could be "fatal".

 

And this happened suddenly, while server was barely doing anything. I wonder what on my motherboard could have done that.

Needless to say I don't trust the bought-used-10-days-ago motherboard any more... after all doesn't see my NVMe at all (even in UEFI/BIOS - except when put in USB-NVMe adapter), while other motherboards can at least acknowledge their existence, both onboard and if put on the PCIe-NVMe card I got.

 

Really this last month UNRAID (well MAYBE not UNRAID itself, but the server system as a whole) has done to me what it hasn't done to me in all those years.

 

I am again forced to split to another thread, so that cleanly someone can help me with the end problem (after NOT solving the issues at their core, I need to now go to the alternatives I have). I will edit this topic with the thread link, but please don't lock this one, to keep it as a closure when this closure FINALLY happens.
(I might need to sell the rights to a documentary)

 

EDIT:

Here...

https://forums.unraid.net/topic/188176-next-day-recovery-of-cache-contents/

 

Edited March 15, 20251 yr by NLS

If you are looking at alternatives to unRAID.

 

 

I would recommend proxmox, open media vault, truenas...

 

(Debian) Proxmox. A virtual machine hyper v for other Linux VM and LXC for services. Look into proxmox helper scripts for community pre setup applications.

 

Truenas scale/core

A free ZFS supported nas scale recommend.

(ZFS only hard drive setups)

Use helm docker, limited but has VM support.

 

 

Debian/rocky Linux 45 drive with Houston web UI. Nas HD first plug-in and service for kvm VM and Dockers exist but a pain to setup properly

 

*Use portainer for Dockers. Plugins are a bit weird.

 

(Debian linix) Open media vault. Docker can be a pain to setup in webui perfect for nas

 

(Ubuntu/Debian) Casa OS. Limited webui for Dockers and bas built for zima SBC boards...

This sounds like a epic bios issues. There was a lot updated and changed on drovers between unraid release and the kernel 

 

You may have had a motherboard that had 3rd party Linux driver that were dropped or not fully tested in kernel revision.(1of the many reasons why I hate kernel 6 as shit just worked under kernal 5) Over 10,000 lines were removed and kernel 6 still hasn't fixed all the fallout from v5 into v6.... 

 

Would love to see a diag to dig in the syslog...

 

I would highly recommend upgrading into v7 and the latter LTS of kernel 6 as there have been other reports similar. Older version of 6 including last v6 didn't write the config correctly.

 

Under main > flash > syslinux 

* Potentially option like noapic force Linux acpi 

 

As this will assist Linux kernel loading and PCI irq renderings....

 

I may have some other grub options that I would have you try to help capture fix and inform unRAID Linux....

 

I would also recommend turning off secure boot.

 

Can you share a syslog as you should have a syslog previous and current before the restarts. Something. We can review to assist you in figuring out what crashed where...

4 hours ago, NLS said:

I am again forced to split to another thread, so that cleanly someone can help me with the end problem (after NOT solving the issues at their core, I need to now go to the alternatives I have). I will edit this topic with the thread link, but please don't lock this one, to keep it as a closure when this closure FINALLY happens.

So are we expected to go back and forth between all your open threads?

 

Explain to me why I shouldn't just merge them all.