Hello,

 

yesterday evening, all of a sudden, my cache drive did not work anymore.

 

I was working with the server, and then I noticed that dockers were not available anymore ("Docker service failed to start"). I rebooted the system, but the array wouldn't start because my cache drive is missing. I have only one cache drive, a 1TB Samsung Evo NVME.

 

 

My questions:

• Can anyone read from the attached logs what the problem could be? Unfortunately I didn't download the diagnostics before I rebooted, my bad, I know that now...
• What could be next steps? I have no easy means to insert the NVME into another computer. Could buy some adapter though.
• Any other means to determine if the drive is just dead and I need to get a new one?

 

I have backups, not fully up to date, but sufficiently. Nevertheless I will go for a mirror pool from now on...

 

Thanks for any help!

 

(Unraid 6.12.10)

tower-diagnostics-20250114-0928.zip

Try power cycling the server, not just rebooting.

Thanks for your help! I have shut the server down, unplugged power and network cable, waited some minutes, plugged everything in again and started the server.

 

Fans and drives are spinning, but I'm not able to log into the WebGUI anymore 😅

 

I will need to attach a monitor to troubleshoot. Hopefully I manage to do it in the evening. I will report back!

Now I managed to connect a monitor.

 

1. When I remove the NVME and start the server, then I get no output at all. Nothing, no Mainboard manufacture's logo, no bios, nothing at all.

2. When I start it with the NVME connected, I get the Mainboard manufacture's logo and then the attached error (Gigabyte board).

 

The USB stick seems to be ok. I was able to connect it to another computer and read and copy all files on it without problems.

 

Any help is greatly appreciated!

28 minutes ago, cj_obob said:

The USB stick seems to be ok.

The USB stick is either NOT ok, or your MB is not set to boot from it.

 

Suggest loading up a trial USB stick and see if it fires up.

I have loaded a trial stick, but I got the same bios error. Edit: The boot order was set correctly to boot from the USB stick.

 

Next, I confirmed "Load optimized defaults then boot", and now it booted without issues!

 

Then, I tried my original stick, and it also worked, I can log in again via the Web GUI.

 

The NVME was still removed. So next, I inserted it again. But then, again no display output whatsoever, no boot menu, nothing.

 

I put the NVME in the second slot on my Mainboard to see if the first slot is broken. But same result, no video output.

 

This leaves me with two questions:

 

1. This indicates that my NVME is broken, right?

2. How can it be that power cycling caused a change in my bios settings which prevent from booting? Note that I didn't have the issue when I just rebooted without power cycling...

Edited January 17, 20251 yr by cj_obob

if i look at the many threads about losing drives i strongly believe its unraid. So i wouldnt throw it out the window, maybe there comes a update.

 

I also "lost" 2 drives, but i cant believe it, because in the last 10 years i didnt lost so many drives.. 

Edited January 17, 20251 yr by NewDisplayName

Glad it's booting now.  Go back into BIOS and verify all these are settings are disabled which can cause different issues:

Quick Boot, Secure Boot, TPM 2.0, and CSM.  

One of those may help clear up your nvme issue.

And while there (in BIOS) see if it is seeing that nvme.  It probably is but worth verifying while there.

 

On 1/17/2025 at 2:22 PM, NewDisplayName said:

if i look at the many threads about losing drives i strongly believe its unraid.

  

On 1/17/2025 at 2:22 PM, NewDisplayName said:

if i look at the many threads about losing drives i strongly believe its unraid. So i wouldnt throw it out the window, maybe there comes a update.

 

I also "lost" 2 drives, but i cant believe it, because in the last 10 years i didnt lost so many drives.. 

I am having these issues also.

I've been researching it a lot the last couple of days.

Still testing out things, but I think it has to do with a Kernel issue since 6.1.46 (commit 8ee39ec): rtsx_pci from drivers/misc/cardreader breaks NVME power state, preventing system boot
https://lore.kernel.org/regressions/995632624f0e4d26b73fb934a8eeaebc%40realtek.com/

It first looks like its regarding a realtek card reader but if you read further it turns out to cause this nvme issue.
https://lore.kernel.org/regressions/[email protected]/

https://bbs.archlinux.org/viewtopic.php?id=288095

https://lore.kernel.org/all/[email protected]/

And i've been using unraid 6.12.14 where i got the issues, which has kernel 6.1.118, which i think is affected.
But I've tried with Unraid 7 with kernel 6.6.68, but that didnt instantly change the situation so i have further testing to do.


I got the issue using a Samsung 990 Pro 1Tb.
Jan 14 04:37:36 BENEMORTASIA kernel: nvme nvme0: I/O 13 QID 0 timeout, disable controller
Jan 14 04:37:36 BENEMORTASIA kernel: nvme nvme0: Device shutdown incomplete; abort shutdown
Jan 14 04:37:36 BENEMORTASIA kernel: nvme nvme0: Removing after probe failure status: -4

 

EDIT: If my theory is correct, then the issues should have started for people with unraid 6.12.4 and higher.
So i am going to try a couple of things and see if using unraid 6.12.3 fixes it.

EDIT2: In my case it eventually turned out to be a faulty power box. I had to swap the power plug from the power supply to a different slot of my power box, and that fixed the issue. It first showed one drive but after a reboot or 3, everything came back working again and my syslog is cleared of all issues.

Edited January 22, 20251 yr by Benemortasia666
Fixed the issue

5 hours ago, Veah said:

Glad it's booting now.  Go back into BIOS and verify all these are settings are disabled which can cause different issues:

Quick Boot, Secure Boot, TPM 2.0, and CSM.  

One of those may help clear up your nvme issue.

I disabled all these in the Bios. Still no luck, when re-attaching the NVME I cannot even access the Bios.

 

To doublecheck, I inserted the NVME into my desktop PC. Same experience here: The manufacturer's logo is shown, but no booting and I also cannot access the Bios. I have a little display on the mainboard which showed error code 64 (ASUS Crosshair VI Hero), which means that some not recognized device is attached. Without the NVME it boots fine.

 

Therefore, it really seems that the NVME is broken. I will RMA the drive, since it is less than 2 years old and still far from the TBW...

For the NVMe dropping issues, try this, sometimes it helps, it's kernel/ASPM issue with some hardware combos:

 

on the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

Reboot and see if it makes a difference.

 

Thanks for the hint, but I don't even get to the point where Unraid starts the boot process. Meaning, I don't get any output at all, the Mainboard directly thinks "there's a strange device attached, I'm doing nothing", it not even displays the manufacturer's logo from which I usually can enter the Bios. So the problem is kind of on hardware level, I don't get to the point where an Unraid configuration could help...

Here is my (high-level) plan:

 

I have a 500GB Sata SSD lying around which I would add as a new single cache drive. Then I can already start to rebuild everything from backups, restore app data, get docker and vms working again.

 

Once I got the replacement NVME, I just add it to the cache pool. Then it should mirror to this drive.

 

Next I remove the Sata SSD, leaving me with my new NVME as single cache drive.

 

And then, maybe I purchase another NVME (I have two NVME slots on the Mainboard), add it to the cache pool and let it mirror again.

 

Then I should have a new raid1 cache pool up and running.

 

Any objections?

Sounds like a solid plan

19 hours ago, cj_obob said:

Thanks for the hint, but I don't even get to the point where Unraid starts the boot process.

Sorry, misunderstood, that's a different issue, and nothing to do with Unraid.

On 1/19/2025 at 10:14 AM, JorgeB said:

Sorry, misunderstood, that's a different issue, and nothing to do with Unraid.

 

@cj_obob

I had the same happening sometimes, which was caused by a faulty power-box/power-supply