DatacenterGuy

Before going on: The problem with the nvme drops still exists. It is really annoying, I tried several things in the last 24 hours: - Making sure SATA-Power was no problem, so I changed the cables first - Added a external psu just for the drives (as 7 of them could draw up to 175 Watts, that could be a problem for a single SATA-Cable) - Removed the pool and physically disconnected NVME except for 2 drives - Build another pool with the 2 drives And after creating the pool, starting the service and clicking for a format of both, a total system crash happened, but I had the chance to copy this dmesg: [ 2558.423648] nvme0n1: I/O Cmd(0x1) @ LBA 0, 40 blocks, I/O Error (sct 0x0 / sc 0x4) DNR [ 2558.423670] I/O error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x0 phys_seg 5 prio class 2 [ 2558.423680] buffer_io_error: 15 callbacks suppressed [ 2558.423683] Buffer I/O error on dev nvme0n1, logical block 0, lost async page write [ 2558.423704] Buffer I/O error on dev nvme0n1, logical block 1, lost async page write [ 2558.423710] Buffer I/O error on dev nvme0n1, logical block 2, lost async page write [ 2558.423715] Buffer I/O error on dev nvme0n1, logical block 3, lost async page write [ 2558.423720] Buffer I/O error on dev nvme0n1, logical block 4, lost async page write [ 2558.475674] nvme0n1: I/O Cmd(0x1) @ LBA 15002931848, 40 blocks, I/O Error (sct 0x0 / sc 0x4) DNR [ 2558.475677] I/O error, dev nvme0n1, sector 15002931848 op 0x1:(WRITE) flags 0x0 phys_seg 5 prio class 2 [ 2558.475686] Buffer I/O error on dev nvme0n1, logical block 1875366481, lost async page write [ 2558.475706] Buffer I/O error on dev nvme0n1, logical block 1875366482, lost async page write [ 2558.475712] Buffer I/O error on dev nvme0n1, logical block 1875366483, lost async page write [ 2558.475717] Buffer I/O error on dev nvme0n1, logical block 1875366484, lost async page write [ 2558.475722] Buffer I/O error on dev nvme0n1, logical block 1875366485, lost async page write [ 2622.544430] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10 [ 2622.544478] nvme nvme1: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10 [ 2682.561487] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 2682.561492] rcu: 8-...0: (5 ticks this GP) idle=3114/1/0x4000000000000000 softirq=62498/62500 fqs=11970 [ 2682.561495] rcu: (detected by 6, t=60004 jiffies, g=216513, q=49920 ncpus=16) [ 2682.561497] Sending NMI from CPU 6 to CPUs 8: [ 2682.561500] NMI backtrace for cpu 8 [ 2682.561503] CPU: 8 UID: 0 PID: 47 Comm: kworker/8:0H Tainted: P O 6.12.54-Unraid #1 [ 2682.561505] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE [ 2682.561506] Hardware name: ASRock B760 Pro RS/B760 Pro RS, BIOS 12.03 05/07/2025 [ 2682.561507] Workqueue: kblockd blk_mq_timeout_work [ 2682.561511] RIP: 0010:qi_submit_sync+0x270/0x5c0 [ 2682.561513] Code: 8b 7c 24 08 48 83 f8 01 49 8b 46 10 45 19 e4 41 83 c4 05 83 3c 38 03 75 0a bd f5 ff ff ff e9 c0 02 00 00 49 8b 45 00 8b 68 34 <f7> c5 70 00 00 00 0f 84 94 02 00 00 49 8b 45 00 8b 80 80 00 00 00 [ 2682.561514] RSP: 0018:ffffc900002ebaa8 EFLAGS: 00000093 [ 2682.561515] RAX: ffffc90000061000 RBX: ffff888100059b40 RCX: 0000000000000003 [ 2682.561516] RDX: 0000000000000001 RSI: ffffc900002ebb00 RDI: 00000000000002bc [ 2682.561517] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000802a0029 [ 2682.561517] R10: 0000000000000000 R11: 0000000000000008 R12: 0000000000000004 [ 2682.561518] R13: ffff88810005e400 R14: ffff888100059b40 R15: 0000000000000001 [ 2682.561519] FS: 0000000000000000(0000) GS:ffff88904f800000(0000) knlGS:0000000000000000 [ 2682.561519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2682.561520] CR2: 000014ff3956cd88 CR3: 0000000005618005 CR4: 0000000000772ef0 [ 2682.561521] PKRU: 55555554 [ 2682.561521] Call Trace: [ 2682.561523] <TASK> [ 2682.561525] ? __slab_free+0x8a/0x240 [ 2682.561527] qi_flush_iec.isra.0+0x5e/0x80 [ 2682.561529] modify_irte.isra.0+0xab/0xd0 [ 2682.561530] ? __synchronize_hardirq+0x88/0xc0 [ 2682.561533] intel_irq_remapping_deactivate+0x46/0x70 [ 2682.561535] __irq_domain_deactivate_irq+0x29/0x40 [ 2682.561537] irq_domain_deactivate_irq+0x1a/0x30 [ 2682.561538] free_irq+0x1ad/0x210 [ 2682.561540] pci_free_irq+0x18/0x30 [ 2682.561543] nvme_suspend_io_queues+0x23/0x30 [ 2682.561545] nvme_dev_disable+0xf5/0x1c0 [ 2682.561546] nvme_timeout+0x3a9/0x400 [ 2682.561548] blk_mq_handle_expired+0x2e/0x50 [ 2682.561550] bt_iter+0x6a/0x90 [ 2682.561551] ? __pfx_bt_iter+0x10/0x10 [ 2682.561553] __sbitmap_for_each_set.constprop.0+0xa6/0xf0 [ 2682.561554] ? __pfx_blk_mq_handle_expired+0x10/0x10 [ 2682.561555] bt_for_each+0x4b/0x70 [ 2682.561557] ? __pfx_blk_mq_handle_expired+0x10/0x10 [ 2682.561558] blk_mq_queue_tag_busy_iter+0x14a/0x190 [ 2682.561560] blk_mq_timeout_work+0x91/0x130 [ 2682.561561] process_scheduled_works+0x199/0x2c0 [ 2682.561564] worker_thread+0x13b/0x1c0 [ 2682.561566] ? __pfx_worker_thread+0x10/0x10 [ 2682.561567] kthread+0xec/0x100 [ 2682.561569] ? __pfx_kthread+0x10/0x10 [ 2682.561570] ret_from_fork+0x21/0x40 [ 2682.561573] ? __pfx_kthread+0x10/0x10 [ 2682.561574] ret_from_fork_asm+0x1a/0x30 [ 2682.561577] </TASK> After reboot I had removed the "nvme_core.io_timeout=4294967295" of the syslinux.cfg, as I did not see it in any solution which had solved nvme problems with Kernel 6.12 too. In general, there seems to be a pretty decent amount of people having problems with NVME dropping due to "Unable to change power state from D3cold to D0, device inaccessible" or such, when running anything above kernel 6.10, but espacially 6.12. Now my syslinux.cfg looks like that: kernel /bzimage append initrd=/bzroot pcie_aspm=off pcie_port_pm=off nvme_core.default_ps_max_latency_us=0 AFAIK the diagnostics will be deleted upon reboot, right? Should I post the anonymized just here as attachement then? I will post them when I got the NVMEs dropping again, as the log is pretty clean for now. For now they are fine, also, the speeds are okay now, which is really strange as I changed almost nothing: hdparm -tT /dev/nvme0n1 /dev/nvme0n1: Timing cached reads: 35840 MB in 2.00 seconds = 17953.52 MB/sec Timing buffered disk reads: 10184 MB in 3.00 seconds = 3394.42 MB/sec hdparm -tT /dev/nvme1n1 /dev/nvme1n1: Timing cached reads: 36074 MB in 2.00 seconds = 18068.53 MB/sec Timing buffered disk reads: 10034 MB in 3.00 seconds = 3343.98 MB/secI wont touch it for now and wait. If theres no crash in the next 24 hours, I will gradually add drives to the pool again.

So, after updating to 7.2.0 the NVMe dropping seems fine for now, will let it run for a while. But the speed is still really slow, around 28-30 MBs per device. Each is connected with x4 PCIe 3.0 (they can make use of x8 but it should be fine) I will try to check them on another system, to make sure if thats something with Unraid or the System itself. For now all of them share a single powerconnection, I guess thats maybe too much, as a single drive can suck up to 25W, minimum 5W in idle. Im not sure what a single 12V SATA Connector and PSU Rail can handle.

@JorgeB Running on 7.1.4 but I will upgrade and test again!

Hi all, this is a special one I guess, but I have problems with my pool of NVMe going down, all of them at the same time. This happens some time after booting, so I already had them up and running to do tests, but they where pretty slow too (around 250-300 MBs instead of up to 4000 MBs each) Setup: ASRock B760 Pro RS Highpoint Rocket 1180 8x Kioxia CD8R I already changed the syslinux.cfg to disable powersaving options: default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage append initrd=/bzroot pcie_aspm=off pcie_port_pm=off nvme_core.default_ps_max_latency_us=0 nvme_core.io_timeout=4294967295 label Unraid OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest But I get following fails: [ 0.000000] Command line: BOOT_IMAGE=/bzimage initrd=/bzroot pcie_aspm=off pcie_port_pm=off nvme_core.default_ps_max_latency_us=0 nvme_core.io_timeout=4294967295 [ 0.085606] Kernel command line: BOOT_IMAGE=/bzimage initrd=/bzroot pcie_aspm=off pcie_port_pm=off nvme_core.default_ps_max_latency_us=0 nvme_core.io_timeout=4294967295 [ 32.814789] nvme 0000:0b:00.0: platform quirk: setting simple suspend [ 32.814832] nvme 0000:0f:00.0: platform quirk: setting simple suspend [ 32.814852] nvme nvme0: pci function 0000:03:00.0 [ 32.815005] nvme nvme1: pci function 0000:04:00.0 [ 32.815017] nvme nvme6: pci function 0000:0a:00.0 [ 32.815021] nvme nvme5: pci function 0000:09:00.0 [ 32.815024] nvme nvme7: pci function 0000:0b:00.0 [ 32.815026] nvme nvme2: pci function 0000:07:00.0 [ 32.815029] nvme nvme4: pci function 0000:08:00.0 [ 32.815030] nvme nvme3: pci function 0000:05:00.0 [ 32.815051] nvme nvme8: pci function 0000:0f:00.0 [ 32.828946] nvme nvme8: D3 entry latency set to 10 seconds [ 32.832564] nvme nvme8: 16/0/0 default/read/poll queues [ 32.835827] nvme nvme7: allocated 64 MiB host memory buffer. [ 32.851029] nvme nvme0: 16/0/0 default/read/poll queues [ 32.853065] nvme nvme5: 16/0/0 default/read/poll queues [ 32.857742] nvme nvme7: 8/0/0 default/read/poll queues [ 32.858838] nvme nvme6: 16/0/0 default/read/poll queues [ 32.868016] nvme nvme3: 16/0/0 default/read/poll queues [ 32.868040] nvme nvme1: 16/0/0 default/read/poll queues [ 32.868183] nvme nvme2: 16/0/0 default/read/poll queues [ 32.868189] nvme nvme4: 16/0/0 default/read/poll queues [ 33.364496] megaraid_sas 0000:0c:00.0: NVMe passthru support : Yes [ 33.385600] megaraid_sas 0000:0c:00.0: NVME page size : (4096) [ 33.484267] scsi 1:0:0:0: Direct-Access NVMe SAMSUNG MZ1LB960 602Q PQ: 0 ANSI: 6 [ 33.579619] scsi 1:0:1:0: Direct-Access NVMe SAMSUNG MZ1LB960 602Q PQ: 0 ANSI: 6 [ 53.445530] nvme nvme8: using unchecked data buffer [ 198.897156] nvme0n1: p1 [ 208.590142] nvme1n1: p1 [ 218.554148] nvme2n1: p1 [ 228.517890] nvme3n1: p1 [ 298.639054] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 298.639072] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 298.639076] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug [ 298.682985] nvme 0000:03:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 298.683100] nvme nvme0: Disabling device after reset failure: -19 [ 304.284799] nvme nvme3: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 304.284817] nvme nvme3: Does your device have a faulty power saving mode enabled? [ 304.284821] nvme nvme3: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug [ 304.329902] nvme 0000:05:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 304.330012] nvme nvme3: Disabling device after reset failure: -19 [ 316.558563] nvme nvme5: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 316.558588] nvme nvme5: Does your device have a faulty power saving mode enabled? [ 316.558589] nvme nvme5: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug [ 316.596468] nvme 0000:09:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 316.596622] nvme nvme5: Disabling device after reset failure: -19 [ 361.614215] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 361.614233] nvme nvme1: Does your device have a faulty power saving mode enabled? [ 361.614237] nvme nvme1: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug [ 361.667177] nvme 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 361.667302] nvme nvme1: Disabling device after reset failure: -19 [ 365.709093] nvme nvme2: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 365.709099] nvme nvme2: Does your device have a faulty power saving mode enabled? [ 365.709099] nvme nvme2: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug [ 365.759086] nvme 0000:07:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 365.759193] nvme nvme2: Disabling device after reset failure: -19 [ 427.147284] nvme nvme4: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 427.147301] nvme nvme4: Does your device have a faulty power saving mode enabled? [ 427.147305] nvme nvme4: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug [ 427.186203] nvme 0000:08:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 427.186319] nvme nvme4: Disabling device after reset failure: -19 [ 427.202270] Buffer I/O error on dev nvme4n1, logical block 0, async page read [ 427.202332] nvme4n1: unable to read partition table [ 427.202337] nvme4n1: partition table beyond EOD, truncated [ 488.584535] nvme nvme6: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 488.584550] nvme nvme6: Does your device have a faulty power saving mode enabled? [ 488.584551] nvme nvme6: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug [ 488.604703] nvme6n1: I/O Cmd(0x2) @ LBA 0, 128 blocks, I/O Error (sct 0x3 / sc 0x71) [ 488.604706] I/O error, dev nvme6n1, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 0 [ 488.619515] nvme 0000:0a:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 488.619630] nvme nvme6: Disabling device after reset failure: -19 [ 488.630530] Buffer I/O error on dev nvme6n1, logical block 0, async page read [ 575.302497] block nvme0n1: No UUID available providing old NGUID [ 575.302515] nvme nvme0: Identify namespace failed (-5) [ 575.302628] nvme nvme1: Identify namespace failed (-5) [ 575.302736] nvme nvme2: Identify namespace failed (-5) [ 575.302846] nvme nvme3: Identify namespace failed (-5) [ 575.302957] nvme nvme4: Identify namespace failed (-5) [ 575.303074] nvme nvme5: Identify namespace failed (-5) [ 575.303197] nvme nvme6: Identify namespace failed (-5) Has anyone a clue of whats going on here? They are all conntected via a single, daisy-chained SATA-Power connector, could that be an problem? All cables and drives are cool, the drives themself around 45-55°C.

Thanks! So, for NVMe or SSD in general, it should be avoided to add them into the array, as they are not making use of TRIM support there?

Hi all, I started using Unraid like two months ago and so far everything works fine. I added some more disks fast, so my first question: When to add a second parity? Do I already have too much disks in, or should I be relaxed until a adding the disk number N to the array? My actual setup: 1 Parity - 18 TB 4 Data - 18 TB (A 5th disk is already prepared) 2 Data - 12 TB 2 Data - 10 TB (Not yet installed but already prepared for) Second Question: Is is possible to add a second array? Background: have all my VMs and stuff running on a mirrored NVMe BTRFS Pool, which is fine but too small. I will add a HighPoint Rocket 1180 HBA with 8 x 7,68 TB NVMe Disks to the Host, and I would want to use the same Unraid-Feature for them too. And add a first backup of the data of the NVMes onto the spinning array each night. Or should I create a ZFS Pool for that? I would love to have the best performance but also keeping some flexibility. BR