warpspeed
Members
-
Joined
Everything posted by warpspeed
-
Oh interesting. I updated one of my drives in a two drive ZFS pool and it seemed to help. Haven't had any additional issues since doing so. Wonder why they pulled it.
-
Update the firmware on the 990 pro to 6B2QJXD7 too. That's also supposed to help.
-
So far it has been okay, but the failure happens randomly. I also found a recent kernel change that may be able to recover the drives without a hard power cycle, I posted about it here: https://forums.unraid.net/bug-reports/prereleases/72-beta-3-request-kernel-bug-fix-for-nvme-ssd-reset-r4044/
-
Not a direct bug in beta2, but a possible bug fix/mitigation for an ongoing NVME issue. Some folks (me included) have had problems with NVME SSDs dropping off and requiring a hard power cycle to recover them. I was just looking through the kernel change log and saw this: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.12.y&id=bc255fec6fd7300b56c7adad5cb8482cd7cac81a It looks like that was released in 6.12.43, and we're currently on 6.12.42 in 7.2 beta 2 Could we move beta 3 to 6.12.43/44 to pick up this fix? The errors mentioned are exactly what I see when the drive drops off and doesn't come back. e.g. an 26 10:31:17 unraid kernel: nvme nvme1: I/O 840 (I/O Cmd) QID 1 timeout, aborting Jan 26 10:31:17 unraid kernel: nvme nvme1: I/O 537 (I/O Cmd) QID 5 timeout, aborting Jan 26 10:31:17 unraid kernel: nvme nvme1: I/O 797 (I/O Cmd) QID 9 timeout, aborting Jan 26 10:31:17 unraid kernel: nvme nvme1: Abort status: 0x0 Jan 26 10:31:17 unraid kernel: nvme nvme1: Abort status: 0x0 Jan 26 10:31:17 unraid kernel: nvme nvme1: Abort status: 0x0 Jan 26 10:31:18 unraid kernel: nvme nvme1: I/O 980 (I/O Cmd) QID 15 timeout, aborting Jan 26 10:31:18 unraid kernel: nvme nvme1: Abort status: 0x0 Jan 26 10:31:18 unraid kernel: nvme nvme1: I/O 536 (I/O Cmd) QID 5 timeout, aborting Jan 26 10:31:47 unraid kernel: nvme nvme1: I/O 840 QID 1 timeout, reset controller Jan 26 10:32:18 unraid kernel: nvme nvme1: I/O 0 QID 0 timeout, reset controller Jan 26 10:33:08 unraid kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1 Jan 26 10:33:08 unraid kernel: nvme nvme1: Abort status: 0x371 Jan 26 10:33:29 unraid kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1 Jan 26 10:33:29 unraid kernel: nvme nvme1: Removing after probe failure status: -19 Jan 26 10:33:49 unraid kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1 Jan 26 10:33:49 unraid kernel: nvme1n1: detected capacity change from 3907029168 to 0 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114661376 size=61440 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114137088 size=12288 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114006016 size=49152 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114530304 size=131072 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908113088512 size=131072 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=1804943826944 size=131072 flags=1808b0 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908114399232 size=131072 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=1804943695872 size=131072 flags=1808b0 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908110991360 size=12288 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=1804943564800 size=131072 flags=1808b0 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1908113874944 size=28672 flags=180880 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Jan 26 10:33:49 unraid kernel: zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 01:46:15 2025] docker0: port 3(vethbb8ba86) entered forwarding state [Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 821 (I/O Cmd) QID 2 timeout, aborting [Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 822 (I/O Cmd) QID 2 timeout, aborting [Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 260 (I/O Cmd) QID 14 timeout, aborting [Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0 [Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0 [Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0 [Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 888 (I/O Cmd) QID 3 timeout, aborting [Sun Jun 1 10:35:55 2025] nvme nvme1: Abort status: 0x0 [Sun Jun 1 10:35:55 2025] nvme nvme1: I/O 931 (I/O Cmd) QID 7 timeout, aborting [Sun Jun 1 10:36:25 2025] nvme nvme1: I/O 821 QID 2 timeout, reset controller [Sun Jun 1 10:36:56 2025] nvme nvme1: I/O 3 QID 0 timeout, reset controller [Sun Jun 1 10:37:47 2025] nvme nvme1: Device not ready; aborting reset, CSTS=0x1 [Sun Jun 1 10:37:47 2025] nvme nvme1: Abort status: 0x371 [Sun Jun 1 10:38:07 2025] nvme nvme1: Device not ready; aborting reset, CSTS=0x1 [Sun Jun 1 10:38:07 2025] nvme nvme1: Removing after probe failure status: -19 [Sun Jun 1 10:38:28 2025] nvme nvme1: Device not ready; aborting reset, CSTS=0x1 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1825897586688 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610704896 size=53248 flags=180880 [Sun Jun 1 10:38:28 2025] nvme1n1: detected capacity change from 3907029168 to 0 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610442752 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610311680 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917606903808 size=12288 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=97925271552 size=131072 flags=1808b0 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610573824 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917607559168 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1825897717760 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=97925140480 size=131072 flags=1808b0 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=1 offset=97925009408 size=131072 flags=1808b0 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1906971295744 size=45056 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610049536 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1825897848832 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=2 offset=1917610180608 size=131072 flags=180880 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 [Sun Jun 1 10:38:28 2025] zio pool=nvme_data vdev=/dev/nvme1n1p1 error=5 type=5 offset=0 size=0 flags=100480 Ref:
-
In my plugins folder I have three folders for disklocation... disklocation disklocation-master disklocation-2024-edition do I need all three of these? the disklocation folder seems to have most of the bulk of the plugin the disklocation-master folder has a disklocation.cron file in it the disklocation-2024-edition folder seems empty
-
This should probably be in the motherboard forum. I have an X13-SAE-F, and the ram I got fits just fine.
-
FYI I've put 6B2QJXD7 on my one problematic drive and so far I've not had any further issues with it. I followed the instructions on how to do the firmware upgrade from here: https://forums.unraid.net/topic/132024-guide-how-to-upgrade-samsung-nvmessd-firmware/ My other drive I've never had any issues with. I will update that one at some point too, and once I've done both and they're stable, I'll try dropping out those kernel commands.
-
Yeah, I put it on my problematic 2TB 990 Pro. One of two in a ZFS mirror kept dropping offline. So far so good. I'll do the other drive in time, but I'm in no rush because that one hasn't given me any problems so far.
-
Thanks for the response Tiffany. I did watch that video, it was a great one.
-
19th January 2011 was when I got my first license key emailed to me by Tom. So not quite in the 15-year category just yet.
-
I'd like to see better posting/updating of discovered/confirmed issues in the release notes. I'd also like to see development status, rather than the current blackhole question mark of what's going on. What I mean by that is, right now we have 7.1.4 and we have heard nothing from Lime Tech about whether they're working on bug fixes in a 7.1.5 release, or whether they've given up and are working on 7.2. i.e. it'd be nice if we could at least have some idea of what's happening/what they're thinking.. e.g. if there's an issue or two in 7.1.4 that are being worked on for a 7.1.5 release, then it'd be nice to know about that so those of us that are more conservative can wait for 7.1.5 rather than wondering if 7.1.4 is stable enough and to try it. If there's not, then it'd be nice to know that too... i.e. we view 7.1.4 as generally pretty stable, we've moved on to 7.2. My problem is, and this is where the lack of confidence comes from... I see 7.0 come out with a heap of new features, and think hmm, I'll wait and see how stable it is.. 7.0.1 comes with some fixes, but still discussions of issues. Nope, I'll wait for 7.0.2 and see how that goes. But no, on to 7.1 which introduces a whole bunch more changes. Wait for 7.1.1, nope, more issues Wait for 7.1.2, nope, more issues Wait for 7.1.3 nope, more issues, almost immediately... sheesh what testing went on here Wait for 7.1.4 rush fix for 7.1.3, but looks to be starting to get fairly stable now, so.... what is the status???? Here we are over a month later with no communication on whether there's a 7.1.5 to clean up a few more of the outstanding issues, or whether they're going to go to 7.2. Going to 7.2 starts the whole conservative cycle all over again because of new introduced features. My gut feel given the radio silence is everything has moved on to 7.2 and we will see a 7.2 release around this anniversary thing coming up. So that'll start the cycle all over again. I've been using unraid for a very long time. To highlight my recent upgrade path I've gone from 5.0.5 > 6.1.9 > 6.9.2 > 6.11.5 > 6.12.10 (tried 6.12.11 but had to revert) > 6.12.15. I am still on 6.12.15. The 7.1.x series is starting to look fairly mature, with the exception of the rush-fix in 7.1.4. But it seems like it's falling just a little bit short if we now move to 7.2. So I guess my ask here is also that it'd be nice to see a given train worked a bit more toward bug fixes before moving to the new features. Maybe an LTS version, effectively. So perhaps this doesn't need to happen every point release. But perhaps pick a sensible/good one, and keep at it with bug fixes in that release a bit longer for those of us that err on the more conservative side.
-
I've seen a few posts that there's a new firmware available: 6B2QJXD7 I wonder if this fixes these dropout issues
-
Has anyone tried the new 6B2QJXD7 firmware to see if it gets rid of the disk drops?
-
Interestingly checking my array drives I have some that have reflink enabled and some that don't. I've never changed any settings to manipulate this myself. Was there a time when reflink wasn't enabled by default?
-
/var/log on my server filled up and was full for quite a while and I didn't know the only way I found out was some obscure errors from cron.daily trying to rotate logs and erroring because it was full can we have unraid monitor and warn about all mounted partitions the same as it does for arrays/mounts/shares ? I'm using 6.12.15 so maybe this has been solved in 7.x but I don't recall reading anything about that in the release notes
-
I'm curious to know if Limetech are working on a 7.1.5 release or whether they're done and now they're moving on to 7.2, etc, etc. It seems there's still a few rough edges for a number of folks with the 7.1.x series. It'd be really good to see if some of those could be further refined and cleaned up before moving on.
-
I'm still sitting on 6.12.15 too. I don't yet have full confidence in the 7.x series based on the threads I've seen.
-
I just had this same issue on my server running 6.12.15. Uptime 38 days. Had to completely stop and then restart nginx, then it came good again.
-
I wonder if stuff like this should be added to Fix Common Problems or some sort of update pre-check
-
I lack confidence with how unraid handle releases too, however, in the case of this release: It has been several weeks since the last release. It's not a new release every week. And I would rather they do smaller, regular updates to work on bug fixes, rather than larger monolithic releases that are more likely to break multiple things. They did do a test build as a pre-release - here's the thread: https://forums.unraid.net/bug-reports/prereleases/unraid-os-version-714-rc1-available-r3912/ However, the bit that I'm critical about here, is that according to the last post in the RC thread, there are some changes between the RC and Final. Why then wasn't there a second RC issued first. Who knows if the other changes haven't introduced further bugs? Within the minor releases, I'd really like to see small iterative and frequent releases to address specific bugs, rather than trying to bundle too many changes into one release. e.g. If a kernel update is done in a minor release.... JUST do the kernel, nothing else. Then if things break or are fixed for someone, then we know it's the kernel at least. When there's a dozen other changes, who knows. I do note in the recent bug fixes the kernel has been held at 6.12.24, which is good to eliminate that as a source of issue. I do wonder at what point with a point release, it would be useful to move to the next minor release kernel in case that addresses some of the bugs. FWIW, I'm still sitting on 6.12.15... and the one thing that really annoys me about that is the most recent Tailscale updates have been restricted to 7.x. So I cannot install further Tailscale updates, which is crazy.
-
Everyone needs to do the table from this post, this is awesome detail
-
Looks like a lot of discussion here: https://bugzilla.kernel.org/show_bug.cgi?id=216809
-
Only one of my drives has the problem, pretty sure its the one not connected directly to the CPU. I dont want to mess around with it because I don't want the stable drive to start having problems too. I wonder if this is related: https://bugzilla.kernel.org/show_bug.cgi?id=215742 When I boot I see the platform quirk thing in dmesg output: root@unraid:~# dmesg | grep nvme [ 0.000000] Command line: BOOT_IMAGE=/bzimage initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off [ 0.154215] Kernel command line: BOOT_IMAGE=/bzimage initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off [ 36.874725] nvme 0000:02:00.0: platform quirk: setting simple suspend [ 36.879291] nvme nvme0: pci function 0000:02:00.0 [ 36.885589] nvme 0000:03:00.0: platform quirk: setting simple suspend [ 36.893575] nvme nvme1: pci function 0000:03:00.0 [ 37.093198] nvme nvme0: Shutdown timeout set to 10 seconds [ 37.093229] nvme nvme1: Shutdown timeout set to 10 seconds [ 37.107599] nvme nvme1: 16/0/0 default/read/poll queues [ 37.112297] nvme nvme0: 16/0/0 default/read/poll queues [ 37.123589] nvme1n1: p1 [ 37.129541] nvme0n1: p1
-
I have this issue too. I have 2x 990 Pro 2TB in a ZFS mirror. One drops the other doesn't. I've tried all the things. From all the things I've read, this is possibly a kernel interaction bug on Linux.
-
This one didnt work for me without taking out PARTN (6.12.15). I wonder if these would be useful added into the Upgrade Assistant or Fix Common Problems
