July 12, 2025Jul 12 Hey so in the last 2 weeks i have noticed im getting a new weird issue.I have two cache drives as two separate pools"cache" and "generic"i use cache for mostly appdata and generic for downloads, transcodes ect.ive started getting " Pool has encountered an uncorrectable I/O failure and has been suspended."a reboot will bring it back but it can last a few days and sometimes only a few hours.90% of the time it happens to the cache drive (nvme0n1)But i have seen it once affect my other one (nvme1n1)Does anyone have any idea what i can do to troubleshoot this? the cache nvme is a drive that has done alot in the last two years (~1.5PB read/write) and i just assumed it was on its way out but then it happend to the newer one which has only seen ~70TB R/W so far which makes me question if its something else.i do not believe it was an unraid update (i am running 7.1.4 atm but it was happening on 7.1.3 also)i have attached a diagnotics zip when the problem occurred and i can see logs ```Jul 12 18:20:58 MEDIABOX kernel: pcieport 0000:00:06.0: AER: Correctable error message received from 0000:00:06.0Jul 12 18:20:58 MEDIABOX kernel: pcieport 0000:00:06.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)Jul 12 18:20:58 MEDIABOX kernel: pcieport 0000:00:06.0: device [8086:a74d] error status/mask=00000001/00002000Jul 12 18:20:58 MEDIABOX kernel: pcieport 0000:00:06.0: [ 0] RxErr (First)Jul 12 18:21:29 MEDIABOX kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10Jul 12 18:21:29 MEDIABOX kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?Jul 12 18:21:29 MEDIABOX kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bugJul 12 18:21:29 MEDIABOX kernel: nvme 0000:02:00.0: enabling device (0000 -> 0002)Jul 12 18:21:29 MEDIABOX kernel: nvme nvme0: Disabling device after reset failure: -19Jul 12 18:21:29 MEDIABOX kernel: I/O error, dev nvme0n1, sector 559389704 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0Jul 12 18:21:29 MEDIABOX kernel: I/O error, dev nvme0n1, sector 42888448 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0Jul 12 18:21:29 MEDIABOX kernel: I/O error, dev nvme0n1, sector 160791304 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0Jul 12 18:21:29 MEDIABOX kernel: I/O error, dev nvme0n1, sector 1257939184 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0Jul 12 18:21:29 MEDIABOX kernel: I/O error, dev nvme0n1, sector 1780483096 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=286406479872 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=2 offset=21957836800 size=131072 flags=3162240Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=82324099072 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: I/O error, dev nvme0n1, sector 1628102984 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=911606296576 size=45056 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=644063813632 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: I/O error, dev nvme0n1, sector 256392024 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=833587679232 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=781692096512 size=45056 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=131271667712 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=286409359360 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=2 offset=874528718848 size=12288 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=911606173696 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=25036091392 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=2 offset=911606902784 size=45056 flags=2148533376Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=911602991104 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=2 offset=911606976512 size=16384 flags=2148533376Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=286409367552 size=45056 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=2 offset=874528731136 size=12288 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=911606784000 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=1 offset=164190846976 size=4096 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: zio pool=cache vdev=/dev/nvme0n1p1 error=5 type=2 offset=874528743424 size=24576 flags=3145856Jul 12 18:21:29 MEDIABOX kernel: WARNING: Pool 'cache' has encountered an uncorrectable I/O failure and has been suspended.### [PREVIOUS LINE REPEATED 27 TIMES] ###Jul 12 18:21:56 MEDIABOX kernel: traps: vwmetrics[15027] general protection fault ip:15323630f50f sp:7ffecb3506c0 error:0 in libc.so.6[2650f,15323630f000+155000]Jul 12 18:21:58 MEDIABOX kernel: WARNING: Pool 'cache' has encountered an uncorrectable I/O failure and has been suspended.```i ran the general repair scrub ect, after reboot it doesn't say there is any data errors. but the drive does insta fail the quick smart testi did see Jul 12 18:21:29 MEDIABOX kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bugBut i find it strange after 2 years of using this it would be a power saving issue right?cc @JorgeB (as i know you have a wealth of info on this based on other posts ive come across)Thanks! mediabox-diagnostics-20250712-1830.zip
July 12, 2025Jul 12 Community Expert 34 minutes ago, phyzical said:nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=offIt's still worth trying to add this line to syslinux, the behavior can sometimes change with different kernels.
July 12, 2025Jul 12 Community Expert 53 minutes ago, phyzical said:but the drive does insta fail the quick smart testForgot to mention, I assume this is only after the device drops offline?
July 12, 2025Jul 12 Author 9 minutes ago, JorgeB said:Forgot to mention, I assume this is only after the device drops offline?no the smart short self test failure is post reboot so to the system everything is "working" the newer one runs fine but the older one, it doesnt even go to a % just returns Completed: failed segments ill give the syslinux line a go then.Have you ever seen it cause a non faulty one go suspended? incase it really is just a close to death ssd?the cache pool nvme drive itself reports in the attributes-Available spare100%-Available spare threshold10%-Percentage used107%-SSD endurance remaining-7 %
July 13, 2025Jul 13 Community Expert 20 hours ago, phyzical said:no the smart short self test failure is post reboot so to the system everything is "working"the newer one runs fine but the older one, it doesnt even go to a % just returns Completed: failed segmentsIn that case you should replace the device.
July 14, 2025Jul 14 Author i do agree, and was going to. until i saw the other drive do the same thing just wanted 2 cents from others incase its actually something else.
August 8, 2025Aug 8 Author Sigh.... everytime you decide its fixed.its not.friday comes i finish and BAM cpu pegged at 100% i was suspicious, but rebooted.Nothing showing in htop as using anything ina. crazy amount.. i see the occasional ls of all disk drivessame thing happend and then The second nvme started doing what the first one did again... this drive s solely used for temp data but is connected to the main pool of drivesthis second nvme is pretty newthis is driving me crazy... Edited August 8, 2025Aug 8 by phyzical
August 8, 2025Aug 8 Author just trying the power nvme append to the boot drive as i didnt try that the first time (its possible ymnewer one does suffer from it and the older one was just dieing)
August 8, 2025Aug 8 Community Expert ZFS is detecting data corruption. That is usually not a device problem; most often it is bad RAM, but since you have a 14700K, it could also be the Intel 13/14 gen issue, since that model is one of the most affected.Look for a BIOS update; that can help if the CPU is not too far gone.
August 8, 2025Aug 8 Author ive already gone through all that in the past. i was running into those issues for quite a while around a year ago, did all the bios updates, ill check for new updates just incase thoughit ended up being a cooling issue i found. and after i replaced the cooler it went away. until recently this started occuring.it was constantly that one nvme. after replacing it is now affecting the second one. this nvme only has 25 days online time.. so i dont think its the driveand sadly it just occurred again with the boot drive append.ill try mem testing again, back when i was running into the cpu issues it never failed though.. Edited August 8, 2025Aug 8 by phyzical
August 8, 2025Aug 8 Author one thought i have had though is this might have all started once i started running two nvme drives
August 8, 2025Aug 8 Community Expert 4 minutes ago, phyzical said:this might have all started once i started running two nvme drivesDon't think that would be an issue, though it's not impossible, but in my experience, 99% of the time data corruption is caused by bad RAM or CPU.
August 8, 2025Aug 8 Author do you think i would have a better time running this nvme drive not on zfs? or it would just surface in a different way?
August 8, 2025Aug 8 Community Expert Btrfs would also detect any data corruption, XFS not, but the data corruption would still happen.
August 8, 2025Aug 8 Author so i have come across a few posts around heat causing the controllers to fail, and this ssd would be sitting directly below the GPU and it has one of those combined heatsink things. and that is pretty hot.i know that nmes are supposed to be warm and fine. but maybe ill try moving it away from the slow directly below the gpu?
August 8, 2025Aug 8 Author also noticed that nvme firmwares are a thing... so ill try one a time ofcnvme firmware mobo biosmove drive to another slot
August 8, 2025Aug 8 Community Expert The cooler the better, but high temps should never cause data corruption.
August 8, 2025Aug 8 Author hmm so the latest firmware for the affect nvme actually states (6B2QJXD7) To address the intermittent non-recognition Some issue posts of people having the 990 pro drive just disconnect on windows pcs mid gaming ect.But there was also 2 more mobo bio updates so ive actioned that and the nvme firmwareim just going to put on my fingers crossed hat
August 8, 2025Aug 8 Author no instant crash like the last 3 hours. but ive seen that before,will update in the coming days.given my track record maybe i should say nothing its like my server is watching me ☠️
August 18, 2025Aug 18 Author touch wood. all is fine after updating the ssd's firmware (or the mobo bios update) Edited August 18, 2025Aug 18 by phyzical
August 31, 2025Aug 31 Author noope ;(i will try xfs next i think and then follow up with removing the 990 pro and instead try the 980 + 990 evo plusedit: switched 990 pro to xfs will see what happens over the next few weeks Edited August 31, 2025Aug 31 by phyzical
August 31, 2025Aug 31 Author well already dropped out within 30 minutes under xfs too..1756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2416483205, offset 164364288, sector 12568148321756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2416483205, offset 160227328, sector 12568067521756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2416483205, offset 122421248, sector 12567329121756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2416483205, offset 95731712, sector 12566807841756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2685179091, offset 18612224, sector 13964746241756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2416483205, offset 64466944, sector 12566197201756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 6242232336, offset 0, sector 32777925681756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2685179091, offset 11272192, sector 13964602881756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1p1: writeback error on inode 2416483205, offset 25690112, sector 12565439841756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: XFS (nvme1n1p1): Please unmount the filesystem and rectify the problem(s).1756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: XFS (nvme1n1p1): Filesystem has been shut down due to log error (0x2).1756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: XFS (nvme1n1p1): log I/O error -51756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: I/O error, dev nvme1n1, sector 1256556272 op 0x1:(WRITE) flags 0x104000 phys_seg 5 prio class 01756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: I/O error, dev nvme1n1, sector 1953518033 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 01756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: I/O error, dev nvme1n1, sector 1256945872 op 0x1:(WRITE) flags 0x104000 phys_seg 72 prio class 01756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme nvme1: Disabling device after reset failure: -191756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme 0000:03:00.0: enabling device (0000 -> 0002)1756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: I/O error, dev nvme1n1, sector 558844616 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 01756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme1n1: I/O Cmd(0x2) @ LBA 558844616, 256 blocks, I/O Error (sct 0x3 / sc 0x71)1756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme nvme1: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug1756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme nvme1: Does your device have a faulty power saving mode enabled?1756616838906 2025-08-31T05:07:18.906Z Aug 31 13:07:18 MEDIABOX kernel: nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x11with xfs though you can still shutdown and reboot via the ui atleast. Edited August 31, 2025Aug 31 by phyzical
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.