This is driving me nuts guys. Need some serious help. Server specs:
- ASUSTeK COMPUTER INC. PRIME B660-PLUS D4
- 12th Gen Intel® Core™ i5-12500T @ 3564 MHz
- Memory: 64 GiB DDR4
- Acer_SSD_FA200_4TB_ASBE64050101841 (nvme0n1)

No smart errors at all. I bought the drive new and it's now 2 months old and the seller does not accept any returns. It always played up but recently is unbearable. 

I'm getting this errors from the Acer on the log system. I have updated the BIOS to the latest, disable ASPM in bios, changed the kernel boot from append initrd=/bzroot pcie_aspm=off  to  append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

 

After working 1 hour without any problems it came back to the same again. Just to open the console takes me 10 minutes. This errors are over and over again maxing out my CPU with I/O waits. For me it does not make sense that for 1 hour is fine and all of a sudden it starts with I/O waits

Apr 7 06:20:06 MotoZona kernel: nvme nvme0: I/O 243 (I/O Cmd) QID 6 timeout, aborting Apr 7 06:20:06 MotoZona kernel: nvme nvme0: I/O 244 (I/O Cmd) QID 6 timeout, aborting Apr 7 06:20:06 MotoZona kernel: nvme nvme0: I/O 245 (I/O Cmd) QID 6 timeout, aborting Apr 7 06:20:10 MotoZona kernel: nvme nvme0: Abort status: 0x0 Apr 7 06:20:13 MotoZona kernel: nvme nvme0: Abort status: 0x0 Apr 7 06:20:15 MotoZona kernel: nvme nvme0: Abort status: 0x0 Apr 7 06:20:16 MotoZona kernel: nvme nvme0: I/O 942 (I/O Cmd) QID 2 timeout, aborting Apr 7 06:20:16 MotoZona kernel: nvme nvme0: I/O 943 (I/O Cmd) QID 2 timeout, aborting Apr 7 06:20:16 MotoZona kernel: nvme nvme0: I/O 944 (I/O Cmd) QID 2 timeout, aborting Apr 7 06:20:23 MotoZona kernel: nvme nvme0: Abort status: 0x0 Apr 7 06:20:23 MotoZona kernel: nvme nvme0: I/O 832 (I/O Cmd) QID 9 timeout, aborting Apr 7 06:20:26 MotoZona kernel: nvme nvme0: Abort status: 0x0 Apr 7 06:20:28 MotoZona kernel: nvme nvme0: Abort status: 0x0 Apr 7 06:20:33 MotoZona kernel: nvme nvme0: Abort status: 0x0

nvme nvme0: I/O 512 QID 10 timeout, reset controller

I have also changed the drive to a different nvme slot and the problem persists

motozona-syslog-20250407-0840.zip

Edited April 7, 20251 yr by errant-sis6255

I found out that when tdarr is running (does not matter if running any tasks), the I/O errors from the cache starts.

Anything that puts stress on the drive leads to I/O errors :,(

Attach Diagnostics ZIP to your NEXT post in this thread.

Thanks for looking into it. The 1045 is from just now and the 2256 is from yesterday

motozona-diagnostics-20250407-1045.zip motozona-diagnostics-20250405-2256.zip

I have put the cache under stress by started the mover from the cache to the array. No I/O errors. It seems the problem lies on writing. So I have also started nzbget and i am writing to the cache at 80mb/s. Under 5 minutes I start to get the same problem again I/O errors

How is this installed? Looks like it is on the same controller as some of your HDD.

The drive is on the nvme slot 1 near the cpu. I have a LSI SAS 9300-16I 12GB/S HBA BUS ADAPTER CARD IT Mode 4*SFF-8643 SATA Cable that feeds the 15 HDD. It is running on the first PCIe slot 4.0 16x

I would recommend trying with a different brand/model device, if possible.

I am going to put a Samsung 990 PRO NVMe M.2 SSD, 4 TB and see if the problem remains or not.

11 hours ago, errant-sis6255 said:

Anything that puts stress on the drive leads to I/O errors :,(

 

10 hours ago, errant-sis6255 said:

I have put the cache under stress by started the mover from the cache to the array. No I/O errors. It seems the problem lies on writing. So I have also started nzbget and i am writing to the cache at 80mb/s. Under 5 minutes I start to get the same problem again I/O errors

 

Had that happen with a Samsung NVME, the drive was dying but not throwing any smart errors. A reboot would make it behave for a while at first, but then it started crapping out on every write to it.

That seems exactly what I am going through.

 

Thanks for that but I looked up on the Acer SSD F240 4TB and I can not see any firmware available to update
S/N: AS8EFA0501Q1841

Firmware version: SN15536

2 hours ago, errant-sis6255 said:

Thanks for that but I looked up on the Acer SSD F240 4TB and I can not see any firmware available to update
S/N: AS8EFA0501Q1841

Firmware version: SN15536

 

the 980 and 990 drives had a firmware problem so if you switch to the 990 make sure you update it (mine was a 970 so it wasn't that)

It was the Acer_SSD_FA200_4TB_ASBE64050101841
I replaced it with a Samsung_SSD_990_PRO_4TB_S7DPNJ0Y102393R - 4 TB with Firmware version:4B2QJXD7

No more problems, everything smooth.