October 20, 20241 yr I have a huge problem and do not know how to solve it. I bought a refurbished Supermicro Server with a LSI SAS 9400-16i HBA. (01/24) I got several 18TB Hitachi drives at different times to grow the system. (4 in 07/23, 2 in 2/24, 1 in 7/24 and 1 in 8/24) (2 RMAs due to sector relocates) started with 3 disks + parity and 2 Samsung 990 Pro 2TB as Cache Pool on Supermicro AOC-SLG3-2M2 PCI card. grew to 5 disks + parity added parity2 From the beginning I had lots of UDMA CRC Errors on all drives. (almos 20k on a drive that has 10k power on hours) The UDMA CRC Errors only increase when there are write operations on the disks. Recently I noticed that the drives are getting loads of WRITE FPDMA QUEUED and ICRC, ABRT at LBA errors ( 300-500 in oldest drives, 47 in my latest parity2 drive). Currently disk1 is not in the system and is beeing emulated, as I needed to recover data from it after a stupid mistake I made. ( After a clean shutdown due to a power failure (UPS covered) parity1 was removed from the array due to errors on the next boot. Then I saw loads of errors piling up on parity2 and I decided to shutdown. Since the mover was running at that time and the shutdown did no go through and the system became unresponsive even in IMPI KVM console, I switched the power off through IPMI. I did try to preclear parity1 to get it back into the array, but the zeroing failed. Extended SMART Test on parity1 completed without error. Currently I am preclearing parity1 again to get it back into the array, because the array is currently without any redundancy. (disk1 and parity1 removed). Since the system is around 800km away from me and I have no spare HBA, SATA cables or PSU I can not really test a lot. Details of the Server: Supermicro CSE-829U X10DRU-i+ 19" 2U 12x 3,5" LFF 2x Intel XEON E5-2600 v3 v4 DDR4 ECC Raid 4x 10GbE X540 2x PSU 2 x Intel Xeon E5-2698V4 SR2JW 20C Server Prozessor 20x 2,20 GHz 50MB 2011-3 128GB Registered ECC DDR4 SDRAM (2x64GB DIMM) LSI SAS 9400-16i PCIe x8 4x SFF-8643 12G SAS3 NVMe HBA HDD SSD NVMe JBOD Tri-Mode Controller (ZFS, Ceph, MS Storage Spaces) 2 x Supermicro 1000W PSU Netzteil PWS-1K02A-1R (redundant PSU) BlueWalker PowerWalker VFI 3000 ICR IoT, USV unraid-01-diagnostics-20241020-2232.zip Edited October 20, 20241 yr by Plak added specifics on cache pool HW. fixed typos.
October 23, 20241 yr Author Strangely the system is running fine currently. Only thing I did was to gather the HBA information in the BIOS. Since that I do not see any increase in UDMA CRC Errors on the disks and the preclear is doing fine. I did see strange behavior of smartctl in the preclear (18TB TOSHIBA MG09ACA18TE disk): Oct 20 22:14:15 preclear_disk_SN_6258: Preclear Disk Version: 1.0.30 Oct 20 22:14:16 preclear_disk_SN_6258: Disk size: 18000207937536 Oct 20 22:14:16 preclear_disk_SN_6258: Disk blocks: 4394582016 Oct 20 22:14:16 preclear_disk_SN_6258: Blocks (512 bytes): 35156656128 Oct 20 22:14:16 preclear_disk_SN_6258: Block size: 4096 Oct 20 22:14:16 preclear_disk_SN_6258: Start sector: 0 Oct 20 22:14:21 preclear_disk_SN_6258: Pre-read: pre-read verification started 1 of 5 retries... Oct 21 02:36:48 preclear_disk_SN_6258: Pre-Read: progress - 25% read @ 277 MB/s Oct 21 07:23:18 preclear_disk_SN_6258: Pre-Read: progress - 50% read @ 246 MB/s Oct 21 12:59:07 preclear_disk_SN_6258: Pre-Read: progress - 75% read @ 198 MB/s Oct 21 20:39:07 preclear_disk_SN_6258: Pre-Read: elapsed time - 22:24:43 Oct 21 20:39:07 preclear_disk_SN_6258: Pre-read: pre-read verification completed! Oct 21 20:39:08 preclear_disk_SN_6258: Zeroing: zeroing the disk started 1 of 5 retries... Oct 21 20:39:08 preclear_disk_SN_6258: Zeroing: emptying the MBR. Oct 22 01:00:29 preclear_disk_SN_6258: Zeroing: progress - 25% zeroed @ 278 MB/s Oct 22 05:46:46 preclear_disk_SN_6258: Zeroing: progress - 50% zeroed @ 244 MB/s Oct 22 11:21:50 preclear_disk_SN_6258: Zeroing: progress - 75% zeroed @ 201 MB/s Oct 22 19:01:35 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:01:35 preclear_disk_SN_6258: Pause (smartctl run time: 34s) Oct 22 19:01:35 preclear_disk_SN_6258: Paused Oct 22 19:01:56 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:02:18 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:02:40 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:03:02 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:03:23 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:03:45 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:04:07 preclear_disk_SN_6258: Resumed Oct 22 19:04:09 preclear_disk_SN_6258: Zeroing: zeroing the disk completed! Oct 22 19:04:10 preclear_disk_SN_6258: Signature: writing signature... Oct 22 19:04:12 preclear_disk_SN_6258: Signature: verifying Unraid's signature on the MBR ... Oct 22 19:04:13 preclear_disk_SN_6258: Signature: Unraid preclear signature is valid! Oct 22 19:04:14 preclear_disk_SN_6258: Post-Read: post-read verification started 1 of 5 retries... Oct 22 19:04:14 preclear_disk_SN_6258: Post-Read: verifying the beginning of the disk. Oct 22 19:04:14 preclear_disk_SN_6258: Post-Read: verifying the rest of the disk. Oct 22 23:26:41 preclear_disk_SN_6258: Post-Read: progress - 25% verified @ 273 MB/s Oct 23 04:13:16 preclear_disk_SN_6258: Post-Read: progress - 50% verified @ 248 MB/s I will get back with updates.
November 3, 20241 yr Author It looks like "Refresh Topology" in the HBA BIOS solved the problem. (https://www.h3c.com/en/d_202207/1655193_294551_0.htm#_Ref533597481) Since that action I have done a lot of heavy writing operations on the system without increase in UDMA CRC Errors on any attached disk. Operations run so far: Rebuild of parity moving 17,5 GB of Data from single disk (new disk6) to Array (Array emulating disk1). preclearing disk1 and adding it to array with rebuild of its contents
December 9, 20241 yr Author I have an update to this problem. The server recently went into throwing lots of those errors on all disks. even leading to read errors on the array and failing preclearing runs. Restarting the server or shutting it down for a couple minutes does not get rid of the errors. "Refresh Topology" did not help either. Switching the server off for a few hours (Power connected and IPMI online) and back on seems to solve the problem until it starts occurring again. Preclears are running without any issues again. I am still at a loss as to what causes the errors to start. For now I am running the server with all fans on max to make sure temperature is not the issue. I will provide updates.
December 9, 20241 yr Community Expert You might want to make sure the HBA us well seated into the MB PCIe slot.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.