New build

4 drives, and I'm getting CRC errors on 3 of 4 drives. No 'Errors' listed on the Main screen but it did drop one disk saying the filesystem was unmountable and made me format it again.

 

Im thinking about yanking the LSI controller and running directly to the onboard SATA as a test but need to get other cables first. Could also be cabling from the LSI to the drives.

 

I'm most concerned about it dropping the one drive and forcing me to reformat it.

Thoughts?

Post up the diagnostics zip file in a new post. 

 

Do try the MB's onboard SATA ports.  Most folks here only use the LSI cards to expand the number of ports when all of the onboard ports are in use. 

 

Cabling is always the prime suspect in cases of CRC errors.  If you order a new SATA cable for your LSI controller, look for one with a cable length of .5M rather than one meter!  (Check that the short cable will be long enough but the half-meter ones will work in 99.9% of all applications and result in much less of 'wire ball' inside of the case.)   Double check that all SATA power and data cables are securely seated.  With only four drives, you should not require any power splitters. 

8 hours ago, MDA said:

but it did drop one disk saying the filesystem was unmountable and made me format it again.

That is not the recommended way to handle a disk showing as unmountable as formatting a drive wipes its contents.   Instead you should follow the steps outlined here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page.

6 hours ago, itimpi said:

That is not the recommended way to handle a disk showing as unmountable as formatting a drive wipes its contents.   Instead you should follow the steps outlined here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page.

Good to know.  Since it was basically empty, it didn't matter much but would if it were loaded.  Thanks for the info. I will review.

Interestingly enough, I copied > 1TB of data overnight to the new array without any CRC errors (no cache pool).  When I was getting the CRC errors yesterday, I was using a share setup with a cache pool. I wouldn't think that would make a difference but an interesting data point.  Later today, I'll do more testing with the cache pool and see what happens.  I also have new cabling arriving today to see if that makes any difference.

It does appear that you do not understand what a CRC error is.  A CRC error occurs when the check sum for a data block that is calculated by a hard drive does not match the checksum that is calculated by the SATA controller when the controller receives that data from that hard drive.  When an error occurs, the data is retransmitted until the check sums match.  (So, basically, it is checking the SATA cable and its connectors for data integrity.)   Most of the time, the only problem caused is that the retransmission can cause slowdowns in the data transfer speed if there are a lot of them.   

 

Oh, one more thing.  The CRC error count is store in flash memory on the hard drive and is not resetable!   That count will stick with that hard drive until it is trashed. 

I switched cabling and restarted.... immediately upon starting the array, one drive spewed CRC errors and other write errors and the disk was taken offline (Red X)

Dec 11 14:39:01 ICENAS kernel: sd 1:0:2:0: [sdd] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB)
Dec 11 14:39:01 ICENAS kernel: sd 1:0:2:0: [sdd] 4096-byte physical blocks
Dec 11 14:39:01 ICENAS kernel: sd 1:0:2:0: [sdd] Write Protect is off
Dec 11 14:39:01 ICENAS kernel: sd 1:0:2:0: [sdd] Mode Sense: 7f 00 10 08
Dec 11 14:39:01 ICENAS kernel: sd 1:0:2:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Dec 11 14:39:01 ICENAS kernel: sdd: sdd1
Dec 11 14:39:01 ICENAS kernel: sd 1:0:2:0: [sdd] Attached SCSI disk
Dec 11 14:39:06 ICENAS  wsdd2[1340]: starting.
Dec 11 14:39:08 ICENAS  emhttpd: ST14000NM001G-2KJ103_ZTM0CE55 (sdd) 512 27344764928
Dec 11 14:39:08 ICENAS kernel: mdcmd (3): import 2 sdd 64 13672382412 0 ST14000NM001G-2KJ103_ZTM0CE55
Dec 11 14:39:08 ICENAS kernel: md: import disk2: (sdd) ST14000NM001G-2KJ103_ZTM0CE55 size: 13672382412 
Dec 11 14:39:08 ICENAS  emhttpd: read SMART /dev/sdd
Dec 11 14:39:08 ICENAS  wsdd2[1340]: 'Terminated' signal received.
Dec 11 14:39:08 ICENAS  wsdd2[1340]: terminating.
Dec 11 14:39:10 ICENAS root:                  /usr/sbin/wsdd2 -d 
Dec 11 14:39:10 ICENAS  wsdd2[1539]: starting.
Dec 11 14:44:41 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3949 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:41 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3949 CDB: opcode=0x88 88 00 00 00 00 06 5d df ff 80 00 00 00 08 00 00
Dec 11 14:44:41 ICENAS kernel: I/O error, dev sdd, sector 27344764800 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Dec 11 14:44:42 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3917 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:42 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3917 CDB: opcode=0x88 88 00 00 00 00 06 5d df fc 70 00 00 00 08 00 00
Dec 11 14:44:42 ICENAS kernel: I/O error, dev sdd, sector 27344764016 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Dec 11 14:44:42 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3933 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:42 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3933 CDB: opcode=0x88 88 00 00 00 00 06 5d df f3 f0 00 00 00 08 00 00
Dec 11 14:44:42 ICENAS kernel: I/O error, dev sdd, sector 27344761840 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Dec 11 14:44:43 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3940 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:43 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3940 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 40 00 00 00 08 00 00
Dec 11 14:44:43 ICENAS kernel: I/O error, dev sdd, sector 64 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Dec 11 14:44:43 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3944 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:43 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3944 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 80 00 00 00 08 00 00
Dec 11 14:44:43 ICENAS kernel: I/O error, dev sdd, sector 128 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3943 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3943 CDB: opcode=0x88 88 00 00 00 00 06 5d df ff c8 00 00 00 28 00 00
Dec 11 14:44:44 ICENAS kernel: I/O error, dev sdd, sector 27344764872 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 0
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3942 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3942 CDB: opcode=0x88 88 00 00 00 00 06 5d df ff 88 00 00 00 38 00 00
Dec 11 14:44:44 ICENAS kernel: I/O error, dev sdd, sector 27344764808 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 0
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3941 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3941 CDB: opcode=0x88 88 00 00 00 00 06 5d df ff 08 00 00 00 78 00 00
Dec 11 14:44:44 ICENAS kernel: I/O error, dev sdd, sector 27344764680 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 0
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3940 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3940 CDB: opcode=0x88 88 00 00 00 00 06 5d df fe 78 00 00 00 80 00 00
Dec 11 14:44:44 ICENAS kernel: I/O error, dev sdd, sector 27344764536 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 0
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3939 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:44:44 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3939 CDB: opcode=0x88 88 00 00 00 00 06 5d df fe 00 00 00 00 70 00 00
Dec 11 14:44:44 ICENAS kernel: I/O error, dev sdd, sector 27344764416 op 0x0:(READ) flags 0x80700 phys_seg 14 prio class 0
Dec 11 14:44:49 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3912 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=2s
Dec 11 14:44:49 ICENAS kernel: sd 1:0:2:0: [sdd] tag#3912 CDB: opcode=0x8a 8a 00 00 00 00 03 00 00 a7 f8 00 00 01 00 00 00
Dec 11 14:44:49 ICENAS kernel: I/O error, dev sdd, sector 12884944888 op 0x1:(WRITE) flags 0x800 phys_seg 32 prio class 0
Dec 11 14:44:50 ICENAS  wsdd2[1539]: 'Terminated' signal received.
Dec 11 14:44:50 ICENAS  wsdd2[1539]: terminating.
Dec 11 14:44:52 ICENAS root:                  /usr/sbin/wsdd2 -d 
Dec 11 14:44:52 ICENAS  wsdd2[3806]: starting.
Dec 11 14:50:34 ICENAS  emhttpd: read SMART /dev/sdd
Dec 11 14:50:42 ICENAS  wsdd2[3806]: 'Terminated' signal received.
Dec 11 14:50:42 ICENAS  wsdd2[3806]: terminating.
Dec 11 14:50:42 ICENAS root:                  /usr/sbin/wsdd2 -d 
Dec 11 14:50:42 ICENAS  wsdd2[10896]: starting.
Dec 11 14:55:45 ICENAS  emhttpd: read SMART /dev/sdd
Dec 11 14:55:45 ICENAS  wsdd2[10896]: 'Terminated' signal received.
Dec 11 14:55:45 ICENAS  wsdd2[10896]: terminating.
Dec 11 14:55:45 ICENAS root:                  /usr/sbin/wsdd2 -d 
Dec 11 14:55:45 ICENAS  wsdd2[16802]: starting.
Dec 11 14:55:56 ICENAS  wsdd2[16802]: 'Terminated' signal received.
Dec 11 14:55:56 ICENAS  wsdd2[16802]: terminating.
Dec 11 14:55:58 ICENAS root:                  /usr/sbin/wsdd2 -d 
Dec 11 14:55:58 ICENAS  wsdd2[17195]: starting.

What was the cable switch that you did???????????

 

And where is your diagnostics file for the current setup?  (The Gurus always want a diagnostics file to look for specific things.)

I swapped out the two mini SAS -> 4x SATA (1 meter) cables for 0.5 meter fan cables (that was before more errors as noted above). That said, I just yanked all that completely and went with 4x straight shielded SATA cables from onboard controller removing the LSI 9207 from the mix. 

No new errors on this restart now... but now I'm trying to follow those instructions to fix the disabled drive without having to rebuild it.

I guess I'm not following the instructions correctly.

 

Disk2 is now disabled and being emulated.  I've checked the filesystem with the -L and output is as follows:
 

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
clearing needsrepair flag and regenerating metadata
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 4
        - agno = 2
        - agno = 11
        - agno = 8
        - agno = 9
        - agno = 1
        - agno = 6
        - agno = 10
        - agno = 12
        - agno = 5
        - agno = 3
        - agno = 7
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Maximum metadata LSN (4:0) is ahead of log (1:2).
Format log to cycle 7.
done

 

Restarting the array in normal mode still shows the disk as disabled. Any ideas?

I am going to ping @JorgeB as he is the real Guru for the procedure(s) to repair disabled disks.   (At this point, I would suspect that you have a bad LSI card which caused all the problems to begin with.)   I believed he is based in Europe so don't except a reply this evening. 

Yes, I too feel the LSI has been at the root of my problems.

Ok. Look forward to what JorgeB says. Thanks

 

From the xfs_repair output the emulated disk2 should now be mounting when the array is started in Normal mode and it’s contents visible.  Is the content what is expected?

 

To clear the disabled state the drive would need rebuilding.    The rebuild process makes the physical drive match the emulated one.

Please post the diagnostics.

I followed the instructions to remove/add the disk back and it rebuilt... all seems functional now and in my testing, no errors presenting.  Seems like the LSI controller was the root of my issues.

 

Thanks all.

Edited December 13, 20223 yr by MDA