jc179 Posted April 10, 2020 Share Posted April 10, 2020 Greetings fellow unraiders: I'm trying to troubleshoot an issue I'm having with poor performance in unraid after migrating to it, but first I noticed that in unraid my Hitachi 8T HDD only negotiate at SATA 1.5 Gb/s, instead of SATA 3.0 Gb/sec as they we previously in slackware 14. Also, while benchmarking the drives, the 8T Hitachi drives, they would not benchmark over 130 meg/sec using "diskspeed addin " or watching on command line with "iostat -m 5". Further checking revealed that they negotiate at Sata 1.5 Gb/s instead as Sata 3.0 Gb/sec when running Slackware 14 on this hardware previously (keep in mind, only change = OS at this point, and boot device). I realize this isn't going to make a large difference either in my overall slow performance issue (see below for more details). Are there any kernel parameters I can pass, or would I need to recompile the kernel using the below "known" good version? I do get this on unraid boot (not slackware), and am not certain if its related to an invalid response receivd back from the disks? All of the other disks attached negotiate at Sata 3.0.. [ 24.694680] mlx4_en 0000:04:00.0: Activating port:2 [ 24.695661] mlx4_en: 0000:04:00.0: Port 2: Using 12 TX rings [ 24.695923] mlx4_en: 0000:04:00.0: Port 2: Using 8 RX rings [ 24.696336] mlx4_en: 0000:04:00.0: Port 2: Initializing port [ 26.927642] mlx4_en: eth1: Link Up [ 26.982588] mlx4_en: eth1: Link Down [ 26.982918] mlx4_en: eth2: Link Up [ 27.137470] mlx4_en: eth1: Link Up \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ [ 27.545855] ata8.00: qc timeout (cmd 0xa1) [ 27.546122] ata8.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 27.546402] ata14.00: qc timeout (cmd 0xa1) [ 27.855354] ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 28.046151] ata14.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 28.856493] ata14: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 38.297869] ata8.00: qc timeout (cmd 0xa1) [ 38.298136] ata8.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 38.608480] ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 39.321861] ata14.00: qc timeout (cmd 0xa1) [ 39.822484] ata14.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 40.632478] ata14: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 69.017833] ata8.00: qc timeout (cmd 0xa1) [ 69.018099] ata8.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 69.328473] ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 71.065879] ata14.00: qc timeout (cmd 0xa1) [ 71.566557] ata14.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 72.376473] ata14: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [ 73.090854] BTRFS: device fsid 7287fbd8-04ca-4508-b947-73813bcf236d devid 1 transid 61 /dev/sdj1 [ 73.118785] BTRFS: device fsid 7287fbd8-04ca-4508-b947-73813bcf236d devid 3 transid 61 /dev/sdl1 [ 73.124181] BTRFS: device fsid 7287fbd8-04ca-4508-b947-73813bcf236d devid 2 transid 61 /dev/sdi1 [ 73.134152] BTRFS: device fsid 7287fbd8-04ca-4508-b947-73813bcf236d devid 4 transid 61 /dev/sdk1 [ 74.668886] sky2: driver version 1.30 [ 74.668998] sky2 0000:06:00.0: Yukon-2 EC Ultra chip revision 3 [ 74.669209] sky2 0000:06:00.0 eth0: addr 20:cf:30:66:b9:a3 [ 74.691230] mlx4_core: Mellanox ConnectX core driver v4.0-0 Under unraid 6.8.3 (4.19.107-Unraid) : root@Tower:~# smartctl --all /dev/sdb smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: HGST HDN728080ALE604 Serial Number: R6GSE49Y LU WWN Device Id: 5 000cca 263caa5db Firmware Version: A4GNW91X User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 1.5 Gb/s) <================= Under slackware 14 (kernel Linux san 4.4.14) root@san:/# smartctl --all /dev/sdb smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.14] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: HGST HDN728080ALE604 Serial Number: R6GSE49Y LU WWN Device Id: 5 000cca 263caa5db Firmware Version: A4GNW91X User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) <================= Local Time is: Fri Apr 10 00:00:08 2020 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled Entire issue: While I'm trying to restore my backup to the disks I freed up for this array, things are going uber slow, about 50-70 mb/sec tops, if I try to do a directory listing, it can take minutes before *anything* comes back! - Drives were pre-cleared, and parity was built - The exact hardware before running Linux raid6 software raid would do 200+ mb/sec over 10 gig for large files. I've run the "fix common problems" and "ca turbo write" but not really any changes. - Spec is Xeon(R) CPU X5650, 24 G ram, LSI SAS2303/ Broadcom / LSI SAS9217-4i4e connected to a SE3016 Rackable array /w the above hitachi disks in it. Thanks for any help Jonathan Quote Link to comment
Vr2Io Posted April 10, 2020 Share Posted April 10, 2020 (edited) Does "SE3016 Rackable" use before and confirm work normal ? I think you need test the hardware from top to bottom, i.e. HBA then SE3016 ( inside expander ) ..... Sure those timeout problem must be fix first. [ 27.545855] ata8.00: qc timeout (cmd 0xa1) Edited April 10, 2020 by Benson Quote Link to comment
jc179 Posted April 10, 2020 Author Share Posted April 10, 2020 9 hours ago, Benson said: Does "SE3016 Rackable" use before and confirm work normal ? I think you need test the hardware from top to bottom, i.e. HBA then SE3016 ( inside expander ) ..... Sure those timeout problem must be fix first. [ 27.545855] ata8.00: qc timeout (cmd 0xa1) Thanks for asking, but Yes, the hardware is 100% reliable, it has been used the last ~4 years by a Linux software raid 6 array of 6 disks. there is no issue /w the controller or SAS expander, zero issues. Note this only happens with unraid 6.8.3 booted - Slackware, and even Ubuntu I tried last night do not show this error, and the drives negotiate at 3.0 Gb/s. I suspect there maybe a bug being caused with another module that's loading or the 2308 driver itsself in the kernel. I have tried recreating unraid 6.8.3, and that does not solve the issue either. Thanks, Jonathan Quote Link to comment
Vr2Io Posted April 10, 2020 Share Posted April 10, 2020 (edited) So weird. I never got problem with LSI 2008 / 2308 + different expander + all Unraid 6.XX ver ( most HDDs are HGST/WD ). Seems you say problem only happen on Hitachi 8T HDD in SE3016, could you try direct connect those problem disk to HBA or onboard ( bypass SE3016 expander ) ? Or Set a wrong controller type for the problem disks I suspect due to Unraid periodic check HDDs SMART data and this cause expander reset self. Edited April 10, 2020 by Benson Quote Link to comment
jc179 Posted April 10, 2020 Author Share Posted April 10, 2020 Certainly odd - managed to fix it by updating the firmware on the LSI Card For anyone else who runs into this Before: Quote ./sas2flash -listall LSI Corporation SAS2 Flash Utility Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2308_2(D1) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr ---------------------------------------------------------------------------- 0 SAS2308_2(D1) 20.00.06.00 14.01.00.06 07.39.02.00 00:05:00:00 Finished Processing Commands Successfully. Exiting SAS2Flash. After; Quote ./sas2flash -listall LSI Corporation SAS2 Flash Utility Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2308_2(D1) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr ---------------------------------------------------------------------------- 0 SAS2308_2(D1) 20.00.07.00 14.01.00.06 07.39.02.00 00:05:00:00 Finished Processing Commands Successfully. Exiting SAS2Flash. The HGST's are now at 3.0 Gb/s Quote root@Tower:~# smartctl --all /dev/sdd | head -15 smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: HGST HDN728080ALE604 Serial Number: R6GSK8NY LU WWN Device Id: 5 000cca 263cab566 Firmware Version: A4GNW91X User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) root@Tower:~# Oddly stil see the ATA messages on startup, suspect that it is the onboard SATA ports causing this, will disable them and see if it clears Thanks for the help/ ideas - Will retest things now! Jonathan Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.