August 18, 20178 yr Community Expert Syslog is filled with this error making much more difficult to analyse: Aug 18 01:39:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339 But, errors on multiple disks would point to cable/power/controller issue, do they share anything? SAS cable, power splitter, etc?
August 18, 20178 yr Author Layout is this currently but has been moving around per things like the Disk4 mentioned above. Disk4 used to be in the top row and worked for weeks (with the SMC controllers) and Disks1-3 were in Row2 directly above current positions. Moved them after the meltdown a few weeks ago and current placement seemed to help. Case is a thing like a Norco-4224 but not branded that. Uses individual SATA connectors per drive though and, not a SFF-8087 so I use 4x breakouts to attach to controller ports. MB is a Crosshair V Formula-Z, CPU is AMD 8370 - RAM is 32GB DDR3. PSU is Corsair HX850i platinum. row 1: [Cache1 (250GB WD SSD)] - [ empty ** ] - [Disk5 4TB Seagate ] - [Disk6 4TB Hitachi ] row 2: [ "Spare" 8TB WD * ] - [ empty ] - [Disk4 4TB Seagate**] - [ empty ] row 3: [Cache2 (250GB WD SSD)] - [Disk1 8TB WD ] - [Disk2 8TB WD ] - [Disk3 8TB WD ] row 4: [Parity1 8TB WD ] - [Disk7 4TB Hitachi ] - [ empty ] - [ empty ] row 5: [ 4TB ESXi datastore ] - [ empty ] - [ empty ] - [ empty ] row 6: [ 250GB SSD ESXi boot ] - [ 1TB Seagate ESXi ds] - [ empty ] - [ empty ] * Has not been a disk here until this rebuild attempt. This drive was the replacement Disk1 that also "failed". Was preserving it incase it was the best recovery option but seems the filesystem is trashed as xfs_repair -vn returned no Superblock and scanned the whole disk (all last night) to not find the backup superblock. ** Slot the 4TB Disk4 was misbehaving in but had previously been in since build. Row 1 to 4 run off 9201-16i (previously the two SMC controllers). Row 5 & 6 run off the Motherboard SATA ports. Never been a problem. Tends me not to suspect power supply but power delivery is another matter. One molex connector feeds each row. Measures +11.94v on 12V and 4.69v on the 5V which worries me. Might try and engineer some better power feeds over the weekend or try it all in another box. The chassis, cables, (old controllers) a similar Motherboard CPU and RAM config and more overall drives (4TB largest) ran on unRaid5.0 for the past 3 years. 2 of that on the same PSU. 3x WD 8TB should be less power than 2x 4TB Seagate by my calc so power should be improved over old arrangement.
August 18, 20178 yr Author 1 hour ago, johnnie.black said: Syslog is filled with this error making much more difficult to analyse: Aug 18 01:39:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339 But, errors on multiple disks would point to cable/power/controller issue, do they share anything? SAS cable, power splitter, etc? Yes that is irritating and seems the log managed to miss the events of the screen capture entirely. Was wondering with the IO counts though if something had had a overflow error or similar between card firmware (circa 2014) and the "latest" drive tech? ps. What is that error and how do I get rid of it? Seems to come up every few seconds. Current error log is this though. Some sort of hissy-fit trying to spin down disk2 and disk3 which are absent having popped up as sdm and sdn as well as sdj and sdi. Should that even be able to happen? What would make that occur? ErrorWarningSystemArrayLogin Aug 19 01:32:24 zStore2 kernel: md: do_drive_cmd: lock_bdev error: -2Aug 19 01:32:24 zStore2 kernel: mdcmd (17395): spindown 3Aug 19 01:32:24 zStore2 kernel: md: do_drive_cmd: lock_bdev error: -2Aug 19 01:32:25 zStore2 emhttp: err: mdcmd: write: No such file or directoryAug 19 01:32:25 zStore2 emhttp: err: mdcmd: write: No such file or directoryAug 19 01:32:25 zStore2 kernel: mdcmd (17396): spindown 2Aug 19 01:32:25 zStore2 kernel: md: do_drive_cmd: lock_bdev error: -2Aug 19 01:32:25 zStore2 kernel: mdcmd (17397): spindown 3Aug 19 01:32:25 zStore2 kernel: md: do_drive_cmd: lock_bdev error: -2Aug 19 01:32:26 zStore2 emhttp: err: mdcmd: write: No such file or directoryAug 19 01:32:26 zStore2 emhttp: err: mdcmd: write: No such file or directoryAug 19 01:32:26 zStore2 kernel: mdcmd (17398): spindown 2
August 18, 20178 yr Community Expert 1 minute ago, fatpipe said: Was wondering with the IO counts though if something had had a overflow error or similar between card firmware (circa 2014) and the "latest" drive tech? The high number of errors is normal when disks drop offline 1 minute ago, fatpipe said: ps. What is that error and how do I get rid of it? Almost certainly some plugin, but don't know which one. 2 minutes ago, fatpipe said: Current error log is this though. Some sort of hissy-fit trying to spin down disk2 and disk3 which are absent having popped up as sdm and sdn as well as sdj and sdi. Should that even be able to happen? What would make that occur? Reboot for the disks to get online.
August 18, 20178 yr Author OK. Reset and in compromise mode with disk1 set to no device and removes the spare disk. This is the log for all that. What is the Floppy error about? There is no floppy drive... Aug 19 01:47:26 zStore2 kernel: mdcmd (13): import 12Aug 19 01:47:26 zStore2 emhttp: import 30 cache device: sdaAug 19 01:47:26 zStore2 emhttp: import 31 cache device: sdfAug 19 01:47:26 zStore2 emhttp: check_pool: /sbin/btrfs filesystem show 0a739b76-085d-4697-be3b-bffb4c26ef19 2>&1Aug 19 01:47:26 zStore2 kernel: mdcmd (14): import 13Aug 19 01:47:26 zStore2 kernel: mdcmd (15): import 14Aug 19 01:47:26 zStore2 kernel: mdcmd (16): import 15Aug 19 01:47:26 zStore2 kernel: mdcmd (17): import 16Aug 19 01:47:26 zStore2 kernel: mdcmd (18): import 17Aug 19 01:47:26 zStore2 kernel: mdcmd (19): import 18Aug 19 01:47:26 zStore2 kernel: mdcmd (20): import 19Aug 19 01:47:26 zStore2 kernel: mdcmd (21): import 20Aug 19 01:47:26 zStore2 kernel: mdcmd (22): import 21Aug 19 01:47:26 zStore2 kernel: mdcmd (23): import 22Aug 19 01:47:26 zStore2 kernel: mdcmd (24): import 23Aug 19 01:47:26 zStore2 kernel: mdcmd (25): import 24Aug 19 01:47:26 zStore2 kernel: mdcmd (26): import 25Aug 19 01:47:26 zStore2 kernel: mdcmd (27): import 26Aug 19 01:47:26 zStore2 kernel: mdcmd (28): import 27Aug 19 01:47:26 zStore2 kernel: mdcmd (29): import 28Aug 19 01:47:26 zStore2 kernel: mdcmd (30): import 29Aug 19 01:47:26 zStore2 kernel: md: import_slot: 29 emptyAug 19 01:47:26 zStore2 kernel: blk_update_request: I/O error, dev fd0, sector 0Aug 19 01:47:26 zStore2 kernel: floppy: error -5 while reading block 0Aug 19 01:47:26 zStore2 emhttp: cacheUUID: 0a739b76-085d-4697-be3b-bffb4c26ef19Aug 19 01:47:26 zStore2 emhttp: cacheNumDevices: 2Aug 19 01:47:26 zStore2 emhttp: cacheTotDevices: 2Aug 19 01:47:26 zStore2 emhttp: cacheNumMissing: 0Aug 19 01:47:26 zStore2 emhttp: cacheNumMisplaced: 0Aug 19 01:47:26 zStore2 emhttp: cacheNumExtra: 0Aug 19 01:47:26 zStore2 emhttp: import flash device: sdeAug 19 01:47:33 zStore2 emhttp: shcmd (1863): rmmod md-mod |& loggerAug 19 01:47:33 zStore2 kernel: md: unRAID driver removedAug 19 01:47:33 zStore2 emhttp: shcmd (1864): modprobe md-mod super=/boot/config/super.dat |& loggerAug 19 01:47:33 zStore2 kernel: md: unRAID driver 2.7.2 installedAug 19 01:47:33 zStore2 emhttp: Pro key detected, GUID: 8644-800A-0000-000000000695 FILE: /boot/config/Pro.keyAug 19 01:47:33 zStore2 emhttp: Device inventory:Aug 19 01:47:33 zStore2 emhttp: shcmd (1865): udevadm settleAug 19 01:47:33 zStore2 emhttp: WDC_WD80EFZX-68UW8N0_VK0GSJTY (sdj) 7814026532Aug 19 01:47:33 zStore2 emhttp: WDC_WD80EFZX-68UW8N0_VK0GPUKY (sdk) 7814026532Aug 19 01:47:33 zStore2 emhttp: HGST_HMS5C4040ALE640_PL2331LAGU95WJ (sdg) 3907018532Aug 19 01:47:33 zStore2 emhttp: ST4000VN000-1H4168_S300NGS6 (sdh) 3907018532Aug 19 01:47:33 zStore2 emhttp: ST4000VN000-1H4168_S300PW7F (sdd) 3907018532Aug 19 01:47:33 zStore2 emhttp: General_USB_Flash_Disk_0000000000000695-0:0 (sde) 1956832Aug 19 01:47:33 zStore2 emhttp: WDC_WD80EFZX-68UW8N0_VK1745LY (sdb) 7814026532Aug 19 01:47:33 zStore2 emhttp: WDC_WDS250G1B0A-00H9H0_170966800186 (sdf) 244198552Aug 19 01:47:33 zStore2 emhttp: HGST_HMS5C4040ALE640_PL2331LAGU9JZJ (sdc) 3907018532Aug 19 01:47:33 zStore2 emhttp: WDC_WD80EFZX-68UW8N0_VK17BTEY (sdi) 7814026532Aug 19 01:47:33 zStore2 emhttp: WDC_WDS250G1B0A-00H9H0_170966801067 (sda) 244198552Aug 19 01:47:33 zStore2 kernel: mdcmd (1): import 0 sdb 7814026532 0 WDC_WD80EFZX-68UW8N0_VK1745LYAug 19 01:47:33 zStore2 kernel: md: import disk0: (sdb) WDC_WD80EFZX-68UW8N0_VK1745LY size: 7814026532 Aug 19 01:47:33 zStore2 kernel: mdcmd (2): import 1Aug 19 01:47:33 zStore2 kernel: md: import_slot: 1 emptyAug 19 01:47:33 zStore2 kernel: mdcmd (3): import 2 sdj 7814026532 0 WDC_WD80EFZX-68UW8N0_VK0GSJTYAug 19 01:47:33 zStore2 kernel: md: import disk2: (sdj) WDC_WD80EFZX-68UW8N0_VK0GSJTY size: 7814026532 Aug 19 01:47:33 zStore2 kernel: mdcmd (4): import 3 sdk 7814026532 0 WDC_WD80EFZX-68UW8N0_VK0GPUKYAug 19 01:47:33 zStore2 kernel: md: import disk3: (sdk) WDC_WD80EFZX-68UW8N0_VK0GPUKY size: 7814026532 Aug 19 01:47:33 zStore2 kernel: mdcmd (5): import 4 sdh 3907018532 0 ST4000VN000-1H4168_S300NGS6Aug 19 01:47:33 zStore2 kernel: md: import disk4: (sdh) ST4000VN000-1H4168_S300NGS6 size: 3907018532 Aug 19 01:47:33 zStore2 kernel: mdcmd (6): import 5 sdd 3907018532 0 ST4000VN000-1H4168_S300PW7FAug 19 01:47:33 zStore2 kernel: md: import disk5: (sdd) ST4000VN000-1H4168_S300PW7F size: 3907018532 Aug 19 01:47:33 zStore2 kernel: mdcmd (7): import 6 sdc 3907018532 0 HGST_HMS5C4040ALE640_PL2331LAGU9JZJAug 19 01:47:33 zStore2 kernel: md: import disk6: (sdc) HGST_HMS5C4040ALE640_PL2331LAGU9JZJ size: 3907018532 Aug 19 01:47:33 zStore2 kernel: mdcmd (8): import 7 sdg 3907018532 0 HGST_HMS5C4040ALE640_PL2331LAGU95WJAug 19 01:47:33 zStore2 kernel: md: import disk7: (sdg) HGST_HMS5C4040ALE640_PL2331LAGU95WJ size: 3907018532 Aug 19 01:47:33 zStore2 kernel: mdcmd (9): import 8Aug 19 01:47:33 zStore2 kernel: mdcmd (10): import 9Aug 19 01:47:33 zStore2 kernel: mdcmd (11): import 10Aug 19 01:47:33 zStore2 kernel: mdcmd (12): import 11Aug 19 01:47:33 zStore2 kernel: mdcmd (13): import 12Aug 19 01:47:33 zStore2 kernel: mdcmd (14): import 13Aug 19 01:47:33 zStore2 kernel: mdcmd (15): import 14Aug 19 01:47:33 zStore2 kernel: mdcmd (16): import 15Aug 19 01:47:33 zStore2 kernel: mdcmd (17): import 16Aug 19 01:47:33 zStore2 kernel: mdcmd (18): import 17Aug 19 01:47:33 zStore2 kernel: mdcmd (19): import 18Aug 19 01:47:33 zStore2 kernel: mdcmd (20): import 19Aug 19 01:47:33 zStore2 kernel: mdcmd (21): import 20Aug 19 01:47:33 zStore2 kernel: mdcmd (22): import 21Aug 19 01:47:33 zStore2 kernel: mdcmd (23): import 22Aug 19 01:47:33 zStore2 kernel: mdcmd (24): import 23Aug 19 01:47:33 zStore2 kernel: mdcmd (25): import 24Aug 19 01:47:33 zStore2 kernel: mdcmd (26): import 25Aug 19 01:47:33 zStore2 kernel: mdcmd (27): import 26Aug 19 01:47:33 zStore2 kernel: mdcmd (28): import 27Aug 19 01:47:33 zStore2 kernel: mdcmd (29): import 28Aug 19 01:47:33 zStore2 kernel: mdcmd (30): import 29Aug 19 01:47:33 zStore2 kernel: md: import_slot: 29 emptyAug 19 01:47:33 zStore2 emhttp: import 30 cache device: sdaAug 19 01:47:33 zStore2 emhttp: import 31 cache device: sdfAug 19 01:47:33 zStore2 emhttp: check_pool: /sbin/btrfs filesystem show 0a739b76-085d-4697-be3b-bffb4c26ef19 2>&1Aug 19 01:47:33 zStore2 kernel: blk_update_request: I/O error, dev fd0, sector 0Aug 19 01:47:33 zStore2 kernel: floppy: error -5 while reading block 0Aug 19 01:47:33 zStore2 emhttp: cacheUUID: 0a739b76-085d-4697-be3b-bffb4c26ef19Aug 19 01:47:33 zStore2 emhttp: cacheNumDevices: 2Aug 19 01:47:33 zStore2 emhttp: cacheTotDevices: 2Aug 19 01:47:33 zStore2 emhttp: cacheNumMissing: 0Aug 19 01:47:33 zStore2 emhttp: cacheNumMisplaced: 0Aug 19 01:47:33 zStore2 emhttp: cacheNumExtra: 0Aug 19 01:47:33 zStore2 emhttp: import flash device: sdeAug 19 01:47:40 zStore2 emhttp: shcmd (1875): /usr/local/sbin/set_ncq sdb 1 &> /dev/nullAug 19 01:47:40 zStore2 kernel: mdcmd (31): set md_num_stripes 1280Aug 19 01:47:40 zStore2 kernel: mdcmd (32): set md_sync_window 384Aug 19 01:47:40 zStore2 kernel: mdcmd (33): set md_sync_thresh 192Aug 19 01:47:40 zStore2 kernel: mdcmd (34): set md_write_methodAug 19 01:47:40 zStore2 kernel: mdcmd (35): set spinup_group 0 0Aug 19 01:47:40 zStore2 kernel: mdcmd (36): set spinup_group 2 0Aug 19 01:47:40 zStore2 kernel: mdcmd (37): set spinup_group 3 0Aug 19 01:47:40 zStore2 kernel: mdcmd (38): set spinup_group 4 0Aug 19 01:47:40 zStore2 kernel: mdcmd (39): set spinup_group 5 0Aug 19 01:47:40 zStore2 kernel: mdcmd (40): set spinup_group 6 0Aug 19 01:47:40 zStore2 kernel: mdcmd (41): set spinup_group 7 0Aug 19 01:47:40 zStore2 emhttp: shcmd (1876): echo 128 > /sys/block/sdb/queue/nr_requestsAug 19 01:47:40 zStore2 emhttp: shcmd (1877): /usr/local/sbin/set_ncq sdj 1 &> /dev/nullAug 19 01:47:40 zStore2 emhttp: shcmd (1878): echo 128 > /sys/block/sdj/queue/nr_requestsAug 19 01:47:40 zStore2 emhttp: shcmd (1879): /usr/local/sbin/set_ncq sdk 1 &> /dev/nullAug 19 01:47:40 zStore2 emhttp: shcmd (1880): echo 128 > /sys/block/sdk/queue/nr_requestsAug 19 01:47:40 zStore2 emhttp: shcmd (1881): /usr/local/sbin/set_ncq sdh 1 &> /dev/nullAug 19 01:47:40 zStore2 emhttp: shcmd (1882): echo 128 > /sys/block/sdh/queue/nr_requestsAug 19 01:47:40 zStore2 emhttp: shcmd (1883): /usr/local/sbin/set_ncq sdd 1 &> /dev/nullAug 19 01:47:40 zStore2 emhttp: shcmd (1884): echo 128 > /sys/block/sdd/queue/nr_requestsAug 19 01:47:40 zStore2 emhttp: shcmd (1885): /usr/local/sbin/set_ncq sdc 1 &> /dev/nullAug 19 01:47:40 zStore2 emhttp: shcmd (1886): echo 128 > /sys/block/sdc/queue/nr_requestsAug 19 01:47:40 zStore2 emhttp: shcmd (1887): /usr/local/sbin/set_ncq sdg 1 &> /dev/nullAug 19 01:47:40 zStore2 emhttp: shcmd (1888): echo 128 > /sys/block/sdg/queue/nr_requestsAug 19 01:47:40 zStore2 kernel: mdcmd (42): start STOPPEDAug 19 01:47:40 zStore2 kernel: unraid: allocating 46460K for 1280 stripes (9 disks)Aug 19 01:47:40 zStore2 kernel: md1: running, size: 7814026532 blocksAug 19 01:47:40 zStore2 kernel: md2: running, size: 7814026532 blocksAug 19 01:47:40 zStore2 kernel: md3: running, size: 7814026532 blocksAug 19 01:47:40 zStore2 kernel: md4: running, size: 3907018532 blocksAug 19 01:47:40 zStore2 kernel: md5: running, size: 3907018532 blocksAug 19 01:47:40 zStore2 kernel: md6: running, size: 3907018532 blocksAug 19 01:47:40 zStore2 kernel: md7: running, size: 3907018532 blocksAug 19 01:47:40 zStore2 emhttp: shcmd (1889): udevadm settleAug 19 01:47:40 zStore2 emhttp: Mounting disks...Aug 19 01:47:40 zStore2 emhttp: shcmd (1890): /sbin/btrfs device scan |& loggerAug 19 01:47:40 zStore2 kernel: blk_update_request: I/O error, dev fd0, sector 0Aug 19 01:47:40 zStore2 kernel: floppy: error -5 while reading block 0Aug 19 01:47:40 zStore2 root: Scanning for Btrfs filesystemsAug 19 01:47:40 zStore2 emhttp: shcmd (1891): mkdir -p /mnt/disk1Aug 19 01:47:40 zStore2 emhttp: shcmd (1892): set -o pipefail ; mount -t xfs -o noatime,nodiratime /dev/md1 /mnt/disk1 |& loggerAug 19 01:47:40 zStore2 kernel: XFS (md1): Mounting V5 FilesystemAug 19 01:47:40 zStore2 kernel: XFS (md1): Starting recovery (logdev: internal)Aug 19 01:47:41 zStore2 kernel: XFS (md1): Ending recovery (logdev: internal)Aug 19 01:47:41 zStore2 emhttp: shcmd (1893): xfs_growfs /mnt/disk1 |& loggerAug 19 01:47:41 zStore2 root: meta-data=/dev/md1 isize=512 agcount=8, agsize=268435455 blksAug 19 01:47:41 zStore2 root: = sectsz=512 attr=2, projid32bit=1Aug 19 01:47:41 zStore2 root: = crc=1 finobt=1 spinodes=0Aug 19 01:47:41 zStore2 root: data = bsize=4096 blocks=1953506633, imaxpct=5Aug 19 01:47:41 zStore2 root: = sunit=0 swidth=0 blksAug 19 01:47:41 zStore2 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1Aug 19 01:47:41 zStore2 root: log =internal bsize=4096 blocks=521728, version=2Aug 19 01:47:41 zStore2 root: = sectsz=512 sunit=0 blks, lazy-count=1Aug 19 01:47:41 zStore2 root: realtime =none extsz=4096 blocks=0, rtextents=0Aug 19 01:47:41 zStore2 emhttp: shcmd (1894): mkdir -p /mnt/disk2Aug 19 01:47:41 zStore2 emhttp: shcmd (1895): set -o pipefail ; mount -t xfs -o noatime,nodiratime /dev/md2 /mnt/disk2 |& loggerAug 19 01:47:41 zStore2 kernel: XFS (md2): Mounting V5 FilesystemAug 19 01:47:41 zStore2 kernel: XFS (md2): Ending clean mountAug 19 01:47:41 zStore2 emhttp: shcmd (1896): xfs_growfs /mnt/disk2 |& loggerAug 19 01:47:41 zStore2 root: meta-data=/dev/md2 isize=512 agcount=8, agsize=268435455 blksAug 19 01:47:41 zStore2 root: = sectsz=512 attr=2, projid32bit=1Aug 19 01:47:41 zStore2 root: = crc=1 finobt=1 spinodes=0Aug 19 01:47:41 zStore2 root: data = bsize=4096 blocks=1953506633, imaxpct=5Aug 19 01:47:41 zStore2 root: = sunit=0 swidth=0 blksAug 19 01:47:41 zStore2 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1Aug 19 01:47:41 zStore2 root: log =internal bsize=4096 blocks=521728, version=2Aug 19 01:47:41 zStore2 root: = sectsz=512 sunit=0 blks, lazy-count=1Aug 19 01:47:41 zStore2 root: realtime =none extsz=4096 blocks=0, rtextents=0Aug 19 01:47:41 zStore2 emhttp: shcmd (1897): mkdir -p /mnt/disk3Aug 19 01:47:41 zStore2 emhttp: shcmd (1898): set -o pipefail ; mount -t xfs -o noatime,nodiratime /dev/md3 /mnt/disk3 |& loggerAug 19 01:47:41 zStore2 kernel: XFS (md3): Mounting V5 FilesystemAug 19 01:47:41 zStore2 kernel: XFS (md3): Starting recovery (logdev: internal)Aug 19 01:47:41 zStore2 kernel: XFS (md3): Ending recovery (logdev: internal)Aug 19 01:47:41 zStore2 emhttp: shcmd (1899): xfs_growfs /mnt/disk3 |& loggerAug 19 01:47:41 zStore2 root: meta-data=/dev/md3 isize=512 agcount=8, agsize=268435455 blksAug 19 01:47:41 zStore2 root: = sectsz=512 attr=2, projid32bit=1Aug 19 01:47:41 zStore2 root: = crc=1 finobt=1 spinodes=0Aug 19 01:47:41 zStore2 root: data = bsize=4096 blocks=1953506633, imaxpct=5Aug 19 01:47:41 zStore2 root: = sunit=0 swidth=0 blksAug 19 01:47:41 zStore2 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1Aug 19 01:47:41 zStore2 root: log =internal bsize=4096 blocks=521728, version=2Aug 19 01:47:41 zStore2 root: = sectsz=512 sunit=0 blks, lazy-count=1Aug 19 01:47:41 zStore2 root: realtime =none extsz=4096 blocks=0, rtextents=0Aug 19 01:47:41 zStore2 emhttp: shcmd (1900): mkdir -p /mnt/disk4Aug 19 01:47:41 zStore2 emhttp: shcmd (1901): set -o pipefail ; mount -t xfs -o noatime,nodiratime /dev/md4 /mnt/disk4 |& loggerAug 19 01:47:41 zStore2 kernel: XFS (md4): Mounting V5 FilesystemAug 19 01:47:41 zStore2 kernel: XFS (md4): Ending clean mountAug 19 01:47:41 zStore2 emhttp: shcmd (1902): xfs_growfs /mnt/disk4 |& loggerAug 19 01:47:41 zStore2 root: meta-data=/dev/md4 isize=512 agcount=4, agsize=244188659 blksAug 19 01:47:41 zStore2 root: = sectsz=512 attr=2, projid32bit=1Aug 19 01:47:41 zStore2 root: = crc=1 finobt=1 spinodes=0Aug 19 01:47:41 zStore2 root: data = bsize=4096 blocks=976754633, imaxpct=5Aug 19 01:47:41 zStore2 root: = sunit=0 swidth=0 blksAug 19 01:47:41 zStore2 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1Aug 19 01:47:41 zStore2 root: log =internal bsize=4096 blocks=476930, version=2Aug 19 01:47:41 zStore2 root: = sectsz=512 sunit=0 blks, lazy-count=1Aug 19 01:47:41 zStore2 root: realtime =none extsz=4096 blocks=0, rtextents=0Aug 19 01:47:41 zStore2 emhttp: shcmd (1903): mkdir -p /mnt/disk5Aug 19 01:47:41 zStore2 emhttp: shcmd (1904): set -o pipefail ; mount -t xfs -o noatime,nodiratime /dev/md5 /mnt/disk5 |& loggerAug 19 01:47:41 zStore2 kernel: XFS (md5): Mounting V5 FilesystemAug 19 01:47:41 zStore2 kernel: XFS (md5): Ending clean mountAug 19 01:47:41 zStore2 emhttp: shcmd (1905): xfs_growfs /mnt/disk5 |& loggerAug 19 01:47:41 zStore2 root: meta-data=/dev/md5 isize=512 agcount=4, agsize=244188659 blksAug 19 01:47:41 zStore2 root: = sectsz=512 attr=2, projid32bit=1Aug 19 01:47:41 zStore2 root: = crc=1 finobt=1 spinodes=0Aug 19 01:47:41 zStore2 root: data = bsize=4096 blocks=976754633, imaxpct=5Aug 19 01:47:41 zStore2 root: = sunit=0 swidth=0 blksAug 19 01:47:41 zStore2 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1Aug 19 01:47:41 zStore2 root: log =internal bsize=4096 blocks=476930, version=2Aug 19 01:47:41 zStore2 root: = sectsz=512 sunit=0 blks, lazy-count=1Aug 19 01:47:41 zStore2 root: realtime =none extsz=4096 blocks=0, rtextents=0Aug 19 01:47:41 zStore2 emhttp: shcmd (1906): mkdir -p /mnt/disk6Aug 19 01:47:41 zStore2 emhttp: shcmd (1907): set -o pipefail ; mount -t xfs -o noatime,nodiratime /dev/md6 /mnt/disk6 |& loggerAug 19 01:47:41 zStore2 kernel: XFS (md6): Mounting V5 FilesystemAug 19 01:47:42 zStore2 kernel: XFS (md6): Ending clean mountAug 19 01:47:42 zStore2 emhttp: shcmd (1908): xfs_growfs /mnt/disk6 |& loggerAug 19 01:47:42 zStore2 root: meta-data=/dev/md6 isize=512 agcount=4, agsize=244188659 blksAug 19 01:47:42 zStore2 root: = sectsz=512 attr=2, projid32bit=1Aug 19 01:47:42 zStore2 root: = crc=1 finobt=1 spinodes=0Aug 19 01:47:42 zStore2 root: data = bsize=4096 blocks=976754633, imaxpct=5Aug 19 01:47:42 zStore2 root: = sunit=0 swidth=0 blksAug 19 01:47:42 zStore2 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1Aug 19 01:47:42 zStore2 root: log =internal bsize=4096 blocks=476930, version=2Aug 19 01:47:42 zStore2 root: = sectsz=512 sunit=0 blks, lazy-count=1Aug 19 01:47:42 zStore2 root: realtime =none extsz=4096 blocks=0, rtextents=0Aug 19 01:47:42 zStore2 emhttp: shcmd (1909): mkdir -p /mnt/disk7Aug 19 01:47:42 zStore2 emhttp: shcmd (1910): set -o pipefail ; mount -t xfs -o noatime,nodiratime /dev/md7 /mnt/disk7 |& loggerAug 19 01:47:42 zStore2 kernel: XFS (md7): Mounting V5 FilesystemAug 19 01:47:43 zStore2 kernel: XFS (md7): Ending clean mountAug 19 01:47:43 zStore2 emhttp: shcmd (1911): xfs_growfs /mnt/disk7 |& loggerAug 19 01:47:43 zStore2 root: meta-data=/dev/md7 isize=512 agcount=4, agsize=244188659 blksAug 19 01:47:43 zStore2 root: = sectsz=512 attr=2, projid32bit=1Aug 19 01:47:43 zStore2 root: = crc=1 finobt=1 spinodes=0Aug 19 01:47:43 zStore2 root: data = bsize=4096 blocks=976754633, imaxpct=5Aug 19 01:47:43 zStore2 root: = sunit=0 swidth=0 blksAug 19 01:47:43 zStore2 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1Aug 19 01:47:43 zStore2 root: log =internal bsize=4096 blocks=476930, version=2Aug 19 01:47:43 zStore2 root: = sectsz=512 sunit=0 blks, lazy-count=1Aug 19 01:47:43 zStore2 root: realtime =none extsz=4096 blocks=0, rtextents=0Aug 19 01:47:43 zStore2 emhttp: shcmd (1912): mkdir -p /mnt/cacheAug 19 01:47:43 zStore2 emhttp: shcmd (1913): set -o pipefail ; mount -t btrfs -o noatime,nodiratime -U 0a739b76-085d-4697-be3b-bffb4c26ef19 /mnt/cache |& loggerAug 19 01:47:43 zStore2 kernel: BTRFS info (device sda1): disk space caching is enabledAug 19 01:47:43 zStore2 kernel: BTRFS info (device sda1): has skinny extentsAug 19 01:47:43 zStore2 kernel: BTRFS info (device sda1): bdev /dev/sda1 errs: wr 8139715, rd 5920440, flush 166432, corrupt 0, gen 0Aug 19 01:47:43 zStore2 kernel: BTRFS error (device sda1): bad tree block start 0 1983463424Aug 19 01:47:43 zStore2 kernel: BTRFS info (device sda1): detected SSD devices, enabling SSD modeAug 19 01:47:43 zStore2 emhttp: shcmd (1914): syncAug 19 01:47:44 zStore2 unassigned.devices: Mounting Devices...Aug 19 01:47:44 zStore2 emhttp: Starting services...Aug 19 01:47:45 zStore2 emhttp: nothing to syncAug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:47:55 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339Aug 19 01:48:26 zStore2 rc.diskinfo[6147]: PHP Warning: strpos(): Empty needle in /etc/rc.d/rc.diskinfo on line 339
August 18, 20178 yr Community Expert Floppy errors mean there's something in your bios enable indicating there's one, but it's not a problem other than syslog pollution.
August 18, 20178 yr Author OK. Thanks. Will look at that when I am rebooting some time. Any comment on the post of the layout and hardware earlier? Any thoughts on the controller, cables (red breakouts) power supply etc?
August 18, 20178 yr Community Expert All 3 disks with errors are on the same row, I'm not familiar with Norco chassis but you mentioned each row shares a molex connector, if true this is where I would start, swap the connector or move all disks to a different row.
August 18, 20178 yr Author Just lost 8TB. All the shares were offline despite the array being online and was going to stop the array. Screen lag meant I hit the parity button as the screen was scrolling and now it is doing a parity resync with disk1 out of the array.
August 18, 20178 yr Author Yeah may have been to hasty there. Thought it was starting a parity rebuild. But why the shares not online if the array is up and parity is synced? Shares config is missing... Edited August 18, 20178 yr by fatpipe
August 18, 20178 yr Community Expert Probably filesystem corruption, but can only guess without the diags.
August 18, 20178 yr Author Parity check is at 2%. Should I let it continue? Will that prove anything useful? Will take 16hrs. Also for what it is worth, read operations seem to be quite benign but anytime there is wholesale writes going on everything sh1ts itself. And that was supposed to be fixed by the new controller. Understandable if the SATA controller driver is constantly crashing but I don't think that is happening anymore (shouldn't be with the LSI controller).
August 18, 20178 yr Community Expert 16 minutes ago, fatpipe said: Parity check is at 2%. Should I let it continue? Will that prove anything useful? If you swapped the row cable or disks yes, if not don't see the point.
August 18, 20178 yr Author Surely 144MB per second of read on 7 disks is as much of a test as reading 8 disks and writing to one (rebuild) at the same rate? The power and cabling demands ought to be similar? What is stressed by writing that is not stressed by reading? I figured the old controller problems made sense with the virtualization issues raised and crashing causing jumbled writes and general chaos but the controller replacement should have eliminated all that. Appart from the controller crashes in the log, this all seems very similar to before. Which implies it was not the controllers. And I don't believe it was the disks unless they are all faulty. Can happen of course but they seem healthy still and don't have SMART errors and the original one passed a pre-clear no drama (and on an old controller). What sort of issues does one get with non-single 12v PSU? Anything like this? Trouble with that theory is largely the same layout worked with v5 for years on more power hungry drives. 4TB and 2TB disks mostly but no cache. Anyhow, thanks for help. Super late now. Have to retire.
August 19, 20178 yr Author Hi Forum, Today I have the array online again after stopping the parity check after 46% and some 9hrs with 0 corrections, stopping the array and running xfs_repair -vn on each disk. No errors reported. Then starting the array still with disk1 not installed. Array online and shares available. Turning attention to the cache state, I have run the below command with the reported results. Does it look like it will correct all the cache errors if I proceed to that (uncorrectable errors: 0)? That is an awful lot of errors in that and implies about 95% corruption. root@zStore2:/dev# btrfs scrub start -rdB /dev/sda1 scrub device /dev/sda1 (id 1) done scrub started at Sat Aug 19 13:01:35 2017 and finished after 00:13:44 total bytes scrubbed: 209.07GiB with 24488756 errors error details: verify=6960 csum=24481796 corrected errors: 0, uncorrectable errors: 0, unverified errors: 0 root@zStore2:/dev# For the other cache device, I get root@zStore2:/dev# btrfs scrub start -rdB /dev/sdf1 scrub device /dev/sdf1 (id 2) done scrub started at Sat Aug 19 13:34:57 2017 and finished after 00:06:39 total bytes scrubbed: 209.07GiB with 0 errors root@zStore2:/dev# Should I just pull Cache out and run mover with Cache 2 only to recover? If the -r switch was in both commands, why has it clocked up some 5000+ writes on Cache and 1360 on Cache2 during these passes?
August 19, 20178 yr Community Expert Scrub the pool and if all errors are corrected you're fine, though what caused those errors may still be present, so run another scrub soon and monitor dev stats:btrfs dev stats /mnt/cache
August 19, 20178 yr Author Hi Forum, So btrfs recovery seems to have gone well. Here are the outputs for the runs of btrfs scrub. root@zStore2:/dev# btrfs scrub start -rdB /dev/sda1 scrub device /dev/sda1 (id 1) done scrub started at Sat Aug 19 13:01:35 2017 and finished after 00:13:44 total bytes scrubbed: 209.07GiB with 24488756 errors error details: verify=6960 csum=24481796 corrected errors: 0, uncorrectable errors: 0, unverified errors: 0 root@zStore2:/dev# btrfs scrub start -rdB /dev/sdf1 scrub device /dev/sdf1 (id 2) done scrub started at Sat Aug 19 13:34:57 2017 and finished after 00:06:39 total bytes scrubbed: 209.07GiB with 0 errors root@zStore2:/dev# root@zStore2:/dev# root@zStore2:/dev# btrfs scrub start -rdB /dev/sda1 scrub device /dev/sda1 (id 1) done scrub started at Sat Aug 19 13:47:18 2017 and finished after 00:13:43 total bytes scrubbed: 209.07GiB with 24487816 errors error details: verify=6105 csum=24481711 corrected errors: 0, uncorrectable errors: 0, unverified errors: 0 root@zStore2:/dev# btrfs scrub start -dB /dev/sda1 scrub device /dev/sda1 (id 1) done scrub started at Sat Aug 19 15:58:22 2017 and finished after 00:31:06 total bytes scrubbed: 209.07GiB with 24487812 errors error details: verify=6101 csum=24481711 corrected errors: 24487812, uncorrectable errors: 0, unverified errors: 0 WARNING: errors detected during scrubbing, corrected root@zStore2:/dev# root@zStore2:/dev# root@zStore2:/dev# btrfs scrub start -rdB /dev/sda1 scrub device /dev/sda1 (id 1) done scrub started at Sat Aug 19 16:50:34 2017 and finished after 00:08:01 total bytes scrubbed: 209.07GiB with 0 errors root@zStore2:/dev# root@zStore2:/dev# btrfs dev stats /mnt/cache [/dev/sda1].write_io_errs 8139715 [/dev/sda1].read_io_errs 5920440 [/dev/sda1].flush_io_errs 166432 [/dev/sda1].corruption_errs 73445218 [/dev/sda1].generation_errs 19166 [/dev/sdf1].write_io_errs 0 [/dev/sdf1].read_io_errs 0 [/dev/sdf1].flush_io_errs 0 [/dev/sdf1].corruption_errs 0 [/dev/sdf1].generation_errs 0 root@zStore2:/dev# I have just doubled the connectivity for power so the +5V now measures at +4.92 on the circuit board with the most drives on it (row1). +12V is +11.98V Have moved Disk1 to row 2 slot 2 to balance better also. Had the array online for most of the afternoon doing the btrfs scrub and no issues or incidents occurred. I have not saved any new data to this since over 2 weeks ago now, ever since before the mover incident of this thread and then it was just that the cache was not being used which caused me to mistakenly select "preferred" which triggered the meltdown that followed. Not actually caused by mover but triggered by it. Reversing the mover action was the major meltdown trigger. Having downed the server again to work on it I have encountered this problem which has plagued this install of unRAID ever since I started. Just about every time I shutdown the unRAID server, the next reboot halts at a kernel panic like the below one. I can't scroll back to get any further info. I have to manually power off the VM, pull the USB and either scan for an impropper dismount, rerun the make-boot or sometimes reload the unRAID software and make-boot to get it to boot again. Next down, the same, over and over. What do you make of that?
August 19, 20178 yr Community Expert 15 minutes ago, fatpipe said: [/dev/sda1].write_io_errs 8139715 [/dev/sda1].read_io_errs 5920440 [/dev/sda1].flush_io_errs 166432 [/dev/sda1].corruption_errs 73445218 [/dev/sda1].generation_errs 19166 As you can see there are millions of read and write errors, these are hardware errors, usually a bad cable/connection, you can reset then using: btrfs dev stats -z /mnt/cache Then keep monitoring them, counters should all stay at 0 21 minutes ago, fatpipe said: I have to manually power off the VM, pull the USB and either scan for an impropper dismount, rerun the make-boot or sometimes reload the unRAID software and make-boot to get it to boot again. Next down, the same, over and over. What do you make of that? If you're using a USB 3.0 port try a 2.0 port instead, if already USB 2.0 try a different flash drive.
August 19, 20178 yr Author changing flash drive and license binding to USB mean it is not trivial. I only moved the license to this USB back in March or so when I built this. As I say been happening from the get-go so may well be a faulty USB bit. Don't I need Lime tech support to make that change though if it is under a year? On the cache btrfs errors. Yes plan to monitor, that was the count from after the recovery mentioned above. Will doublecheck the USB type of the port I am using but I believe it is USB2. Anything specific about USB3 port problems to cite? The USB I use is a USB2 only for sure and old since consigning it to unRAID meant it would probably never do anything else.
August 19, 20178 yr Community Expert 4 minutes ago, fatpipe said: Don't I need Lime tech support to make that change though if it is under a year? If it's another changer in the same year yes. 4 minutes ago, fatpipe said: On the cache btrfs errors. Yes plan to monitor, that was the count from after the recovery mentioned above. Checksum errors are the result of the read and write errors, and like I said those are hardware related. 5 minutes ago, fatpipe said: Anything specific about USB3 port problems to cite? Many users have issues with USB 3.0 ports, always use USB 2.0
August 27, 20178 yr Author Hi Forum again, Ever since the meltdown a few weeks back, at one point, all the shares were not showing up when the array was online and I recreated one of them at that time. That same share is the one that is not using the cache despite being set to do so (what I was fiddling with when I left the "preferred" setting for that share and the mover ran). Other shares are OK and use the cache fine so the cache is not "faulty". Cache is 2x 250G WD SSDs. Keeping on this thread as this relates to the initial situation that lead me to make the cache setting change that started this thread. I have stabilized the unRAID setup and the upshot basically is the controller change from SuperMicro to LSI to stop the driver crashes and the addition of more power connections inside the chassis as the +5V was dipping below threshold I believe and causing disks to go offline. Since remeding all this I have not had any issues except the one here and maybe a sus USB (but ignoring that for now). All the 8TB drives are working perfectly. Including the previously "failed" ones. Preclears done on all the spares too. Any suggestions on how to remedy the cache for the one share? Do I have to empty the cache and recreate it or something? Edited August 29, 20178 yr by fatpipe no responses
Archived
This topic is now archived and is closed to further replies.