January 16, 20206 yr LIke the subject says, my array wont start. I am not sure what happened exactly, but my parity drive is showing as disabled. When i try to start the array it just sticks at "Array Starting Moutning disks." I thought perhaps the parity drive had failed, so it tried removing the parity drive and starting the array, but the result is the same. I logged in and did a tail -f on /var/log/syslog prior to clicking start eh array, and this is what I got: Linux 4.19.56-Unraid. root@dt-ur01:~# tail -f /var/log/syslog Jan 16 14:19:48 dt-ur01 avahi-daemon[5075]: Joining mDNS multicast group on interface bond0.IPv6 with address fe80::225:90ff:fe67:edd4. Jan 16 14:19:48 dt-ur01 avahi-daemon[5075]: New relevant interface bond0.IPv6 for mDNS. Jan 16 14:19:48 dt-ur01 avahi-daemon[5075]: Registering new address record for fe80::225:90ff:fe67:edd4 on bond0.*. Jan 16 14:19:51 dt-ur01 ntpd[1960]: Listen normally on 4 br0 192.168.0.15:123 Jan 16 14:19:51 dt-ur01 ntpd[1960]: Listen normally on 5 br0 [fe80::c0ea:1ff:fe18:e34%13]:123 Jan 16 14:19:51 dt-ur01 ntpd[1960]: new interface(s) found: waking up resolver Jan 16 14:20:17 dt-ur01 login[4947]: ROOT LOGIN on '/dev/tty1' Jan 16 14:20:29 dt-ur01 ntpd[1960]: receive: Unexpected origin timestamp 0xe1cb440c.faff7bf0 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe1cb440d.6acf5e45 Jan 16 14:20:29 dt-ur01 ntpd[1960]: receive: Unexpected origin timestamp 0xe1cb440c.fb00e459 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe1cb440d.6bd6d743 Jan 16 14:20:47 dt-ur01 sshd[7419]: Accepted password for root from 192.168.0.101 port 63658 ssh2 Jan 16 14:21:20 dt-ur01 emhttpd: req (1): startState=STOPPED&file=&csrf_token=****************&cmdStart=Start Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (145): /usr/local/sbin/set_ncq sdb 1 Jan 16 14:21:20 dt-ur01 kernel: mdcmd (41): set md_num_stripes 1280 Jan 16 14:21:20 dt-ur01 kernel: mdcmd (42): set md_sync_window 384 Jan 16 14:21:20 dt-ur01 kernel: mdcmd (43): set md_sync_thresh 192 Jan 16 14:21:20 dt-ur01 kernel: mdcmd (44): set md_write_method Jan 16 14:21:20 dt-ur01 kernel: mdcmd (45): set spinup_group 1 0 Jan 16 14:21:20 dt-ur01 kernel: mdcmd (46): set spinup_group 2 0 Jan 16 14:21:20 dt-ur01 kernel: mdcmd (47): set spinup_group 3 0 Jan 16 14:21:20 dt-ur01 kernel: mdcmd (48): set spinup_group 4 0 Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (146): echo 128 > /sys/block/sdb/queue/nr_requests Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (147): /usr/local/sbin/set_ncq sdc 1 Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (148): echo 128 > /sys/block/sdc/queue/nr_requests Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (149): /usr/local/sbin/set_ncq sde 1 Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (150): echo 128 > /sys/block/sde/queue/nr_requests Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (151): /usr/local/sbin/set_ncq sdd 1 Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (152): echo 128 > /sys/block/sdd/queue/nr_requests Jan 16 14:21:20 dt-ur01 kernel: mdcmd (49): start STOPPED Jan 16 14:21:20 dt-ur01 kernel: unraid: allocating 31100K for 1280 stripes (6 disks) Jan 16 14:21:20 dt-ur01 kernel: md1: running, size: 5860522532 blocks Jan 16 14:21:20 dt-ur01 kernel: md2: running, size: 2930266532 blocks Jan 16 14:21:20 dt-ur01 kernel: md3: running, size: 5860522532 blocks Jan 16 14:21:20 dt-ur01 kernel: md4: running, size: 2930266532 blocks Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (153): udevadm settle Jan 16 14:21:20 dt-ur01 root: Starting diskload Jan 16 14:21:20 dt-ur01 emhttpd: Mounting disks... Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (160): /sbin/btrfs device scan Jan 16 14:21:21 dt-ur01 root: Scanning for Btrfs filesystems Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (161): mkdir -p /mnt/disk1 Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (162): mount -t xfs -o noatime,nodiratime /dev/md1 /mnt/disk1 Jan 16 14:21:21 dt-ur01 kernel: SGI XFS with ACLs, security attributes, no debug enabled Jan 16 14:21:21 dt-ur01 kernel: XFS (md1): Mounting V5 Filesystem Jan 16 14:21:21 dt-ur01 kernel: XFS (md1): Ending clean mount Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (163): xfs_growfs /mnt/disk1 Jan 16 14:21:21 dt-ur01 root: meta-data=/dev/md1 isize=512 agcount=8, agsize=183141659 blks Jan 16 14:21:21 dt-ur01 root: = sectsz=512 attr=2, projid32bit=1 Jan 16 14:21:21 dt-ur01 root: = crc=1 finobt=1, sparse=0, rmapbt=0 Jan 16 14:21:21 dt-ur01 root: = reflink=0 Jan 16 14:21:21 dt-ur01 root: data = bsize=4096 blocks=1465130633, imaxpct=5 Jan 16 14:21:21 dt-ur01 root: = sunit=0 swidth=0 blks Jan 16 14:21:21 dt-ur01 root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jan 16 14:21:21 dt-ur01 root: log =internal log bsize=4096 blocks=357698, version=2 Jan 16 14:21:21 dt-ur01 root: = sectsz=512 sunit=0 blks, lazy-count=1 Jan 16 14:21:21 dt-ur01 root: realtime =none extsz=4096 blocks=0, rtextents=0 Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (164): mkdir -p /mnt/disk2 Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (165): mount -t xfs -o noatime,nodiratime /dev/md2 /mnt/disk2 Jan 16 14:21:21 dt-ur01 kernel: XFS (md2): Mounting V5 Filesystem Jan 16 14:21:21 dt-ur01 kernel: XFS (md2): Ending clean mount Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (166): xfs_growfs /mnt/disk2 Jan 16 14:21:21 dt-ur01 root: meta-data=/dev/md2 isize=512 agcount=4, agsize=183141659 blks Jan 16 14:21:21 dt-ur01 root: = sectsz=512 attr=2, projid32bit=1 Jan 16 14:21:21 dt-ur01 root: = crc=1 finobt=1, sparse=0, rmapbt=0 Jan 16 14:21:21 dt-ur01 root: = reflink=0 Jan 16 14:21:21 dt-ur01 root: data = bsize=4096 blocks=732566633, imaxpct=5 Jan 16 14:21:21 dt-ur01 root: = sunit=0 swidth=0 blks Jan 16 14:21:21 dt-ur01 root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jan 16 14:21:21 dt-ur01 root: log =internal log bsize=4096 blocks=357698, version=2 Jan 16 14:21:21 dt-ur01 root: = sectsz=512 sunit=0 blks, lazy-count=1 Jan 16 14:21:21 dt-ur01 root: realtime =none extsz=4096 blocks=0, rtextents=0 Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (167): mkdir -p /mnt/disk3 Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (168): mount -t xfs -o noatime,nodiratime /dev/md3 /mnt/disk3 Jan 16 14:21:21 dt-ur01 kernel: XFS (md3): Mounting V5 Filesystem Jan 16 14:21:21 dt-ur01 kernel: XFS (md3): Ending clean mount Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (169): xfs_growfs /mnt/disk3 Jan 16 14:21:21 dt-ur01 root: meta-data=/dev/md3 isize=512 agcount=6, agsize=268435455 blks Jan 16 14:21:21 dt-ur01 root: = sectsz=512 attr=2, projid32bit=1 Jan 16 14:21:21 dt-ur01 root: = crc=1 finobt=1, sparse=1, rmapbt=0 Jan 16 14:21:21 dt-ur01 root: = reflink=0 Jan 16 14:21:21 dt-ur01 root: data = bsize=4096 blocks=1465130633, imaxpct=5 Jan 16 14:21:21 dt-ur01 root: = sunit=0 swidth=0 blks Jan 16 14:21:21 dt-ur01 root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jan 16 14:21:21 dt-ur01 root: log =internal log bsize=4096 blocks=521728, version=2 Jan 16 14:21:21 dt-ur01 root: = sectsz=512 sunit=0 blks, lazy-count=1 Jan 16 14:21:21 dt-ur01 root: realtime =none extsz=4096 blocks=0, rtextents=0 Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (170): mkdir -p /mnt/disk4 Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (171): mount -t xfs -o noatime,nodiratime /dev/md4 /mnt/disk4 Jan 16 14:21:21 dt-ur01 kernel: XFS (md4): Mounting V5 Filesystem Jan 16 14:21:22 dt-ur01 kernel: XFS (md4): Ending clean mount Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (172): xfs_growfs /mnt/disk4 Jan 16 14:21:22 dt-ur01 root: meta-data=/dev/md4 isize=512 agcount=4, agsize=183141659 blks Jan 16 14:21:22 dt-ur01 root: = sectsz=512 attr=2, projid32bit=1 Jan 16 14:21:22 dt-ur01 root: = crc=1 finobt=1, sparse=0, rmapbt=0 Jan 16 14:21:22 dt-ur01 root: = reflink=0 Jan 16 14:21:22 dt-ur01 root: data = bsize=4096 blocks=732566633, imaxpct=5 Jan 16 14:21:22 dt-ur01 root: = sunit=0 swidth=0 blks Jan 16 14:21:22 dt-ur01 root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jan 16 14:21:22 dt-ur01 root: log =internal log bsize=4096 blocks=357698, version=2 Jan 16 14:21:22 dt-ur01 root: = sectsz=512 sunit=0 blks, lazy-count=1 Jan 16 14:21:22 dt-ur01 root: realtime =none extsz=4096 blocks=0, rtextents=0 Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (173): mkdir -p /mnt/cache Jan 16 14:21:22 dt-ur01 emhttpd: cache uuid: 437a0de9-148c-436a-823b-1f32632b68f2 Jan 16 14:21:22 dt-ur01 emhttpd: cache TotDevices: 2 Jan 16 14:21:22 dt-ur01 emhttpd: cache NumDevices: 2 Jan 16 14:21:22 dt-ur01 emhttpd: cache NumFound: 2 Jan 16 14:21:22 dt-ur01 emhttpd: cache NumMissing: 0 Jan 16 14:21:22 dt-ur01 emhttpd: cache NumMisplaced: 0 Jan 16 14:21:22 dt-ur01 emhttpd: cache NumExtra: 0 Jan 16 14:21:22 dt-ur01 emhttpd: cache LuksState: 0 Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (174): mount -t btrfs -o noatime,nodiratime -U 437a0de9-148c-436a-823b-1f32632b68f2 /mnt/cache Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): disk space caching is enabled Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): has skinny extents Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): bdev /dev/sdg1 errs: wr 3408448785, rd 2878657541, flush 77125088, corrupt 0, gen 0 Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): bdev /dev/sdh1 errs: wr 387, rd 1, flush 0, corrupt 0, gen 0 Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): enabling ssd optimizations Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (175): /sbin/btrfs filesystem resize 1:max /mnt/cache Jan 16 14:21:22 dt-ur01 root: Resize '/mnt/cache' of '1:max' Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (176): /sbin/btrfs filesystem resize 2:max /mnt/cache Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): resizing devid 1 Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): new size for /dev/sdg1 is 500107829248 Jan 16 14:21:22 dt-ur01 root: Resize '/mnt/cache' of '2:max' Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): resizing devid 2 Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): new size for /dev/sdh1 is 500107829248 Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (177): sync Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (178): mkdir /mnt/user0 Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (179): /usr/local/sbin/shfs /mnt/user0 -disks 30 -o noatime,big_writes,allow_other |& logger Jan 16 14:21:22 dt-ur01 shfs: stderr redirected to syslog Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (180): mkdir /mnt/user Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (181): /usr/local/sbin/shfs /mnt/user -disks 31 2048000000 -o noatime,big_writes,allow_other -o remember=0 |& logger Jan 16 14:21:22 dt-ur01 shfs: stderr redirected to syslog Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (183): /usr/local/sbin/update_cron Jan 16 14:21:22 dt-ur01 root: Delaying execution of fix common problems scan for 10 minutes Jan 16 14:22:01 dt-ur01 sSMTP[9952]: Creating SSL connection to host Jan 16 14:22:01 dt-ur01 sSMTP[9952]: SSL connection using TLS_AES_256_GCM_SHA384 Jan 16 14:22:05 dt-ur01 sSMTP[9952]: Sent mail for [email protected] (221 2.0.0 closing connection k6sm10383601ywh.56 - gsmtp) uid=0 username=root outbytes=920 Jan 16 14:24:20 dt-ur01 nginx: 2020/01/16 14:24:20 [error] 5132#5132: *68 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.0.101, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "192.168.0.15", referrer: "http://192.168.0.15/Main" Jan 16 14:25:30 dt-ur01 ntpd[1960]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Jan 16 14:31:00 dt-ur01 root: Fix Common Problems Version 2019.12.29 Jan 16 14:31:01 dt-ur01 root: Fix Common Problems: Warning: Share appdata set to cache-only, but files / folders exist on the array Jan 16 14:31:09 dt-ur01 sSMTP[25208]: Creating SSL connection to host Jan 16 14:31:09 dt-ur01 sSMTP[25208]: SSL connection using TLS_AES_256_GCM_SHA384 Jan 16 14:31:11 dt-ur01 sSMTP[25208]: Sent mail for [email protected] (221 2.0.0 closing connection d186sm10664771ywe.0 - gsmtp) uid=0 username=root outbytes=865 Any idea what the hang up is? Thanks
January 17, 20206 yr When you say stuck do you mean just the GUI is stuck? The server mounted all disks and appears to be functioning normally, shares should be available. I do see a problem with your cache pool, there are read/write errors on both devices, likely from them dropping offline, you need to run a scrub, see here for more details.
January 17, 20206 yr Author Correct, the GUI just continues to show "Array Starting - Mounting disks". I can see at least some of the shares on the network, but the problem for me is that Dockers and VMs are not starting. I do see tons of errors on the cache pool: Quote root@dt-ur01:~# btrfs dev stats /mnt/cache [/dev/sdg1].write_io_errs 18446744072823033105 [/dev/sdg1].read_io_errs 18446744072293241861 [/dev/sdg1].flush_io_errs 77125088 [/dev/sdg1].corruption_errs 0 [/dev/sdg1].generation_errs 0 [/dev/sdh1].write_io_errs 387 [/dev/sdh1].read_io_errs 1 [/dev/sdh1].flush_io_errs 0 [/dev/sdh1].corruption_errs 0 [/dev/sdh1].generation_errs 0 unfortunately I am not physically local to this array, so I cannot swap the cables right now. I will work on getting some cables and hands to the location. Would it be worthwhile to run the scrub prior to swapping the cables, and if so, forgive my ignorance but how do I do that? I did a bit of searching and have found a few different commands, etc. Thanks
January 17, 20206 yr Docker and VM services are not starting, not even attempting to start, might or not be be related to cache issues, and pool is kind of a mess, with multiple profiles, doubt scrub alone can fix it, especially since docker and VM shares are by default NOCOW (this means it won't have checksums, so can't be fixed by scrub), stop docker/VM services and see if the array starts normally, if it does best to backup anything still on cache and re-format, then don' forget to check/replace cables or it will likely happen again.
January 17, 20206 yr Author I disabled VMs and Docker, and attempted to start. but am still getting the same result. Just hangs on Mounting disks New diagnostics attached if its helpful. dt-ur01-diagnostics-20200117-1633.zip
January 17, 20206 yr Nothing jumps out, take to opportunity to upgrade to v.6.8.1, if it still doesn't start completely try safe mode or re-doing the flash drive.
January 17, 20206 yr Author ok i upgraded to 6.8.1 - no change I was able to start the array in maintenance mode (i presume thats what you means by safe mode). Not sure what I should do from here.
January 17, 20206 yr 3 minutes ago, daemian said: I was able to start the array in maintenance mode (i presume thats what you means by safe mode). No. He meant select Safe Mode from unRaid boot menu and then try starting the array after it's booted up without any plugins installed (safe mode)
January 17, 20206 yr Author ah sure, thanks for the clarification. I rebooted into safe mode and was able to start the array successfully. So I guess the question is how do I figure out what is hosing things up.
January 17, 20206 yr While in safe mode, set the array to not autostart (Settings - Disk Settings) Then reboot From the plugins page, you uninstall them one at a time with a reboot in between I'd start with NerdPack, followed by user scripts, followed by Preclear and then dealers choice after that.
January 17, 20206 yr Author it was user scripts. Once i removed that one the array started. THANK YOU BOTH!
January 17, 20206 yr 11 minutes ago, daemian said: it was user scripts. Once i removed that one the array started. THANK YOU BOTH! User Scripts by itself would not have been the problem, but it might well have been a script you were asking it to run.
January 17, 20206 yr 4 minutes ago, itimpi said: User Scripts by itself would not have been the problem, but it might well have been a script you were asking it to run. Yeah, this ^^^ Your script never finished executing, so it tied up the boot process
January 18, 20206 yr 9 hours ago, Squid said: Your script never finished executing, so it tied up the boot process Maybe a good idea to have user scripts log any script start/finish on the syslog by default, at least scripts run at array start, it would make it easier to spot these issues.
Archived
This topic is now archived and is closed to further replies.