Stuck at Array Starting Mounting disks


daemian

Recommended Posts

LIke the subject says, my array wont start. I am not sure what happened exactly, but my parity drive is showing as disabled. When i try to start the array it just sticks at "Array Starting Moutning disks." I thought perhaps the parity drive had failed, so it tried removing the parity drive and starting the array, but the result is the same. I logged in and did a tail -f on /var/log/syslog prior to clicking start eh array, and this is what I got:

 

Linux 4.19.56-Unraid.
root@dt-ur01:~# tail -f /var/log/syslog
Jan 16 14:19:48 dt-ur01 avahi-daemon[5075]: Joining mDNS multicast group on interface bond0.IPv6 with address fe80::225:90ff:fe67:edd4.
Jan 16 14:19:48 dt-ur01 avahi-daemon[5075]: New relevant interface bond0.IPv6 for mDNS.
Jan 16 14:19:48 dt-ur01 avahi-daemon[5075]: Registering new address record for fe80::225:90ff:fe67:edd4 on bond0.*.
Jan 16 14:19:51 dt-ur01 ntpd[1960]: Listen normally on 4 br0 192.168.0.15:123
Jan 16 14:19:51 dt-ur01 ntpd[1960]: Listen normally on 5 br0 [fe80::c0ea:1ff:fe18:e34%13]:123
Jan 16 14:19:51 dt-ur01 ntpd[1960]: new interface(s) found: waking up resolver
Jan 16 14:20:17 dt-ur01 login[4947]: ROOT LOGIN  on '/dev/tty1'
Jan 16 14:20:29 dt-ur01 ntpd[1960]: receive: Unexpected origin timestamp 0xe1cb440c.faff7bf0 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe1cb440d.6acf5e45
Jan 16 14:20:29 dt-ur01 ntpd[1960]: receive: Unexpected origin timestamp 0xe1cb440c.fb00e459 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe1cb440d.6bd6d743
Jan 16 14:20:47 dt-ur01 sshd[7419]: Accepted password for root from 192.168.0.101 port 63658 ssh2
Jan 16 14:21:20 dt-ur01 emhttpd: req (1): startState=STOPPED&file=&csrf_token=****************&cmdStart=Start
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (145): /usr/local/sbin/set_ncq sdb 1
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (41): set md_num_stripes 1280
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (42): set md_sync_window 384
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (43): set md_sync_thresh 192
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (44): set md_write_method
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (45): set spinup_group 1 0
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (46): set spinup_group 2 0
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (47): set spinup_group 3 0
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (48): set spinup_group 4 0
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (146): echo 128 > /sys/block/sdb/queue/nr_requests
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (147): /usr/local/sbin/set_ncq sdc 1
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (148): echo 128 > /sys/block/sdc/queue/nr_requests
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (149): /usr/local/sbin/set_ncq sde 1
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (150): echo 128 > /sys/block/sde/queue/nr_requests
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (151): /usr/local/sbin/set_ncq sdd 1
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (152): echo 128 > /sys/block/sdd/queue/nr_requests
Jan 16 14:21:20 dt-ur01 kernel: mdcmd (49): start STOPPED
Jan 16 14:21:20 dt-ur01 kernel: unraid: allocating 31100K for 1280 stripes (6 disks)
Jan 16 14:21:20 dt-ur01 kernel: md1: running, size: 5860522532 blocks
Jan 16 14:21:20 dt-ur01 kernel: md2: running, size: 2930266532 blocks
Jan 16 14:21:20 dt-ur01 kernel: md3: running, size: 5860522532 blocks
Jan 16 14:21:20 dt-ur01 kernel: md4: running, size: 2930266532 blocks
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (153): udevadm settle
Jan 16 14:21:20 dt-ur01 root: Starting diskload
Jan 16 14:21:20 dt-ur01 emhttpd: Mounting disks...
Jan 16 14:21:20 dt-ur01 emhttpd: shcmd (160): /sbin/btrfs device scan
Jan 16 14:21:21 dt-ur01 root: Scanning for Btrfs filesystems
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (161): mkdir -p /mnt/disk1
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (162): mount -t xfs -o noatime,nodiratime /dev/md1 /mnt/disk1
Jan 16 14:21:21 dt-ur01 kernel: SGI XFS with ACLs, security attributes, no debug enabled
Jan 16 14:21:21 dt-ur01 kernel: XFS (md1): Mounting V5 Filesystem
Jan 16 14:21:21 dt-ur01 kernel: XFS (md1): Ending clean mount
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (163): xfs_growfs /mnt/disk1
Jan 16 14:21:21 dt-ur01 root: meta-data=/dev/md1               isize=512    agcount=8, agsize=183141659 blks
Jan 16 14:21:21 dt-ur01 root:          =                       sectsz=512   attr=2, projid32bit=1
Jan 16 14:21:21 dt-ur01 root:          =                       crc=1        finobt=1, sparse=0, rmapbt=0
Jan 16 14:21:21 dt-ur01 root:          =                       reflink=0
Jan 16 14:21:21 dt-ur01 root: data     =                       bsize=4096   blocks=1465130633, imaxpct=5
Jan 16 14:21:21 dt-ur01 root:          =                       sunit=0      swidth=0 blks
Jan 16 14:21:21 dt-ur01 root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Jan 16 14:21:21 dt-ur01 root: log      =internal log           bsize=4096   blocks=357698, version=2
Jan 16 14:21:21 dt-ur01 root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Jan 16 14:21:21 dt-ur01 root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (164): mkdir -p /mnt/disk2
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (165): mount -t xfs -o noatime,nodiratime /dev/md2 /mnt/disk2
Jan 16 14:21:21 dt-ur01 kernel: XFS (md2): Mounting V5 Filesystem
Jan 16 14:21:21 dt-ur01 kernel: XFS (md2): Ending clean mount
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (166): xfs_growfs /mnt/disk2
Jan 16 14:21:21 dt-ur01 root: meta-data=/dev/md2               isize=512    agcount=4, agsize=183141659 blks
Jan 16 14:21:21 dt-ur01 root:          =                       sectsz=512   attr=2, projid32bit=1
Jan 16 14:21:21 dt-ur01 root:          =                       crc=1        finobt=1, sparse=0, rmapbt=0
Jan 16 14:21:21 dt-ur01 root:          =                       reflink=0
Jan 16 14:21:21 dt-ur01 root: data     =                       bsize=4096   blocks=732566633, imaxpct=5
Jan 16 14:21:21 dt-ur01 root:          =                       sunit=0      swidth=0 blks
Jan 16 14:21:21 dt-ur01 root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Jan 16 14:21:21 dt-ur01 root: log      =internal log           bsize=4096   blocks=357698, version=2
Jan 16 14:21:21 dt-ur01 root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Jan 16 14:21:21 dt-ur01 root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (167): mkdir -p /mnt/disk3
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (168): mount -t xfs -o noatime,nodiratime /dev/md3 /mnt/disk3
Jan 16 14:21:21 dt-ur01 kernel: XFS (md3): Mounting V5 Filesystem
Jan 16 14:21:21 dt-ur01 kernel: XFS (md3): Ending clean mount
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (169): xfs_growfs /mnt/disk3
Jan 16 14:21:21 dt-ur01 root: meta-data=/dev/md3               isize=512    agcount=6, agsize=268435455 blks
Jan 16 14:21:21 dt-ur01 root:          =                       sectsz=512   attr=2, projid32bit=1
Jan 16 14:21:21 dt-ur01 root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Jan 16 14:21:21 dt-ur01 root:          =                       reflink=0
Jan 16 14:21:21 dt-ur01 root: data     =                       bsize=4096   blocks=1465130633, imaxpct=5
Jan 16 14:21:21 dt-ur01 root:          =                       sunit=0      swidth=0 blks
Jan 16 14:21:21 dt-ur01 root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Jan 16 14:21:21 dt-ur01 root: log      =internal log           bsize=4096   blocks=521728, version=2
Jan 16 14:21:21 dt-ur01 root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Jan 16 14:21:21 dt-ur01 root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (170): mkdir -p /mnt/disk4
Jan 16 14:21:21 dt-ur01 emhttpd: shcmd (171): mount -t xfs -o noatime,nodiratime /dev/md4 /mnt/disk4
Jan 16 14:21:21 dt-ur01 kernel: XFS (md4): Mounting V5 Filesystem
Jan 16 14:21:22 dt-ur01 kernel: XFS (md4): Ending clean mount
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (172): xfs_growfs /mnt/disk4
Jan 16 14:21:22 dt-ur01 root: meta-data=/dev/md4               isize=512    agcount=4, agsize=183141659 blks
Jan 16 14:21:22 dt-ur01 root:          =                       sectsz=512   attr=2, projid32bit=1
Jan 16 14:21:22 dt-ur01 root:          =                       crc=1        finobt=1, sparse=0, rmapbt=0
Jan 16 14:21:22 dt-ur01 root:          =                       reflink=0
Jan 16 14:21:22 dt-ur01 root: data     =                       bsize=4096   blocks=732566633, imaxpct=5
Jan 16 14:21:22 dt-ur01 root:          =                       sunit=0      swidth=0 blks
Jan 16 14:21:22 dt-ur01 root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Jan 16 14:21:22 dt-ur01 root: log      =internal log           bsize=4096   blocks=357698, version=2
Jan 16 14:21:22 dt-ur01 root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Jan 16 14:21:22 dt-ur01 root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (173): mkdir -p /mnt/cache
Jan 16 14:21:22 dt-ur01 emhttpd: cache uuid: 437a0de9-148c-436a-823b-1f32632b68f2
Jan 16 14:21:22 dt-ur01 emhttpd: cache TotDevices: 2
Jan 16 14:21:22 dt-ur01 emhttpd: cache NumDevices: 2
Jan 16 14:21:22 dt-ur01 emhttpd: cache NumFound: 2
Jan 16 14:21:22 dt-ur01 emhttpd: cache NumMissing: 0
Jan 16 14:21:22 dt-ur01 emhttpd: cache NumMisplaced: 0
Jan 16 14:21:22 dt-ur01 emhttpd: cache NumExtra: 0
Jan 16 14:21:22 dt-ur01 emhttpd: cache LuksState: 0
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (174): mount -t btrfs -o noatime,nodiratime -U 437a0de9-148c-436a-823b-1f32632b68f2 /mnt/cache
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): disk space caching is enabled
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): has skinny extents
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): bdev /dev/sdg1 errs: wr 3408448785, rd 2878657541, flush 77125088, corrupt 0, gen 0
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): bdev /dev/sdh1 errs: wr 387, rd 1, flush 0, corrupt 0, gen 0
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): enabling ssd optimizations
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (175): /sbin/btrfs filesystem resize 1:max /mnt/cache
Jan 16 14:21:22 dt-ur01 root: Resize '/mnt/cache' of '1:max'
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (176): /sbin/btrfs filesystem resize 2:max /mnt/cache
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): resizing devid 1
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): new size for /dev/sdg1 is 500107829248
Jan 16 14:21:22 dt-ur01 root: Resize '/mnt/cache' of '2:max'
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): resizing devid 2
Jan 16 14:21:22 dt-ur01 kernel: BTRFS info (device sdg1): new size for /dev/sdh1 is 500107829248
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (177): sync
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (178): mkdir /mnt/user0
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (179): /usr/local/sbin/shfs /mnt/user0 -disks 30 -o noatime,big_writes,allow_other   |& logger
Jan 16 14:21:22 dt-ur01 shfs: stderr redirected to syslog
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (180): mkdir /mnt/user
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (181): /usr/local/sbin/shfs /mnt/user -disks 31 2048000000 -o noatime,big_writes,allow_other  -o remember=0  |& logger
Jan 16 14:21:22 dt-ur01 shfs: stderr redirected to syslog
Jan 16 14:21:22 dt-ur01 emhttpd: shcmd (183): /usr/local/sbin/update_cron
Jan 16 14:21:22 dt-ur01 root: Delaying execution of fix common problems scan for 10 minutes
Jan 16 14:22:01 dt-ur01 sSMTP[9952]: Creating SSL connection to host
Jan 16 14:22:01 dt-ur01 sSMTP[9952]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 16 14:22:05 dt-ur01 sSMTP[9952]: Sent mail for [email protected] (221 2.0.0 closing connection k6sm10383601ywh.56 - gsmtp) uid=0 username=root outbytes=920
Jan 16 14:24:20 dt-ur01 nginx: 2020/01/16 14:24:20 [error] 5132#5132: *68 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.0.101, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "192.168.0.15", referrer: "http://192.168.0.15/Main"
Jan 16 14:25:30 dt-ur01 ntpd[1960]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Jan 16 14:31:00 dt-ur01 root: Fix Common Problems Version 2019.12.29
Jan 16 14:31:01 dt-ur01 root: Fix Common Problems: Warning: Share appdata set to cache-only, but files / folders exist on the array
Jan 16 14:31:09 dt-ur01 sSMTP[25208]: Creating SSL connection to host
Jan 16 14:31:09 dt-ur01 sSMTP[25208]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 16 14:31:11 dt-ur01 sSMTP[25208]: Sent mail for [email protected] (221 2.0.0 closing connection d186sm10664771ywe.0 - gsmtp) uid=0 username=root outbytes=865

 

Any idea what the hang up is?

 

Thanks

 

Link to comment

Correct, the GUI just continues to show "Array Starting - Mounting disks".

 

I can see at least some of the shares on the network, but the problem for me is that Dockers and VMs are not starting.

 

I do see tons of errors on the cache pool:

Quote

root@dt-ur01:~# btrfs dev stats /mnt/cache
[/dev/sdg1].write_io_errs    18446744072823033105
[/dev/sdg1].read_io_errs     18446744072293241861
[/dev/sdg1].flush_io_errs    77125088
[/dev/sdg1].corruption_errs  0
[/dev/sdg1].generation_errs  0
[/dev/sdh1].write_io_errs    387
[/dev/sdh1].read_io_errs     1
[/dev/sdh1].flush_io_errs    0
[/dev/sdh1].corruption_errs  0
[/dev/sdh1].generation_errs  0

unfortunately I am not physically local to this array, so I cannot swap the cables right now. I will work on getting some cables and hands to the location.

 

Would it be worthwhile to run the scrub prior to swapping the cables, and if so, forgive my ignorance but how do I do that? I did a bit of searching and have found a few different commands, etc. Thanks

Screen Shot 2020-01-17 at 8.27.13 AM.png

Link to comment

Docker and VM services are not starting, not even attempting to start, might or not be be related to cache issues, and pool is kind of a mess, with multiple profiles, doubt scrub alone can fix it, especially since docker and VM shares are by default NOCOW (this means it won't have checksums, so can't be fixed by scrub), stop docker/VM services and see if the array starts normally, if it does best to backup anything still on cache and re-format, then don' forget to check/replace cables or it will likely happen again.

Link to comment

While in safe mode, set the array to not autostart (Settings - Disk Settings)  Then reboot

 

From the plugins page, you uninstall them one at a time with a reboot in between

 

I'd start with NerdPack, followed by user scripts, followed by Preclear and then dealers choice after that.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.