THIS WAS RESOLVED WITH A SERVER REBOOT.

Posting anyway for others.

rack-diagnostics-20190512-2023.zip

 

Diag attached. I upgraded to 6.7 24 hours ago, all worked fine up front (for >12 hours). Some time in the past few hours all my /mnt/user/ shares became unavailable.

 

My "Unnasigned Devices" disk still works fine.

 

I can access /mnt/disk[1,2,3,4,5,6,7,8] fine and read all the data.

 

I cannot access /mnt/user/. I have pasted some commands below to demonstrate the errors.

 

root@rack:/mnt# ls
/bin/ls: cannot access 'user': Transport endpoint is not connected
cache/  disk2/  disk4/  disk6/  disk8/  user/
disk1/  disk3/  disk5/  disk7/  disks/  user0/

root@rack:/mnt# ls -lah
/bin/ls: cannot access 'user': Transport endpoint is not connected
total 32K
drwxr-xr-x 14 root   root  280 May 13 06:21 ./
drwxr-xr-x 19 root   root  400 May 13 06:23 ../
drwxrwxrwx  1 nobody users   0 May 13 06:21 cache/
drwxrwxrwx 12 nobody users 215 Mar  3  2018 disk1/
drwxrwxrwx  9 nobody users 137 Nov 16  2017 disk2/
drwxrwxrwx  9 nobody users 137 Aug 10  2018 disk3/
drwxrwxrwx  7 nobody users 124 Nov 16  2017 disk4/
drwxrwxrwx  4 nobody users  36 Nov 16  2017 disk5/
drwxrwxrwx  8 nobody users 119 Nov 16  2017 disk6/
drwxrwxrwx  8 nobody users  91 Nov 16  2017 disk7/
drwxrwxrwx  1 nobody users 124 May 12 07:38 disk8/
drwxrwxrwx  3 nobody users  60 May 13 06:20 disks/
d?????????  ? ?      ?       ?            ? user/
drwxrwxrwx  1 nobody users 215 May 12 07:38 user0/

root@rack:/mnt# cd ./user
-bash: cd: ./user: Transport endpoint is not connected

root@rack:/mnt# mount
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /var/log type tmpfs (rw,size=128m,mode=0755)
/dev/sda1 on /boot type vfat (rw,noatime,nodiratime,flush,umask=0,shortname=mixed)
/boot/bzmodules on /lib/modules type squashfs (ro)
/boot/bzfirmware on /lib/firmware type squashfs (ro)
/mnt on /mnt type none (rw,bind)
shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
/dev/md1 on /mnt/disk1 type xfs (rw,noatime,nodiratime)
/dev/md2 on /mnt/disk2 type xfs (rw,noatime,nodiratime)
/dev/md3 on /mnt/disk3 type xfs (rw,noatime,nodiratime)
/dev/md4 on /mnt/disk4 type xfs (rw,noatime,nodiratime)
/dev/md5 on /mnt/disk5 type xfs (rw,noatime,nodiratime)
/dev/md6 on /mnt/disk6 type xfs (rw,noatime,nodiratime)
/dev/md7 on /mnt/disk7 type xfs (rw,noatime,nodiratime)
/dev/md8 on /mnt/disk8 type btrfs (rw,noatime,nodiratime)
shfs on /mnt/user0 type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
/dev/sdj1 on /mnt/disks/Kingston240SSD type xfs (rw,noatime,nodiratime,discard)
/dev/sdk1 on /mnt/cache type btrfs (rw,noatime,nodiratime)

 

As a failsafe, I'd Check Disk Filesystem on disk 1, as that's what started the errors

May 13 03:40:01 rack move: move: skip /mnt/disk1/system/docker/docker.img
May 13 03:40:01 rack kernel: shfs[23376]: segfault at 0 ip 000000000040277f sp 000014b6af68c8f0 error 4 in shfs[402000+b000]
May 13 03:40:01 rack kernel: Code: 89 7d f8 48 89 75 f0 eb 1d 48 8b 55 f0 48 8d 42 01 48 89 45 f0 48 8b 45 f8 48 8d 48 01 48 89 4d f8 0f b6 12 88 10 48 8b 45 f0 <0f> b6 00 84 c0 74 0b 48 8b 45 f0 0f b6 00 3c 2f 75 cd 48 8b 45 f8
May 13 03:40:01 rack move: move: create_parent: /mnt/disk1 error: Software caused connection abort