XtremeOwnage

I upgraded to 6.12.4 earlier this week, after upgrading, I went from a perfectly stable, and reliable system, to one that now... just breaks randomly. The symptoms I notice, include maxed out CPU. I am unable to ssh, or even use the root console while this is occuring. Disabled a few things, doubled the allocated resources. No VMs, only docker, storage, nfs, and smb. Going to downgrade to 6.12.3. Diagnostics are attached. tower-diagnostics-20230907-1750.zip

I found an odd bug, which I so far, have reproduced twice. The symptoms- logging into ssh will just time out. Jun 10 10:07:18 Tower sshd[51491]: Connection from 10.100.10.3 port 2782 on 10.100.4.24 port 22 rdomain "" Jun 10 10:07:46 Tower sshd[53503]: Connection from 10.100.10.3 port 2790 on 10.100.5.2 port 22 rdomain "" Jun 10 10:08:46 Tower sshd[58123]: Connection from 10.100.10.3 port 2803 on 10.100.5.2 port 22 rdomain "" Jun 10 10:08:59 Tower sshd[58123]: Received disconnect from 10.100.10.3 port 2803:11: Session closed [preauth] Jun 10 10:08:59 Tower sshd[58123]: Disconnected from 10.100.10.3 port 2803 [preauth] Jun 10 10:09:15 Tower sshd[51491]: Received disconnect from 10.100.10.3 port 2782:11: Session closed [preauth] Jun 10 10:09:15 Tower sshd[51491]: Disconnected from authenticating user root 10.100.10.3 port 2782 [preauth] Jun 10 10:09:36 Tower sshd[62244]: Connection from 10.100.10.3 port 2812 on 10.100.5.2 port 22 rdomain "" Jun 10 10:09:43 Tower sshd[53503]: Connection closed by authenticating user root 10.100.10.3 port 2790 [preauth] Jun 10 10:09:54 Tower sshd[63481]: Connection from 10.100.10.3 port 2818 on 10.100.4.24 port 22 rdomain "" However, it will never connect. As well, when trying "su root" (when run as root), this also hangs. the logs, however, will show a successful su. Jun 10 10:34:05 Tower su[51135]: Successful su for root by root Jun 10 10:34:05 Tower su[51135]: + /dev/pts/0 root:root When this issue occurs, there is nothing of interest from the dmesg output. As well, using the terminal from the web interface works fine, SMB and NFS works fine, and I am unable to identify anything else impacted aside from su, and ssh/sftp. SFTP exhibits the same symptoms noticed by SSH, which is expected. I can confirm, restarting the server does correct this. Restarting the service using: root@Tower:~# /etc/rc.d/rc.sshd stop root@Tower:~# /etc/rc.d/rc.sshd start However, does not help this. Will note, when this does occur, I am unable to su to ANY user account. The logs, however, do show success. Jun 10 10:41:06 Tower su[17603]: Successful su for scanner by root Jun 10 10:41:06 Tower su[17603]: + /dev/pts/0 root:scanner Jun 10 10:41:16 Tower su[18337]: Successful su for root by root Jun 10 10:41:16 Tower su[18337]: + /dev/pts/0 root:root Jun 10 10:41:19 Tower su[18640]: Successful su for root by root Jun 10 10:41:19 Tower su[18640]: + /dev/pts/0 root:root Jun 10 10:41:23 Tower su[18844]: Successful su for root by root Jun 10 10:41:23 Tower su[18844]: + /dev/pts/0 root:root Based on what I have seen, I don't think this is related to SSH, but, rather, it seems something is getting hung when creating a new session somewhere in linux. I have attached diagnostics, and I don't have any need to reboot currently, so- if you have any other steps you would like executed, please let me know.... and I should be able to execute them while in this "broken" state. Diagnostics are attached. tower-diagnostics-20230610-1044.zip

This likely is not related to 6.12, however, I have an issue where my disks are being spun up around every hour for no reason. Hardware: Dell R730XD. HBA 330 mini. May 18 14:42:49 Tower emhttpd: read SMART /dev/sdk May 18 14:57:38 Tower emhttpd: spinning down /dev/sdk May 18 16:07:46 Tower emhttpd: read SMART /dev/sdk May 18 16:22:36 Tower emhttpd: spinning down /dev/sdk May 18 17:32:46 Tower emhttpd: read SMART /dev/sdk May 18 17:47:36 Tower emhttpd: spinning down /dev/sdk May 18 18:57:41 Tower emhttpd: read SMART /dev/sdk May 18 19:12:31 Tower emhttpd: spinning down /dev/sdk May 18 20:22:36 Tower emhttpd: read SMART /dev/sdk May 18 20:37:26 Tower emhttpd: spinning down /dev/sdk May 18 21:47:32 Tower emhttpd: read SMART /dev/sdk May 18 22:02:22 Tower emhttpd: spinning down /dev/sdk May 18 23:12:26 Tower emhttpd: read SMART /dev/sdk May 18 23:27:15 Tower emhttpd: spinning down /dev/sdk May 19 00:37:20 Tower emhttpd: read SMART /dev/sdk May 19 00:52:10 Tower emhttpd: spinning down /dev/sdk May 19 02:02:12 Tower emhttpd: read SMART /dev/sdk May 19 02:17:02 Tower emhttpd: spinning down /dev/sdk May 19 03:27:05 Tower emhttpd: read SMART /dev/sdk May 19 03:41:55 Tower emhttpd: spinning down /dev/sdk May 19 04:52:01 Tower emhttpd: read SMART /dev/sdk May 19 05:06:50 Tower emhttpd: spinning down /dev/sdk May 19 06:16:55 Tower emhttpd: read SMART /dev/sdk May 19 06:31:44 Tower emhttpd: spinning down /dev/sdk May 19 07:41:48 Tower emhttpd: read SMART /dev/sdk May 19 07:56:37 Tower emhttpd: spinning down /dev/sdk I use /dev/sdk in this example, for the simple reason- it has no file system. It is literally just an empty drive which happens to be inserted into unraid. This drive has NO data. This drive has NO filesystem. It has no mounts. There are no docker containers with access to this device. Everything is isolated to /mnt/user or /mnt/cache shares. There are no VMs running. But, it spins up pretty consistently, every 50 minutes after it last spun down. I have.... 1. Updated the firmware / bios for my HBA, BIOS, etc. No Change. 2. Removed most of the plugins I thought could be contributing to this. So, Folder Caching, etc. 3. Trim schedule is only once a week. Mover schedule wouldn't apply to this disk. There is, nothing in syslog to assist, other than viewing the timing of the other drives randomly spinning up as well. May 19 04:50:08 Tower rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="1287" x-info="https://www.rsyslog.com"] rsyslogd was HUPed May 19 04:52:01 Tower emhttpd: read SMART /dev/sdk May 19 05:04:10 Tower emhttpd: read SMART /dev/sdm May 19 05:04:21 Tower emhttpd: read SMART /dev/sdn May 19 05:06:50 Tower emhttpd: spinning down /dev/sdk May 19 05:18:59 Tower emhttpd: spinning down /dev/sdm May 19 05:19:12 Tower emhttpd: spinning down /dev/sdn May 19 05:40:29 Tower emhttpd: read SMART /dev/sde May 19 05:55:19 Tower emhttpd: spinning down /dev/sde May 19 06:00:01 Tower root: mover: started May 19 06:00:01 Tower root: mover: finished May 19 06:16:55 Tower emhttpd: read SMART /dev/sdk May 19 06:29:02 Tower emhttpd: read SMART /dev/sdm May 19 06:29:14 Tower emhttpd: read SMART /dev/sdn May 19 06:31:44 Tower emhttpd: spinning down /dev/sdk May 19 06:43:52 Tower emhttpd: spinning down /dev/sdm May 19 06:44:06 Tower emhttpd: spinning down /dev/sdn May 19 07:20:11 Tower emhttpd: spinning down /dev/sde May 19 07:41:48 Tower emhttpd: read SMART /dev/sdk May 19 07:53:57 Tower emhttpd: read SMART /dev/sdm May 19 07:54:09 Tower emhttpd: read SMART /dev/sdn May 19 07:56:37 Tower emhttpd: spinning down /dev/sdk May 19 08:00:01 Tower root: mover: started May 19 08:00:01 Tower root: mover: finished May 19 08:08:47 Tower emhttpd: spinning down /dev/sdm May 19 08:08:59 Tower emhttpd: spinning down /dev/sdn May 19 08:30:17 Tower emhttpd: read SMART /dev/sde May 19 08:45:08 Tower emhttpd: spinning down /dev/sde For lsblk- note- no partitions. I would have uploaded the diagnostics.zip, but, it appears to have crashed after running sed -i against the nextcloud's trashbin directory...... BUT, I did run root@Tower:~# lsof -r 1 /dev/sdk | tee sdkaccess for a few hours And, came up with this. So, the only process accessing that disk, is hdparm. Disk's settings: Unraid Disk Settings:

This was emailed in as well. #25089 From the remote host- root@kube01:~# mount -t nfs4 -o nfsvers=4.0,actimeo=1,soft,timeo=300,retry=2 10.100.5.2:/mnt/user/longhorn-backups test/ mount.nfs4: mounting 10.100.5.2:/mnt/user/longhorn-backups failed, reason given by server: No such file or directory From Unraid root@Tower:/mnt/user# cat /etc/exports ... "/mnt/user/longhorn-backups" -fsid=105,async,no_subtree_check 10.100.5.0/24(sec=sys,rw,insecure,anongid=100,anonuid=99,all_squash) ... root@Tower:/mnt/user# ls -al /mnt/user longhorn-backups -> /mnt/main/longhorn-backups/ NFS Shares: root@Tower:/mnt/main/home/root# ./showNfsClients v3 clients: (via /var/lib/nfs/rmtab) 10.100.5.100:/mnt/main/photos:0x00000001 10.100.5.105:/mnt/main/nextcloud:0x00000001 10.100.5.105:/mnt/cache/scanned:0x00000001 10.100.5.105:/mnt/main/documents:0x00000001 v4 clients: (via /proc/fs/nfsd/clients) address: "10.100.5.100:798" name: "Linux NFSv4.2 longhorn-manager-j2wk7" address: "10.100.5.106:847" name: "Linux NFSv4.0 longhorn-manager-hgtt8/10.100.5.2" address: "10.100.5.105:681" name: "Linux NFSv4.2 instance-manager-r-23866f720a1af5bf2236875845f4f132" address: "10.100.5.105:701" name: "Linux NFSv4.2 longhorn-manager-7txwq" address: "10.100.5.106:909" name: "Linux NFSv4.2 instance-manager-r-9613737d8c4e4685af4dda69f50a4f50" address: "10.100.5.100:659" name: "Linux NFSv4.2 instance-manager-r-ce2be797b831f1d30f228947e0bec7a3" To note, this ONLY impacts the following conditions: 1. Remote client is using NFSv4, instead of NFSv3. 2. Share, is "Exclusive share" added in rc6. 3. ZFS may, or may not be a factor here. I do believe this issue is due to nfsv4 not liking the symlink. Adding a secondary storage, converting back to a FUSE/SHFS mount, addresses issue.

Had a share named temp. Main storage was zfs pool, named "cache". Secondary storage, was the main unraid array. Set the secondary storage to "none", and clicked saved. Instead, of expected confirmation, received popup message saying share was deleted. Found broken symlink under /mnt/user for temp, pointing to non-existent directory /mnt/cache/temp. What I believe happened here- There were no files on the "cache" pool. The ZFS dataset didn't even exist yet, as I had not written anything to it. There were, however files on the unraid array. So, when I remove the secondary storage, there was "nothing", hence the confusing message. However, the files still do exist on /mnt/disk1/temp, as I would expect. TLDR / To Reproduce- Create a share on the array. Add a few files. Update the share, to have primary storage pointing at a zfs pool, and secondary storage, on the array, with mover action set move files to array. Save. Do note, zfs list will not yet show the dataset, until something has been written to it. But, dont' write anything to it. Instead, update the share again. Remove the secondary storage (array). Save. You will be presented with a broken symlink, and a "Deleted" dialog.

Had a random issue- Went to check my dashboard, and nothing was working properly. Tons of errors when trying to look at docker/vms/etc... Under the array, noticed it was rather weird, my ssd had an impossible number of writes. SSH-d into the host, discovered /boot was not mounted. So- I assumed my brand new flash drive somehow ended up corrupted.... So, i copied a backup of my flash drive, and got ready to start the process of moving everything over. After getting my local backup flash drive ready, I rebooted the host. Acts like nothing at all was wrong. Works fine now. I do have the syslog configured to record to a Splunk instance, and I did attach the files, if they are of use to anybody... or can be used to identify a bug-fix for a future issue. log_dump_3.txt