Why is my server crashing? "nfsd: non-standard errno: -103"

[Howboys]

Happened twice today. The first time, I just shutdown the server and made sure all disks were properly cabled. Everything was fine.

 

A few hours later (just now), the server is unresponse and I see this in the syslog:

 

Mar 25 09:36:37 Tower elogind-daemon[1352]: Removed session c33.
Mar 25 09:57:29 Tower kernel: shfs[10680]: segfault at 10 ip 00001480d2ee85c2 sp 00001480d27dcc20 error 4 in libfuse3.so.3.12.0[1480d2ee4000+19000]
Mar 25 09:57:29 Tower kernel: Code: f4 c8 ff ff 8b b3 08 01 00 00 85 f6 0f 85 46 01 00 00 4c 89 ee 48 89 df 45 31 ff e8 18 dc ff ff 4c 89 e7 45 31 e4 48 8b 40 20 <4c> 8b 68 10 e8 15 c2 ff ff 48 8d 4c 24 18 45 31 c0 31 d2 4c 89 ee
Mar 25 09:57:29 Tower kernel: ------------[ cut here ]------------
Mar 25 09:57:29 Tower kernel: nfsd: non-standard errno: -103
Mar 25 09:57:29 Tower kernel: WARNING: CPU: 14 PID: 10307 at fs/nfsd/nfsproc.c:889 nfserrno+0x45/0x51 [nfsd]
Mar 25 09:57:29 Tower kernel: Modules linked in: rpcsec_gss_krb5 xt_mark tcp_diag udp_diag inet_diag nf_tables nvidia_uvm(PO) veth tun xt_nat xt_tcpudp ipvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm nvme i2c_piix4 edac_mce_amd edac_core kvm_amd kvm gigabyte_wmi wmi_bmof crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel backlight crypto_simd cryptd mpt3sas rapl r8169 i2c_core nvme_core raid_class k10temp syscopyarea ccp ahci scsi_transport_sas sysfillrect sysimgblt realtek fb_sys_fops libahci wmi thermal button acpi_cpufreq unix
Mar 25 09:57:29 Tower kernel: CPU: 14 PID: 10307 Comm: nfsd Tainted: P S         O      5.19.17-Unraid #2
Mar 25 09:57:29 Tower kernel: Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F63a 02/17/2022
Mar 25 09:57:29 Tower kernel: RIP: 0010:nfserrno+0x45/0x51 [nfsd]
Mar 25 09:57:29 Tower kernel: Code: e9 f3 bc 0c de 48 ff c0 48 83 f8 26 75 e0 80 3d bb 47 05 00 00 75 15 48 c7 c7 17 54 b6 a3 c6 05 ab 47 05 00 01 e8 42 57 ce dd <0f> 0b b8 00 00 00 05 e9 c2 bc 0c de 48 83 ec 18 31 c9 ba ff 07 00
Mar 25 09:57:29 Tower kernel: RSP: 0018:ffffc90000adfd58 EFLAGS: 00010282
Mar 25 09:57:29 Tower kernel: RAX: 0000000000000000 RBX: ffff8881b3448030 RCX: 0000000000000027
Mar 25 09:57:29 Tower kernel: RDX: 0000000000000001 RSI: ffffffff820d7be1 RDI: 00000000ffffffff
Mar 25 09:57:29 Tower kernel: RBP: ffffc90000adfda0 R08: 0000000000000000 R09: ffffffff828653f0
Mar 25 09:57:29 Tower kernel: R10: 00003fffffffffff R11: ffff88843f3427b6 R12: ffff8881b3448108
Mar 25 09:57:29 Tower kernel: R13: ffff8882ba3eb0c0 R14: 0000000000000080 R15: ffff8882ba3eb400
Mar 25 09:57:29 Tower kernel: FS:  0000000000000000(0000) GS:ffff88842eb80000(0000) knlGS:0000000000000000
Mar 25 09:57:29 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 25 09:57:29 Tower kernel: CR2: 000015284b7418a8 CR3: 00000001f97a0000 CR4: 00000000003506e0
Mar 25 09:57:29 Tower kernel: Call Trace:
Mar 25 09:57:29 Tower kernel: <TASK>
Mar 25 09:57:29 Tower kernel: fh_getattr+0x43/0x61 [nfsd]
Mar 25 09:57:29 Tower kernel: fh_fill_post_attrs+0x55/0xa4 [nfsd]
Mar 25 09:57:29 Tower kernel: fh_unlock+0x12/0x3b [nfsd]
Mar 25 09:57:29 Tower kernel: nfsd_unlink+0x13a/0x1b9 [nfsd]
Mar 25 09:57:29 Tower kernel: nfsd4_remove+0x4f/0x76 [nfsd]
Mar 25 09:57:29 Tower kernel: nfsd4_proc_compound+0x437/0x56c [nfsd]
Mar 25 09:57:29 Tower kernel: nfsd_dispatch+0x1a9/0x262 [nfsd]
Mar 25 09:57:29 Tower kernel: svc_process+0x3f1/0x5d6 [sunrpc]
Mar 25 09:57:29 Tower kernel: ? nfsd_svc+0x2b6/0x2b6 [nfsd]
Mar 25 09:57:29 Tower kernel: ? nfsd_shutdown_threads+0x5b/0x5b [nfsd]
Mar 25 09:57:29 Tower kernel: nfsd+0xd5/0x155 [nfsd]
Mar 25 09:57:29 Tower kernel: kthread+0xe7/0xef
Mar 25 09:57:29 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Mar 25 09:57:29 Tower kernel: ret_from_fork+0x22/0x30
Mar 25 09:57:29 Tower kernel: </TASK>
Mar 25 09:57:29 Tower kernel: ---[ end trace 0000000000000000 ]---

 

WebUI works, SSH works. However shares are not found:

 

root@Tower:~# ls /mnt/user
/bin/ls: cannot access '/mnt/user': Transport endpoint is not connected

 

Diags attached.

 

tower-diagnostics-20230325-1002.zip

 

Edited March 25, 2023 by Howboys

[Howboys]

Possibly related: 

 

[Howboys]

This seems to happen when I delete a large directory (tens of GB) at once.