Hello, 

 

I'm moving data from an old linux NAS to unRAID 6.8.3, in the third day got this error:

 

[Sat May 16 19:48:37 2020] shfs[12108]: segfault at 10 ip 000014bb1a525624 sp 000014badb5bac50 error 4 in libfuse3.so.3.9.0[14bb1a521000+18000]
[Sat May 16 19:48:37 2020] Code: 7d 68 4c 89 ff e8 ec c7 ff ff 8b 85 00 01 00 00 85 c0 0f 85 4e 01 00 00 4c 89 ee 48 89 ef e8 83 d7 ff ff 4c 89 ff 48 8b 40 20 <4c> 8b 68 10 e8 43 c1 ff ff 45 31 c0 48 8d 4c 24 18 31 d2 4c 89 ee
[Sat May 16 19:48:37 2020] ------------[ cut here ]------------
[Sat May 16 19:48:37 2020] nfsd: non-standard errno: -103
[Sat May 16 19:48:37 2020] WARNING: CPU: 7 PID: 29651 at fs/nfsd/nfsproc.c:820 nfserrno+0x47/0x4f [nfsd]
[Sat May 16 19:48:37 2020] Modules linked in: nfsd lockd grace sunrpc xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs md_mod bonding pcc_cpufreq virtio_net net_failover i2c_i801 i2c_core mpt3sas ahci failover libahci intel_agp raid_class intel_gtt scsi_transport_sas virtio_scsi virtio_console agpgart button [last unloaded: md_mod]
[Sat May 16 19:48:37 2020] CPU: 7 PID: 29651 Comm: nfsd Not tainted 4.19.107-Unraid #1
[Sat May 16 19:48:37 2020] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[Sat May 16 19:48:37 2020] RIP: 0010:nfserrno+0x47/0x4f [nfsd]
[Sat May 16 19:48:37 2020] Code: ff c0 48 83 f8 22 75 e1 80 3d 9a 06 01 00 00 41 bc 00 00 00 05 75 15 48 c7 c7 e3 c9 25 a0 c6 05 84 06 01 00 01 e8 eb f1 df e0 <0f> 0b 44 89 e0 41 5c c3 48 83 ec 18 31 c9 ba ff 07 00 00 65 48 8b
[Sat May 16 19:48:37 2020] RSP: 0018:ffffc90000c73d20 EFLAGS: 00010282
[Sat May 16 19:48:37 2020] RAX: 0000000000000000 RBX: ffff888176594c08 RCX: 0000000000000007
[Sat May 16 19:48:37 2020] RDX: 00000000000004bd RSI: 0000000000000002 RDI: ffff88817bbd64f0
[Sat May 16 19:48:37 2020] RBP: ffffc90000c73de0 R08: 0000000000000003 R09: 0000000000016700
[Sat May 16 19:48:37 2020] R10: 0000000000000000 R11: 0000000000000040 R12: 0000000005000000
[Sat May 16 19:48:37 2020] R13: 0000000000000011 R14: ffff888176594c08 R15: ffff88800b1a8600
[Sat May 16 19:48:37 2020] FS:  0000000000000000(0000) GS:ffff88817bbc0000(0000) knlGS:0000000000000000
[Sat May 16 19:48:37 2020] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sat May 16 19:48:37 2020] CR2: 0000000000000010 CR3: 0000000176728000 CR4: 00000000000006e0
[Sat May 16 19:48:37 2020] Call Trace:
[Sat May 16 19:48:37 2020]  fill_pre_wcc+0x6b/0x155 [nfsd]
[Sat May 16 19:48:37 2020]  nfsd_create+0xd8/0x182 [nfsd]
[Sat May 16 19:48:37 2020]  nfsd3_proc_mkdir+0x9b/0xdf [nfsd]
[Sat May 16 19:48:37 2020]  nfsd_dispatch+0xb2/0x163 [nfsd]
[Sat May 16 19:48:37 2020]  svc_process+0x4fd/0x6b7 [sunrpc]
[Sat May 16 19:48:37 2020]  nfsd+0xea/0x141 [nfsd]
[Sat May 16 19:48:37 2020]  ? nfsd_destroy+0x48/0x48 [nfsd]
[Sat May 16 19:48:37 2020]  kthread+0x10c/0x114
[Sat May 16 19:48:37 2020]  ? kthread_park+0x89/0x89
[Sat May 16 19:48:37 2020]  ret_from_fork+0x35/0x40
[Sat May 16 19:48:37 2020] ---[ end trace 84f742d85a5c98c6 ]---

And clients cannot access the FS anymore. Probably I can stop/start disks, but I'm concerned there's something fishy hidden, any hints?

 

The disks don't report any error at this point in time (no apparent physical failure):

 

nas03-diagnostics-20200516-2013.zip

Edited May 17, 20206 yr by cyruspy

You'll likely need to restart the server, segfault appears to be NFS related, so disable if not in use.

NFS is my main access protocol. Restarted the VM and it's working again.

 

Had a remote Nextcloud server accessing a share while it crashed. Disabled cache for its share, at least for the moment (maybe a data mover thing?).

Did you ever figure a fix for this?

 

I've been running up against this too.

 

NFS completely crashes, my only options are reboot the server, or a "better" solution is to just stop and restart the array.

 

If I was to guess, I'd have to say its related to caching as well.   I can reproduce the problem at will by copying a few dozen files (a few GB), and then using my Linux desktop file manager (Pop!_OS "files" which I believe is AKA Nautilus) try to delete the files/directories, and hard crash of the NFS.    no shares show up after that in GUI or CLI mode.   Even if I log into my unRAID box, the shares are borked, again in GUI/web or in CLI mode.

 

[1818073.719919] shfs[11335]: segfault at 10 ip 0000151ab370c4f1 sp 0000151a73ffeba0 error 4 in libfuse3.so.3.10.5[151ab3708000+19000]
[1818073.719940] Code: e8 44 c9 ff ff 8b 85 00 01 00 00 85 c0 0f 85 b6 01 00 00 4c 89 f6 48 89 ef 31 db e8 19 dc ff ff 4c 89 ef 45 31 ed 48 8b 40 20 <4c> 8b 70 10 e8 86 c2 ff ff 48 8d 4c 24 18 45 31 c0 31 d2 4c 89 f6

 

After several months without problems my server crashed this night with a similar error (shfs segfault error in combination with nfs):

 

Nov 16 03:02:10 horus emhttpd: spinning down /dev/sde
Nov 16 03:34:15 horus kernel: shfs[9050]: segfault at 10 ip 0000146e69c975c2 sp 0000146e69288c20 error 4 in libfuse3.so.3.12.0[146e69c93000+19000] likely on CPU 5 (core 5, socket 0)
Nov 16 03:34:15 horus kernel: Code: f4 c8 ff ff 8b b3 08 01 00 00 85 f6 0f 85 46 01 00 00 4c 89 ee 48 89 df 45 31 ff e8 18 dc ff ff 4c 89 e7 45 31 e4 48 8b 40 20 <4c> 8b 68 10 e8 15 c2 ff ff 48 8d 4c 24 18 45 31 c0 31 d2 4c 89 ee
Nov 16 03:34:15 horus kernel: ------------[ cut here ]------------
Nov 16 03:34:15 horus kernel: nfsd: non-standard errno: -103
Nov 16 03:34:15 horus kernel: WARNING: CPU: 5 PID: 5142 at fs/nfsd/vfs.c:111 nfserrno+0x45/0x51 [nfsd]
Nov 16 03:34:15 horus kernel: Modules linked in: xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) it87(O) hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 kvm iosf_mbi drm_buddy ttm crct10dif_pclmul crc32_pclmul crc32c_intel drm_display_helper ghash_clmulni_intel sha512_ssse3 drm_kms_helper sha256_ssse3 sha1_ssse3 mei_pxp mei_hdcp gigabyte_wmi wmi_bmof aesni_intel drm crypto_simd igb cryptd i2c_algo_bit intel_gtt i2c_i801 agpgart rapl syscopyarea mpt3sas intel_cstate nvme i2c_smbus sysfillrect mei_me intel_uncore raid_class ahci libahci scsi_transport_sas intel_pch_thermal sysimgblt i2c_core nvme_core fb_sys_fops mei thermal fan video wmi backlight intel_pmc_core acpi_pad button unix
Nov 16 03:34:15 horus kernel: CPU: 5 PID: 5142 Comm: nfsd Tainted: P           O       6.1.106-Unraid #1
Nov 16 03:34:15 horus kernel: Hardware name: Gigabyte Technology Co., Ltd. C246N-WU2/C246N-WU2-CF, BIOS F4 12/21/2023
Nov 16 03:34:15 horus kernel: RIP: 0010:nfserrno+0x45/0x51 [nfsd]
Nov 16 03:34:15 horus kernel: Code: c3 cc cc cc cc 48 ff c0 48 83 f8 26 75 e0 80 3d 41 b6 05 00 00 75 15 48 c7 c7 e6 6a 0f a1 c6 05 31 b6 05 00 01 e8 44 a2 fa df <0f> 0b b8 00 00 00 05 c3 cc cc cc cc 0f 1f 44 00 00 41 57 41 56 41
Nov 16 03:34:15 horus kernel: RSP: 0018:ffffc90001347bb8 EFLAGS: 00010286
Nov 16 03:34:15 horus kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000003
Nov 16 03:34:15 horus kernel: RDX: 0000000000000002 RSI: 0000000000000003 RDI: 00000000ffffffff
Nov 16 03:34:15 horus kernel: RBP: ffffc90001347d90 R08: 0000000000000000 R09: ffffffff82245fb0
Nov 16 03:34:15 horus kernel: R10: 00007fffffffffff R11: ffffffff8296fd6e R12: 000000000000000b
Nov 16 03:34:15 horus kernel: R13: ffff88813662a4a0 R14: ffffffff82a1af40 R15: 000000000010011a
Nov 16 03:34:15 horus kernel: FS:  0000000000000000(0000) GS:ffff88902e540000(0000) knlGS:0000000000000000
Nov 16 03:34:15 horus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 03:34:15 horus kernel: CR2: 000000000052adc4 CR3: 000000000420a006 CR4: 00000000003706e0
Nov 16 03:34:15 horus kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 16 03:34:15 horus kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 16 03:34:15 horus kernel: Call Trace:
Nov 16 03:34:15 horus kernel: <TASK>
Nov 16 03:34:15 horus kernel: ? __warn+0xab/0x122
Nov 16 03:34:15 horus kernel: ? report_bug+0x109/0x17e
Nov 16 03:34:15 horus kernel: ? nfserrno+0x45/0x51 [nfsd]
Nov 16 03:34:15 horus kernel: ? handle_bug+0x41/0x6f
Nov 16 03:34:15 horus kernel: ? exc_invalid_op+0x13/0x60
Nov 16 03:34:15 horus kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 16 03:34:15 horus kernel: ? nfserrno+0x45/0x51 [nfsd]
Nov 16 03:34:15 horus kernel: ? nfserrno+0x45/0x51 [nfsd]
Nov 16 03:34:15 horus kernel: nfsd4_encode_fattr+0x12b6/0x131c [nfsd]
Nov 16 03:34:15 horus kernel: ? slab_post_alloc_hook+0x4d/0x15e
Nov 16 03:34:15 horus kernel: ? nfsd_setuser+0x18b/0x1ab [nfsd]
Nov 16 03:34:15 horus kernel: ? nfsd_setuser_and_check_port+0x76/0xb4 [nfsd]
Nov 16 03:34:15 horus kernel: ? nfsd_setuser_and_check_port+0x76/0xb4 [nfsd]
Nov 16 03:34:15 horus kernel: nfsd4_encode_getattr+0x28/0x2e [nfsd]
Nov 16 03:34:15 horus kernel: nfsd4_encode_operation+0xa6/0x20a [nfsd]
Nov 16 03:34:15 horus kernel: nfsd4_proc_compound+0x1cc/0x57f [nfsd]
Nov 16 03:34:15 horus kernel: nfsd_dispatch+0x1db/0x28f [nfsd]
Nov 16 03:34:15 horus kernel: svc_process_common+0x300/0x511 [sunrpc]
Nov 16 03:34:15 horus kernel: ? ktime_get+0x35/0x49
Nov 16 03:34:15 horus kernel: ? nfsd_svc+0x2cf/0x2cf [nfsd]
Nov 16 03:34:15 horus kernel: ? nfsd_shutdown_threads+0x5b/0x5b [nfsd]
Nov 16 03:34:15 horus kernel: svc_process+0xcc/0xe9 [sunrpc]
Nov 16 03:34:15 horus kernel: nfsd+0x96/0x108 [nfsd]
Nov 16 03:34:15 horus kernel: kthread+0xe4/0xef
Nov 16 03:34:15 horus kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov 16 03:34:15 horus kernel: ret_from_fork+0x1f/0x30
Nov 16 03:34:15 horus kernel: </TASK>
Nov 16 03:34:15 horus kernel: ---[ end trace 0000000000000000 ]---
Nov 16 03:37:40 horus rpc.mountd[5153]: Cannot export /mnt/user/ShareA, possibly unsupported filesystem or fsid= required
Nov 16 03:42:01 horus rpc.mountd[5153]: Cannot export /mnt/user/ShareB, possibly unsupported filesystem or fsid= required
Nov 16 03:50:59 horus emhttpd: spinning down /dev/sdd

I'm using NFS between two Unraid servers (heavy traffic all the time).

 

Any solution yet?

 

Nov 21 09:58:14 Fatsally kernel: shfs[12430]: segfault at 10 ip 000014cf6d31b5c2 sp 000014cf6c2e4c20 error 4 in libfuse3.so.3.12.0[14cf6d317000+19000] likely on CPU 4 (core 5, socket 0)
Nov 21 09:58:14 Fatsally kernel: Code: f4 c8 ff ff 8b b3 08 01 00 00 85 f6 0f 85 46 01 00 00 4c 89 ee 48 89 df 45 31 ff e8 18 dc ff ff 4c 89 e7 45 31 e4 48 8b 40 20 <4c> 8b 68 10 e8 15 c2 ff ff 48 8d 4c 24 18 45 31 c0 31 d2 4c 89 ee
Nov 21 09:58:14 Fatsally kernel: ------------[ cut here ]------------
Nov 21 09:58:14 Fatsally kernel: nfsd: non-standard errno: -103
Nov 21 09:58:14 Fatsally kernel: WARNING: CPU: 16 PID: 3954 at fs/nfsd/vfs.c:111 nfserrno+0x45/0x51 [nfsd]
Nov 21 09:58:14 Fatsally kernel: Modules linked in: rpcsec_gss_krb5 nf_tables xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth ipvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nvidia_uvm(PO) xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) i915 drm_buddy i2c_algo_bit ttm drm_display_helper intel_gtt agpgart tcp_diag inet_diag wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls e1000e atlantic nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel nvidia(PO)
Nov 21 09:58:14 Fatsally kernel: kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel video crypto_simd cryptd rapl drm_kms_helper btusb btrtl btbcm btintel intel_cstate nvme drm bluetooth mxm_wmi intel_wmi_thunderbolt mpt3sas intel_uncore thunderbolt nvme_core backlight i2c_i801 syscopyarea mei_me sysfillrect i2c_smbus raid_class sysimgblt fb_sys_fops i2c_core scsi_transport_sas mei ahci input_leds libahci ecdh_generic joydev led_class ecc tpm_crb tpm_tis tpm_tis_core tpm wmi acpi_tad button unix [last unloaded: e1000e]
Nov 21 09:58:14 Fatsally kernel: CPU: 16 PID: 3954 Comm: nfsd Tainted: P           O       6.1.106-Unraid #1
Nov 21 09:58:14 Fatsally kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X299 Creator, BIOS L1.52C 08/05/2022
Nov 21 09:58:14 Fatsally kernel: RIP: 0010:nfserrno+0x45/0x51 [nfsd]
Nov 21 09:58:14 Fatsally kernel: Code: c3 cc cc cc cc 48 ff c0 48 83 f8 26 75 e0 80 3d 41 b6 05 00 00 75 15 48 c7 c7 e6 0a 24 a5 c6 05 31 b6 05 00 01 e8 44 02 e6 db <0f> 0b b8 00 00 00 05 c3 cc cc cc cc 0f 1f 44 00 00 41 57 41 56 41
Nov 21 09:58:14 Fatsally kernel: RSP: 0018:ffffc90000e1fd38 EFLAGS: 00010282
Nov 21 09:58:14 Fatsally kernel: RAX: 0000000000000000 RBX: ffff888173b8e028 RCX: 0000000000000027
Nov 21 09:58:14 Fatsally kernel: RDX: 0000000000000002 RSI: ffffffff820d99e1 RDI: 00000000ffffffff
Nov 21 09:58:14 Fatsally kernel: RBP: ffffc90000e1fd80 R08: 0000000000000000 R09: ffffffff829573f0
Nov 21 09:58:14 Fatsally kernel: R10: 00003fffffffffff R11: ffff88909ffbd0ce R12: ffff888173b8e100
Nov 21 09:58:14 Fatsally kernel: R13: ffff888108610340 R14: 0000000000000080 R15: ffff88810e3ea3c0
Nov 21 09:58:14 Fatsally kernel: FS:  0000000000000000(0000) GS:ffff889060000000(0000) knlGS:0000000000000000
Nov 21 09:58:14 Fatsally kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 21 09:58:14 Fatsally kernel: CR2: 0000146bd27edf88 CR3: 000000000220a002 CR4: 00000000003726e0
Nov 21 09:58:14 Fatsally kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 09:58:14 Fatsally kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 21 09:58:14 Fatsally kernel: Call Trace:
Nov 21 09:58:14 Fatsally kernel: <TASK>
Nov 21 09:58:14 Fatsally kernel: ? __warn+0xab/0x122
Nov 21 09:58:14 Fatsally kernel: ? report_bug+0x109/0x17e
Nov 21 09:58:14 Fatsally kernel: ? nfserrno+0x45/0x51 [nfsd]
Nov 21 09:58:14 Fatsally kernel: ? handle_bug+0x41/0x6f
Nov 21 09:58:14 Fatsally kernel: ? exc_invalid_op+0x13/0x60
Nov 21 09:58:14 Fatsally kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 21 09:58:14 Fatsally kernel: ? nfserrno+0x45/0x51 [nfsd]
Nov 21 09:58:14 Fatsally kernel: ? nfserrno+0x45/0x51 [nfsd]
Nov 21 09:58:14 Fatsally kernel: fh_getattr+0x43/0x61 [nfsd]
Nov 21 09:58:14 Fatsally kernel: fh_fill_post_attrs+0x55/0xa4 [nfsd]
Nov 21 09:58:14 Fatsally kernel: nfsd_unlink+0x1bd/0x250 [nfsd]
Nov 21 09:58:14 Fatsally kernel: nfsd4_remove+0x4f/0x6e [nfsd]
Nov 21 09:58:14 Fatsally kernel: nfsd4_proc_compound+0x444/0x57f [nfsd]
Nov 21 09:58:14 Fatsally kernel: nfsd_dispatch+0x1db/0x28f [nfsd]
Nov 21 09:58:14 Fatsally kernel: svc_process_common+0x300/0x511 [sunrpc]
Nov 21 09:58:14 Fatsally kernel: ? ktime_get+0x35/0x49
Nov 21 09:58:14 Fatsally kernel: ? nfsd_svc+0x2cf/0x2cf [nfsd]
Nov 21 09:58:14 Fatsally kernel: ? nfsd_shutdown_threads+0x5b/0x5b [nfsd]
Nov 21 09:58:14 Fatsally kernel: svc_process+0xcc/0xe9 [sunrpc]
Nov 21 09:58:14 Fatsally kernel: nfsd+0x96/0x108 [nfsd]
Nov 21 09:58:14 Fatsally kernel: kthread+0xe4/0xef
Nov 21 09:58:14 Fatsally kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov 21 09:58:14 Fatsally kernel: ret_from_fork+0x1f/0x30
Nov 21 09:58:14 Fatsally kernel: </TASK>
Nov 21 09:58:14 Fatsally kernel: ---[ end trace 0000000000000000 ]---
Nov 21 09:58:14 Fatsally rsyslogd: file '/mnt/user/appdata/syslog-192.168.88.217.log'[2] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Transport endpoint is not connected [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Nov 21 09:58:14 Fatsally rsyslogd: file '/mnt/user/appdata/syslog-192.168.88.217.log': open error: Transport endpoint is not connected [v8.2102.0 try https://www.rsyslog.com/e/2433 ]

 

To rule it out, have you run memtest for at least a single pass?