Shares disappear randomly when accessing through NFS

[ruepel]

hi!

 

My unraid server runs on 6.6.7 and my client is a Manjaro desktop with a wired connection to the server. I have mounted the shares with autofs and it works flawlessly via NFS ... except when it doesn't.

 

Sometimes, seemingly randomly, the entire file sharing would crash and all the shares are gone. Only a system reboot off the unraid system will bring them back. I have noticed that it only occurs when working on files in the file explorer pcmanfm and I believe ranger. It seems something borks up when moving, copying, or deleting files. It doesn't happen all the time. Most of the time I can perform file operations in pcmanfm or ranger without a problem.

 

I have no idea whether it is client or server related. I hope the diagnostic file might help. Thanks for helping!

vulcan-diagnostics-20190505-1735.zip

[ruepel]

still seeing the error. this time moving files in Dolphin resulted in nfsd crashing. I really hope, someone can help me. file management in the unraid console is no fun.

 

Interesting observation: SMB shares weren't affected at all.

 

this is part of the log when it happended:

Jun 20 14:50:47 vulcan rpcbind[38947]: connect from 10.10.10.207 to getport/addr(nlockmgr)
Jun 20 15:02:10 vulcan rpcbind[43080]: connect from 10.10.10.207 to getport/addr(nlockmgr)
Jun 20 15:02:10 vulcan rpcbind[43081]: connect from 127.0.0.1 to getport/addr(status)
Jun 20 15:02:10 vulcan rpcbind[43082]: connect from 127.0.0.1 to getport/addr(status)
Jun 20 15:03:11 vulcan rpcbind[43751]: connect from 10.10.10.207 to getport/addr(nlockmgr)
Jun 20 15:04:01 vulcan kernel: shfs[44161]: segfault at 10 ip 0000151606ddb381 sp 00001515d57adc10 error 4 in libfuse.so.2.9.9[151606dd7000+18000]
Jun 20 15:04:01 vulcan kernel: Code: e8 94 cb ff ff 8b 83 cc 00 00 00 85 c0 0f 85 4e 01 00 00 4c 89 e6 48 89 df 4c 8d 7c 24 60 e8 b6 db ff ff 48 89 ef 48 8b 40 20 <4c> 8b 60 10 e8 36 c4 ff ff 45 31 c0 48 8d 4c 24 18 31 d2 4c 89 e6
Jun 20 15:04:01 vulcan kernel: ------------[ cut here ]------------
Jun 20 15:04:01 vulcan kernel: nfsd: non-standard errno: -103
Jun 20 15:04:01 vulcan kernel: WARNING: CPU: 3 PID: 7295 at fs/nfsd/nfsproc.c:820 nfserrno+0x44/0x4a [nfsd]
Jun 20 15:04:01 vulcan kernel: Modules linked in: xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 xt_nat iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap veth ipt_MASQUERADE iptable_nat nf_nat_ipv4 iptable_filter ip_tables nf_nat dm_crypt algif_skcipher af_alg dm_mod dax nfsd lockd grace sunrpc md_mod ipmi_devintf bonding netxen_nic tg3 sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ipmi_ssif i2c_core intel_cstate intel_uncore acpi_power_meter intel_rapl_perf ahci libahci wmi megaraid_sas button ipmi_si pcc_cpufreq [last unloaded: netxen_nic]
Jun 20 15:04:01 vulcan kernel: CPU: 3 PID: 7295 Comm: nfsd Not tainted 4.19.41-Unraid #1
Jun 20 15:04:01 vulcan kernel: Hardware name: Dell Inc. PowerEdge R520/051XDX, BIOS 2.5.1 02/08/2018
Jun 20 15:04:01 vulcan kernel: RIP: 0010:nfserrno+0x44/0x4a [nfsd]
Jun 20 15:04:01 vulcan kernel: Code: c0 48 83 f8 22 75 e2 80 3d a7 06 01 00 00 bb 00 00 00 05 75 17 89 fe 48 c7 c7 bb a9 16 a0 c6 05 90 06 01 00 01 e8 02 f6 ee e0 <0f> 0b 89 d8 5b c3 48 83 ec 18 31 c9 ba ff 07 00 00 65 48 8b 04 25
Jun 20 15:04:01 vulcan kernel: RSP: 0018:ffffc90003bb7e20 EFLAGS: 00010282
Jun 20 15:04:01 vulcan kernel: RAX: 0000000000000000 RBX: 0000000005000000 RCX: 0000000000000007
Jun 20 15:04:01 vulcan kernel: RDX: 0000000000000006 RSI: ffff88881f6564f0 RDI: ffff88881f6564f0
Jun 20 15:04:01 vulcan kernel: RBP: ffffc90003bb7e50 R08: 0000000000000003 R09: 0000000000020300
Jun 20 15:04:01 vulcan kernel: R10: 000000000000063b R11: 0000000000020990 R12: ffff888675796018
Jun 20 15:04:01 vulcan kernel: R13: ffffffffa016a4d0 R14: 000000000000001c R15: ffffffffa016a220
Jun 20 15:04:01 vulcan kernel: FS:  0000000000000000(0000) GS:ffff88881f640000(0000) knlGS:0000000000000000
Jun 20 15:04:01 vulcan kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 20 15:04:01 vulcan kernel: CR2: 000014568cf72d00 CR3: 0000000001e0a005 CR4: 00000000000626e0
Jun 20 15:04:01 vulcan kernel: Call Trace:
Jun 20 15:04:01 vulcan kernel: fill_post_wcc+0x69/0x122 [nfsd]
Jun 20 15:04:01 vulcan kernel: nfsd3_proc_rmdir+0x6e/0x96 [nfsd]
Jun 20 15:04:01 vulcan kernel: nfsd_dispatch+0xb4/0x169 [nfsd]
Jun 20 15:04:01 vulcan kernel: svc_process+0x4fc/0x6b6 [sunrpc]
Jun 20 15:04:01 vulcan kernel: ? nfsd_destroy+0x48/0x48 [nfsd]
Jun 20 15:04:01 vulcan kernel: nfsd+0xeb/0x142 [nfsd]
Jun 20 15:04:01 vulcan kernel: kthread+0x10b/0x113
Jun 20 15:04:01 vulcan kernel: ? kthread_park+0x71/0x71
Jun 20 15:04:01 vulcan kernel: ret_from_fork+0x35/0x40
Jun 20 15:04:01 vulcan kernel: ---[ end trace fdda59ca4cc54630 ]---
Jun 20 15:04:52 vulcan emhttpd: req (2): startState=STARTED&file=&optionCorrect=correct&csrf_token=****************&cmdStop=Stop
Jun 20 15:04:52 vulcan kernel: mdcmd (59): nocheck cancel
Jun 20 15:04:53 vulcan emhttpd: Spinning up all drives...
Jun 20 15:04:53 vulcan emhttpd: shcmd (12791): /usr/sbin/hdparm -S0 /dev/sdj
Jun 20 15:04:53 vulcan kernel: mdcmd (60): spinup 0
Jun 20 15:04:53 vulcan kernel: mdcmd (61): spinup 1
Jun 20 15:04:53 vulcan kernel: mdcmd (62): spinup 2
Jun 20 15:04:53 vulcan kernel: mdcmd (63): spinup 3
Jun 20 15:04:53 vulcan kernel: mdcmd (64): spinup 4
Jun 20 15:04:53 vulcan kernel: mdcmd (65): spinup 5
Jun 20 15:04:53 vulcan kernel: mdcmd (66): spinup 6
Jun 20 15:04:53 vulcan kernel: mdcmd (67): spinup 29
Jun 20 15:04:53 vulcan root: 
Jun 20 15:04:53 vulcan root: /dev/sdj:
Jun 20 15:04:53 vulcan root:  setting standby to 0 (off)
Jun 20 15:04:56 vulcan emhttpd: Stopping services...
Jun 20 15:04:56 vulcan emhttpd: shcmd (12793): /etc/rc.d/rc.libvirt stop
Jun 20 15:04:56 vulcan root: Domain 5cc43df6-aaa3-0d66-e14a-fb15375bf33c is being shutdown
Jun 20 15:04:56 vulcan root: 
Jun 20 15:05:57 vulcan root: Waiting on VMs to shutdown............................................................

 

[ruepel]

could someone please take a look at this? it is still an issue and very, very annoying. thank you!

[trurl]

It looks like those diagnostics were taken immediately after rebooting, and in fact before the array had completely started, since they only show one mounted disk.

 

Please post new diagnostics.

[ruepel]

will do as soon as it happens again -- rebooted the server earlier without saving the diagnostic file first

 

I guess we can close this thread and continue in @digitalformula's thread:
https://forums.unraid.net/topic/81481-shares-disappearing-still-happening/

[trurl]

1 hour ago, ruepel said:

will do as soon as it happens again -- rebooted the server earlier without saving the diagnostic file first

 

I guess we can close this thread and continue in @digitalformula's thread:
https://forums.unraid.net/topic/81481-shares-disappearing-still-happening/

It would really be better if you let that thread support that user and not hijack it for your support. Maybe they are similar but that remains to be seen. Lets look at his diagnostics on his thread and not get them mixed up with your diagnostics on his thread. If it turns out that there is useful information between the threads we can link to them.

[digitalformula]

On 7/20/2019 at 8:41 AM, trurl said:

It would really be better if you let that thread support that user and not hijack it for your support. Maybe they are similar but that remains to be seen. Lets look at his diagnostics on his thread and not get them mixed up with your diagnostics on his thread. If it turns out that there is useful information between the threads we can link to them.

FWIW, the symptoms are identical.