• /mnt/user dissappeared


    Can0n
    • Retest Urgent

    i just got an error on my Plex docker that please make sure the drive is attached.  I found that /mnt/user is not showing in the docker container but when i do an ls -l in the CLI i have some weird permissions screenshots and diagnostics attached

    cli.png

    docker.png

    thor-diagnostics-20180920-1011.zip




    User Feedback

    Recommended Comments



    Your log if spammed with wrong csrf_token errors, which makes very hard to read, close any stale browser windows and repost diags if/when it happens again.

    Link to comment

    unassigned devices were hammering the logs i uninstalled the plugin before rebooting I will let you all apprised thanks

    Edited by Can0nfan
    Link to comment

    the plugin was it was specifically saying it was due to unassigned devices and as soon as the plugin was removed the crsf token alarms stopped

    Link to comment

    ok so closed out all the tabs for this server on my imac, iphone, Work computer and windows VM...logs stopped spamming me....thanks @trurl for the link that helped

    Link to comment
    1 hour ago, johnnie.black said:

    Your log if spammed with wrong csrf_token errors, which makes very hard to read, close any stale browser windows and repost diags if/when it happens again.

    thanks i didnt quite understand the error before...good to know that wrong token is stale browser tabs...i'll be more mindful of leaving them open from now on

    Link to comment

    Also upgrade to 6.6.0, have one time user/disk share unavailable, stop array faild too. No this problem in all 6.6 RC. Keep monitoring and parity need further check/sync.

    Link to comment

    @johnnie.black and @trurl user folder broken again, was working before i left work...30min later it is broken i want 6.5.3 back this needed more RC's than 4, i shouldnt need to reboot the whole server 4 times a day

     

    drwxrwxrwx  1 nobody users  38 Sep 20 10:53 cache/

    drwxrwxrwx 10 nobody users 162 Sep 20 10:53 disk1/

    drwxrwxrwx  4 nobody users  50 Sep 20 10:53 disk10/

    drwxrwxrwx  6 nobody users  90 Sep 20 10:53 disk2/

    drwxrwxrwx  5 nobody users  69 Sep 20 10:53 disk3/

    drwxrwxrwx  7 nobody users  85 Sep 20 10:53 disk4/

    drwxrwxrwx  8 nobody users 125 Sep 20 10:53 disk5/

    drwxrwxrwx  8 nobody users 123 Sep 20 10:53 disk6/

    drwxrwxrwx  8 nobody users 125 Sep 20 10:53 disk7/

    drwxrwxrwx 11 nobody users 179 Sep 20 10:53 disk8/

    drwxrwxrwx  6 nobody users  91 Sep 20 10:53 disk9/

    drwxrwxrwx  2 nobody users  40 Sep 20 12:23 disks/

    d?????????  ? ?      ?       ?            ? user/

    drwxrwxrwx  1 nobody users 162 Sep 20 10:53 user0/

    root@Thor:/mnt# 

     

     

    new diags attached.

    thor-diagnostics-20180920-1538.zip

    Edited by Can0nfan
    Link to comment

    @Can0nfan: Perhaps try booting in safe mode and see if you still have the same issue? It cut out the possibility of misconfigured plugins.

     

    Your log is (still) flooded with acpid events every 2 seconds but no longer because of csrf_token.

     

    Link to comment

    I'm having the same issue. I booted in safe mode. The mount point does not fail as long as I don't mount and utilize the nfs share. Once I've mounted the share, the share fails after about 1-3 hours of use.

     

    The process that fails is this:
    /usr/local/sbin/shfs /mnt/user -disks 63 2048000000 -o noatime,big_writes,allow_other -o remember=330 |& logger

     

    The only error I see in syslog is this:

    Sep 20 18:01:03 systemname root: mover: finished
    Sep 20 18:05:02 systemname rpcbind[18092]: connect from 10.10.10.18 to getport/addr(nlockmgr)
    Sep 20 18:10:02 systemname rpcbind[20809]: connect from 10.10.10.18 to getport/addr(nlockmgr)
    Sep 20 18:15:02 systemname rpcbind[23574]: connect from 10.10.10.18 to getport/addr(nlockmgr)
    Sep 20 18:20:01 systemname rpcbind[26396]: connect from 10.10.10.18 to getport/addr(nlockmgr)
    Sep 20 18:21:33 systemname kernel: ------------[ cut here ]------------
    Sep 20 18:21:33 systemname kernel: nfsd: non-standard errno: -107
    Sep 20 18:21:33 systemname kernel: WARNING: CPU: 0 PID: 3578 at fs/nfsd/nfsproc.c:817 nfserrno+0x44/0x4a [nfsd]
    Sep 20 18:21:33 systemname kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables veth xt_nat macvlan ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod sb_edac crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd isci libsas glue_helper vmxnet3 intel_rapl_perf e1000e intel_agp scsi_transport_sas i2c_piix4 i2c_core ahci intel_gtt libahci agpgart ata_piix button [last unloaded: kvm]
    Sep 20 18:21:33 systemname kernel: CPU: 0 PID: 3578 Comm: nfsd Not tainted 4.18.8-unRAID #1
    Sep 20 18:21:33 systemname kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
    Sep 20 18:21:33 systemname kernel: RIP: 0010:nfserrno+0x44/0x4a [nfsd]
    Sep 20 18:21:33 systemname kernel: Code: c0 48 83 f8 22 75 e2 80 3d b3 06 01 00 00 bb 00 00 00 05 75 17 89 fe 48 c7 c7 3b ea 24 a0 c6 05 9c 06 01 00 01 e8 8a 9c e0 e0 <0f> 0b 89 d8 5b c3 48 83 ec 18 31 c9 ba ff 07 00 00 65 48 8b 04 25 
    Sep 20 18:21:33 systemname kernel: RSP: 0018:ffffc90002283db8 EFLAGS: 00010286
    Sep 20 18:21:33 systemname kernel: RAX: 0000000000000000 RBX: 0000000005000000 RCX: 0000000000000007
    Sep 20 18:21:33 systemname kernel: RDX: 0000000000000000 RSI: ffff88042d616470 RDI: ffff88042d616470
    Sep 20 18:21:33 systemname kernel: RBP: ffffc90002283e08 R08: 0000000000000003 R09: ffff88043ff05e00
    Sep 20 18:21:33 systemname kernel: R10: 000000000000068e R11: 0000000000022eb4 R12: ffff8804286dac08
    Sep 20 18:21:33 systemname kernel: R13: ffff8804085bea58 R14: 0000000000000001 R15: ffffffffa024e2a0
    Sep 20 18:21:33 systemname kernel: FS: 0000000000000000(0000) GS:ffff88042d600000(0000) knlGS:0000000000000000
    Sep 20 18:21:33 systemname kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Sep 20 18:21:33 systemname kernel: CR2: 000014c2bf3ecf98 CR3: 0000000001e0a005 CR4: 00000000000606f0
    Sep 20 18:21:33 systemname kernel: Call Trace:
    Sep 20 18:21:33 systemname kernel: nfsd_open+0x15e/0x17c [nfsd]
    Sep 20 18:21:33 systemname kernel: nfsd_write+0x4c/0xaa [nfsd]
    Sep 20 18:21:33 systemname kernel: nfsd3_proc_write+0xad/0xdb [nfsd]
    Sep 20 18:21:33 systemname kernel: nfsd_dispatch+0xb4/0x169 [nfsd]
    Sep 20 18:21:33 systemname kernel: svc_process+0x4b5/0x666 [sunrpc]
    Sep 20 18:21:33 systemname kernel: ? nfsd_destroy+0x48/0x48 [nfsd]
    Sep 20 18:21:33 systemname kernel: nfsd+0xeb/0x142 [nfsd]
    Sep 20 18:21:33 systemname kernel: kthread+0x10b/0x113
    Sep 20 18:21:33 systemname kernel: ? kthread_flush_work_fn+0x9/0x9
    Sep 20 18:21:33 systemname kernel: ret_from_fork+0x35/0x40
    Sep 20 18:21:33 systemname kernel: ---[ end trace e750f2a3f27398a6 ]---
    Sep 20 18:21:34 systemname kernel: docker0: port 5(vethb58ec71) entered disabled state
    Sep 20 18:21:34 systemname kernel: vethde93cc2: renamed from eth0

    Link to comment
    18 minutes ago, testdasi said:

    @Can0nfan: Perhaps try booting in safe mode and see if you still have the same issue? It cut out the possibility of misconfigured plugins.

     

    Your log is (still) flooded with acpid events every 2 seconds but no longer because of csrf_token.

     

    Yeah I’m seeing a ton of traffic of connects and disconnects every second or two I’ll grab a screenshot when I get back home in about 15min

     

    what are acpid errors?

    Link to comment

    FYI, to fix the issue without rebooting the server you can unmount the /mnt/user mount point using:

    fusermount -uz /mnt/user

    And remount the system using:

    /usr/local/sbin/shfs /mnt/user -disks 63 2048000000 -o noatime,big_writes,allow_other -o remember=330 |& logger

     

    I'm still unsure why this process fails.

    • Like 1
    Link to comment

    may have found out the issue with the spamming logs still not sure if the permissions issue with user/ will come back.....gui mode does not like this server on 6.6 (6.5.3 was fine)......my console monitor is just flickering the login screen to black screen with white cursor in the upper left corner and back and forth rapidly...under no gui mode my logs are pretty quiet

     

    attached is a video

    Link to comment

    That's a good first step. Perhaps the acpid spam was due to the GUI having problems. Maybe try reboot in non-GUI to see if the hang goes away.

    Link to comment

    non-gui already booted and /mnt/user went haywire again just now..i finished up a show on plex and the next episode said unavailable checked the CLI and mnt/user is toast again

     

    what is up with this....i have friends and family hitting me up with sooo many messages becasue this keeps happening i came from RC4 so i dont have an option in the gui to downgrade to 6.5.3 is there anyone that can help i need stability to my server and this is 4th time today this has happened.....other unraid server is ok. they all have the same plugins installed.

     

    here is my diagnostics now

     

     

    thor-diagnostics-20180920-1900.zip

    Link to comment

    here is last snippet before /user went away again

     

     

    Sep 20 18:58:17 Thor kernel: nfsd: non-standard errno: -103
    Sep 20 18:58:17 Thor kernel: WARNING: CPU: 0 PID: 6510 at fs/nfsd/nfsproc.c:817 nfserrno+0x44/0x4a [nfsd]
    Sep 20 18:58:17 Thor kernel: Modules linked in: xt_nat macvlan xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat cmac arc4 ecb md4 cifs ccm xfs nfsd lockd grace sunrpc md_mod i915 i2c_algo_bit iosf_mbi drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd mpt3sas alx glue_helper intel_cstate i2c_i801 intel_uncore i2c_core raid_class mxm_wmi intel_rapl_perf scsi_transport_sas ahci mdio libahci wmi mlx4_core video backlight pcc_cpufreq thermal acpi_pad button
    Sep 20 18:58:17 Thor kernel: fan
    Sep 20 18:58:17 Thor kernel: CPU: 0 PID: 6510 Comm: nfsd Not tainted 4.18.8-unRAID #1
    Sep 20 18:58:17 Thor kernel: Hardware name: MSI MS-7978/H170 GAMING M3 (MS-7978), BIOS C.B0 01/25/2018
    Sep 20 18:58:17 Thor kernel: RIP: 0010:nfserrno+0x44/0x4a [nfsd]
    Sep 20 18:58:17 Thor kernel: Code: c0 48 83 f8 22 75 e2 80 3d b3 06 01 00 00 bb 00 00 00 05 75 17 89 fe 48 c7 c7 3b da 45 a0 c6 05 9c 06 01 00 01 e8 8a ac bf e0 <0f> 0b 89 d8 5b c3 48 83 ec 18 31 c9 ba ff 07 00 00 65 48 8b 04 25 
    Sep 20 18:58:17 Thor kernel: RSP: 0018:ffffc9000375bdc0 EFLAGS: 00010282
    Sep 20 18:58:17 Thor kernel: RAX: 0000000000000000 RBX: 0000000005000000 RCX: 0000000000000007
    Sep 20 18:58:17 Thor kernel: RDX: 0000000000000000 RSI: ffff88086dc16470 RDI: ffff88086dc16470
    Sep 20 18:58:17 Thor kernel: RBP: ffffc9000375be10 R08: 0000000000000003 R09: ffffffff8220db00
    Sep 20 18:58:17 Thor kernel: R10: 0000000000000469 R11: 00000000000160fc R12: ffff880845f53808
    Sep 20 18:58:17 Thor kernel: R13: 0000000002201000 R14: ffff880845f53968 R15: 0000000000000100
    Sep 20 18:58:17 Thor kernel: FS: 0000000000000000(0000) GS:ffff88086dc00000(0000) knlGS:0000000000000000
    Sep 20 18:58:17 Thor kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Sep 20 18:58:17 Thor kernel: CR2: 000014be468e6000 CR3: 0000000001e0a001 CR4: 00000000003626f0
    Sep 20 18:58:17 Thor kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Sep 20 18:58:17 Thor kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    Sep 20 18:58:17 Thor kernel: Call Trace:
    Sep 20 18:58:17 Thor kernel: nfsd_open+0x15e/0x17c [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd_read+0x45/0xec [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd3_proc_read+0x95/0xda [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd_dispatch+0xb4/0x169 [nfsd]
    Sep 20 18:58:17 Thor kernel: svc_process+0x4b5/0x666 [sunrpc]
    Sep 20 18:58:17 Thor kernel: ? nfsd_destroy+0x48/0x48 [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd+0xeb/0x142 [nfsd]
    Sep 20 18:58:17 Thor kernel: kthread+0x10b/0x113
    Sep 20 18:58:17 Thor kernel: ? kthread_flush_work_fn+0x9/0x9
    Sep 20 18:58:17 Thor kernel: ret_from_fork+0x35/0x40
    Sep 20 18:58:17 Thor kernel: ---[ end trace af5d84512fe95036 ]---

    Link to comment
    2 minutes ago, Can0nfan said:

    here is last snippet before /user went away again

     

     

    Sep 20 18:58:17 Thor kernel: nfsd: non-standard errno: -103
    Sep 20 18:58:17 Thor kernel: WARNING: CPU: 0 PID: 6510 at fs/nfsd/nfsproc.c:817 nfserrno+0x44/0x4a [nfsd]
    Sep 20 18:58:17 Thor kernel: Modules linked in: xt_nat macvlan xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat cmac arc4 ecb md4 cifs ccm xfs nfsd lockd grace sunrpc md_mod i915 i2c_algo_bit iosf_mbi drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd mpt3sas alx glue_helper intel_cstate i2c_i801 intel_uncore i2c_core raid_class mxm_wmi intel_rapl_perf scsi_transport_sas ahci mdio libahci wmi mlx4_core video backlight pcc_cpufreq thermal acpi_pad button
    Sep 20 18:58:17 Thor kernel: fan
    Sep 20 18:58:17 Thor kernel: CPU: 0 PID: 6510 Comm: nfsd Not tainted 4.18.8-unRAID #1
    Sep 20 18:58:17 Thor kernel: Hardware name: MSI MS-7978/H170 GAMING M3 (MS-7978), BIOS C.B0 01/25/2018
    Sep 20 18:58:17 Thor kernel: RIP: 0010:nfserrno+0x44/0x4a [nfsd]
    Sep 20 18:58:17 Thor kernel: Code: c0 48 83 f8 22 75 e2 80 3d b3 06 01 00 00 bb 00 00 00 05 75 17 89 fe 48 c7 c7 3b da 45 a0 c6 05 9c 06 01 00 01 e8 8a ac bf e0 <0f> 0b 89 d8 5b c3 48 83 ec 18 31 c9 ba ff 07 00 00 65 48 8b 04 25 
    Sep 20 18:58:17 Thor kernel: RSP: 0018:ffffc9000375bdc0 EFLAGS: 00010282
    Sep 20 18:58:17 Thor kernel: RAX: 0000000000000000 RBX: 0000000005000000 RCX: 0000000000000007
    Sep 20 18:58:17 Thor kernel: RDX: 0000000000000000 RSI: ffff88086dc16470 RDI: ffff88086dc16470
    Sep 20 18:58:17 Thor kernel: RBP: ffffc9000375be10 R08: 0000000000000003 R09: ffffffff8220db00
    Sep 20 18:58:17 Thor kernel: R10: 0000000000000469 R11: 00000000000160fc R12: ffff880845f53808
    Sep 20 18:58:17 Thor kernel: R13: 0000000002201000 R14: ffff880845f53968 R15: 0000000000000100
    Sep 20 18:58:17 Thor kernel: FS: 0000000000000000(0000) GS:ffff88086dc00000(0000) knlGS:0000000000000000
    Sep 20 18:58:17 Thor kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Sep 20 18:58:17 Thor kernel: CR2: 000014be468e6000 CR3: 0000000001e0a001 CR4: 00000000003626f0
    Sep 20 18:58:17 Thor kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Sep 20 18:58:17 Thor kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    Sep 20 18:58:17 Thor kernel: Call Trace:
    Sep 20 18:58:17 Thor kernel: nfsd_open+0x15e/0x17c [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd_read+0x45/0xec [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd3_proc_read+0x95/0xda [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd_dispatch+0xb4/0x169 [nfsd]
    Sep 20 18:58:17 Thor kernel: svc_process+0x4b5/0x666 [sunrpc]
    Sep 20 18:58:17 Thor kernel: ? nfsd_destroy+0x48/0x48 [nfsd]
    Sep 20 18:58:17 Thor kernel: nfsd+0xeb/0x142 [nfsd]
    Sep 20 18:58:17 Thor kernel: kthread+0x10b/0x113
    Sep 20 18:58:17 Thor kernel: ? kthread_flush_work_fn+0x9/0x9
    Sep 20 18:58:17 Thor kernel: ret_from_fork+0x35/0x40
    Sep 20 18:58:17 Thor kernel: ---[ end trace af5d84512fe95036 ]---

    I believe your other issues may stem from the failure of the /mnt/user mount point. As I said earlier, you can unmount the share using:

    fusermount -uz /mnt/user

    And remount the system using:

    /usr/local/sbin/shfs /mnt/user -disks 63 2048000000 -o noatime,big_writes,allow_other -o remember=330 |& logger

     

    I don't understand why it's failing. Hopefully someone is able to troubleshoot and resolve that issue. I've had to reboot my system 5 times today. Will probably roll back to 6.5.3 until this is fixed.

    Link to comment
    23 minutes ago, edgedog said:

    I believe your other issues may stem from the failure of the /mnt/user mount point. As I said earlier, you can unmount the share using:

    fusermount -uz /mnt/user

    And remount the system using:

    /usr/local/sbin/shfs /mnt/user -disks 63 2048000000 -o noatime,big_writes,allow_other -o remember=330 |& logger

     

    I don't understand why it's failing. Hopefully someone is able to troubleshoot and resolve that issue. I've had to reboot my system 5 times today. Will probably roll back to 6.5.3 until this is fixed.

    yeah just on my 5th reboot too ill try your CLI stuff if it fails again....wish i could go back to 6.5.3 (i went to stable version) from rc4 so my only option is RC4

    i hope this is patched and fixed ASAP i can mount my plex to to /mnt/user0/Media i guess in the meantime no cache files will show until mover moves to the array but better than nothing i guess

     

    Edited by Can0nfan
    Link to comment

    @limetech @eschultz @jonp @bonienl you guys able to take a look at this find out why /user keeps getting these broken permissions every few hours (1hr 38min in the last time i had to reboot)

     

    drwxrwxrwx  1 nobody users  38 Sep 20 17:41 cache/

    drwxrwxrwx 10 nobody users 162 Sep 20 17:41 disk1/

    drwxrwxrwx  4 nobody users  50 Sep 20 17:41 disk10/

    drwxrwxrwx  6 nobody users  90 Sep 20 17:41 disk2/

    drwxrwxrwx  5 nobody users  69 Sep 20 17:41 disk3/

    drwxrwxrwx  7 nobody users  85 Sep 20 17:41 disk4/

    drwxrwxrwx  8 nobody users 125 Sep 20 17:41 disk5/

    drwxrwxrwx  8 nobody users 123 Sep 20 17:41 disk6/

    drwxrwxrwx  8 nobody users 125 Sep 20 17:41 disk7/

    drwxrwxrwx 11 nobody users 179 Sep 20 17:41 disk8/

    drwxrwxrwx  6 nobody users  91 Sep 20 17:41 disk9/

    drwxrwxrwx  4 nobody users  80 Sep 20 17:31 disks/

    d?????????  ? ?      ?       ?            ? user/

    drwxrwxrwx  1 nobody users 162 Sep 20 17:41 user0/

    Link to comment
    7 minutes ago, Can0nfan said:

    wish i could go back to 6.5.3

    Download 6.5.3 from the Downloads link at the upper right of this forum. Unzip the download and replace all the bz* files on your flash with the ones from the download.

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.