Hello everyone,

I'm experiencing persistent issues with NFS mounts between my Proxmox server and Unraid server, specifically involving LXC containers. I'm hoping to get some insights or suggestions from the community to resolve this problem.

 

Issue Description:

• Symptom:
  • The NFS mounts on the Proxmox server hang, causing commands like ls /mnt/data to become unresponsive.
• Observation on Unraid Server:
  • nfsd processes are stuck in the D (uninterruptible sleep) state.
  • Restarting NFS services on Unraid doesn't resolve the issue.
• Attempts to Unmount NFS Shares:
  • Running umount -f /mnt/data on Proxmox results in "device is busy" errors.
• Network Connectivity:
  • Both servers can ping each other.
  • No apparent network issues like packet loss or high latency.
• Additional Notes:
  • Other services on both servers appear to function normally.

 

Network Configuration:

• Proxmox Server (pve):
  • IP Address: 10.0.0.2
  • Role: Hosts several LXC containers and mounts NFS shares from the Unraid server.
• Unraid Server (Tower):
  • IP Address: 10.0.0.3
  • Role: Provides NFS shares to the Proxmox server.
• Network Details:
  • Subnet: 10.0.0.0/12
  • Gateway: 10.0.0.1
• DNS Servers: 10.0.0.1 (FYI: my router had some dns issues last week)
• Both servers have static IP addresses assigned.

 

 

LXC Container Configuration:

I have an LXC container running on the Proxmox server configured as follows:

• Container ID: 105
• Hostname: qbittorrent
• Purpose: Runs a torrent client that downloads directly to an NFS-mounted share.
• Network:
  • Type: Virtual bridge (vmbr0)
  • IP Address: Assigned via DHCP within the same subnet (10.0.0.x).
• Mounts:
  • NFS share from Unraid mounted directly inside the container at /mnt/data.

 

My host (proxmox) fstab:
 

Quote

10.0.0.3:/mnt/user/data     /mnt/data      nfs    nfsvers=4.2,hard,timeo=50,retrans=12,_netdev    0  0

 

 

and 

my LXC config

Quote

root@pve:~# cat /etc/pve/lxc/105.conf
arch: amd64
cores: 2
features: nesting=1
hostname: qbittorrent
memory: 4096
mp0: /mnt/data/,mp=/data
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:53:8C:EE,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-105-disk-0,size=8G
swap: 2048
unprivileged: 1

 

 

Debug Findings:

On Unraid Server (Tower):

 

NFS Daemon Processes:

Quote

root@Tower:~# ps aux | grep [n]fsd
root      5769  0.0  0.0      0     0 ?        I    Oct02   0:06 [nfsd]
root      5770  0.0  0.0      0     0 ?        I    Oct02   0:07 [nfsd]
root      5771  0.0  0.0      0     0 ?        I    Oct02   0:08 [nfsd]
root      5772  0.0  0.0      0     0 ?        I    Oct02   0:10 [nfsd]
root      5773  0.0  0.0      0     0 ?        I    Oct02   0:13 [nfsd]
root      5774  0.0  0.0      0     0 ?        I    Oct02   0:18 [nfsd]
root      5775  0.0  0.0      0     0 ?        I    Oct02   0:45 [nfsd]
root      5776  0.2  0.0      0     0 ?        D    Oct02   2:03 [nfsd]
root     20767  0.0  0.0      0     0 ?        D    Oct02   0:05 [kworker/u16:2+nfsd4_callbacks]
 

 

RPC Services:

Quote

root@Tower:~# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  46732  status
    100024    1   tcp  36489  status
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100003    3   udp   2049  nfs
    100021    1   udp  50490  nlockmgr
    100021    3   udp  50490  nlockmgr
    100021    4   udp  50490  nlockmgr
    100021    1   tcp  36883  nlockmgr
    100021    3   tcp  36883  nlockmgr
    100021    4   tcp  36883  nlockmgr
    100005    1   udp  45627  mountd
    100005    1   tcp  38783  mountd
    100005    2   udp  50467  mountd
    100005    2   tcp  56561  mountd
    100005    3   udp  41573  mountd
    100005    3   tcp  41299  mountd
 

 

System Logs:

No significant errors related to disks, network, or NFS found in /var/log/syslog or dmesg.

nfsd processes remain in the D state even after restarting NFS services.
 

 

On Proxmox Server (pve):

Attempted unmount:
 

Quote

root@pve:~# umount -f /mnt/data
umount.nfs4: /mnt/data: device is busy
 

 

Kernel messages

Quote

root@pve:~# dmesg | tail -n 100
[63399.549584] task:P[diskspace]    state:D stack:0     pid:659829 tgid:659414 ppid:1      flags:0x00000002
[63399.549586] Call Trace:
[63399.549586]  <TASK>
[63399.549587]  __schedule+0x401/0x15e0
[63399.549589]  ? find_busiest_group+0x4d/0x500
[63399.549591]  schedule+0x33/0x110
[63399.549593]  rpc_wait_bit_killable+0x11/0x90 [sunrpc]
[63399.549617]  __wait_on_bit+0x4a/0x120
[63399.549618]  ? __pfx_rpc_wait_bit_killable+0x10/0x10 [sunrpc]
[63399.549642]  out_of_line_wait_on_bit+0x8c/0xb0
[63399.549643]  ? __pfx_wake_bit_function+0x10/0x10
[63399.549644]  __rpc_execute+0x134/0x4b0 [sunrpc]
[63399.549668]  rpc_execute+0xda/0x110 [sunrpc]
[63399.549693]  rpc_run_task+0x12e/0x190 [sunrpc]
[63399.549710]  nfs4_do_call_sync+0x6b/0xc0 [nfsv4]
[63399.549724]  _nfs4_proc_getattr+0x13b/0x180 [nfsv4]
[63399.549737]  nfs4_proc_getattr+0x6e/0x110 [nfsv4]
[63399.549751]  __nfs_revalidate_inode+0xd4/0x3b0 [nfs]
[63399.549766]  nfs_getattr+0x363/0x4d0 [nfs]
[63399.549781]  vfs_getattr_nosec+0xb7/0x100
[63399.549782]  vfs_statx+0x16f/0x1d0
[63399.549784]  vfs_fstatat+0xaa/0xe0
[63399.549786]  __do_sys_newfstatat+0x44/0x90
[63399.549788]  __x64_sys_newfstatat+0x1c/0x30
[63399.549790]  do_syscall_64+0x84/0x180
[63399.549791]  ? __count_memcg_events+0x6f/0xe0
[63399.549794]  ? count_memcg_events.constprop.0+0x2a/0x50
[63399.549796]  ? handle_mm_fault+0xad/0x380
[63399.549798]  ? do_user_addr_fault+0x343/0x6b0
[63399.549800]  ? irqentry_exit_to_user_mode+0x7b/0x260
[63399.549801]  ? irqentry_exit+0x43/0x50
[63399.549802]  ? exc_page_fault+0x94/0x1b0
[63399.549803]  entry_SYSCALL_64_after_hwframe+0x73/0x7b
[63399.549805] RIP: 0033:0x7b4e8e31675a
 

 

 

Questions and assistance needed:

1. Could the nfsd processes being stuck in the D state be caused by the LXC container accessing the NFS share directly?
2. The container runs a torrent client that downloads directly to the NFS-mounted share while keeping all AppData on Proxmox for easier backups.
3. Could network misconfigurations cause this issue, even though both servers have static IPs and can ping each other?
4. What did I set up wrong?

 

 

What I've tried so far:

• Restarted NFS services on both servers.
• Attempted to unmount NFS shares using umount -f, but received "device is busy" errors.
• Checked for disk errors and system logs on Unraid; found no significant issues.
• Verified network connectivity and configurations; no apparent issues found.

 

Additional Information:

• System Versions:
  • Unraid OS: Version 6.12.3
  • Proxmox VE: Version 7.4-3
  • NFS Version Used: NFSv4.2

Export options on Unraid:

Quote

/mnt/user/data 10.0.0.2(async,wdelay,no_subtree_check,fsid=104,anonuid=99,anongid=99,sec=sys,rw,insecure,root_squash,all_squash)

Mount Options on Proxmox:

Quote

mount -t nfs -o vers=4.2 10.0.0.3:/mnt/user/data /mnt/data

 

Any help or suggestions would be greatly appreciated!

I'm open to any troubleshooting steps or best practices that could help resolve this issue. If more information is needed, please let me know, and I'll provide it. I've been fighting with it for over a week now and nothing besides hard reset helps. 

On 10/3/2024 at 11:25 AM, MarcMo said:

Hello everyone,

I'm experiencing persistent issues with NFS mounts between my Proxmox server and Unraid server, specifically involving LXC containers. I'm hoping to get some insights or suggestions from the community to resolve this problem.

 

Issue Description:

• Symptom:
  • The NFS mounts on the Proxmox server hang, causing commands like ls /mnt/data to become unresponsive.
• Observation on Unraid Server:
  • nfsd processes are stuck in the D (uninterruptible sleep) state.
  • Restarting NFS services on Unraid doesn't resolve the issue.
• Attempts to Unmount NFS Shares:
  • Running umount -f /mnt/data on Proxmox results in "device is busy" errors.
• Network Connectivity:
  • Both servers can ping each other.
  • No apparent network issues like packet loss or high latency.
• Additional Notes:
  • Other services on both servers appear to function normally.

 

Network Configuration:

• Proxmox Server (pve):
  • IP Address: 10.0.0.2
  • Role: Hosts several LXC containers and mounts NFS shares from the Unraid server.
• Unraid Server (Tower):
  • IP Address: 10.0.0.3
  • Role: Provides NFS shares to the Proxmox server.
• Network Details:
  • Subnet: 10.0.0.0/12
  • Gateway: 10.0.0.1
• DNS Servers: 10.0.0.1 (FYI: my router had some dns issues last week)
• Both servers have static IP addresses assigned.

 

 

LXC Container Configuration:

I have an LXC container running on the Proxmox server configured as follows:

• Container ID: 105
• Hostname: qbittorrent
• Purpose: Runs a torrent client that downloads directly to an NFS-mounted share.
• Network:
  • Type: Virtual bridge (vmbr0)
  • IP Address: Assigned via DHCP within the same subnet (10.0.0.x).
• Mounts:
  • NFS share from Unraid mounted directly inside the container at /mnt/data.

 

My host (proxmox) fstab:
 

 

 

and 

my LXC config

 

 

Debug Findings:

On Unraid Server (Tower):

 

NFS Daemon Processes:

 

RPC Services:

 

System Logs:

No significant errors related to disks, network, or NFS found in /var/log/syslog or dmesg.

nfsd processes remain in the D state even after restarting NFS services.
 

 

On Proxmox Server (pve):

Attempted unmount:
 

 

Kernel messages

 

 

Questions and assistance needed:

1. Could the nfsd processes being stuck in the D state be caused by the LXC container accessing the NFS share directly?
2. The container runs a torrent client that downloads directly to the NFS-mounted share while keeping all AppData on Proxmox for easier backups.
3. Could network misconfigurations cause this issue, even though both servers have static IPs and can ping each other?
4. What did I set up wrong?

 

 

What I've tried so far:

• Restarted NFS services on both servers.
• Attempted to unmount NFS shares using umount -f, but received "device is busy" errors.
• Checked for disk errors and system logs on Unraid; found no significant issues.
• Verified network connectivity and configurations; no apparent issues found.

 

Additional Information:

• System Versions:
  • Unraid OS: Version 6.12.3
  • Proxmox VE: Version 7.4-3
  • NFS Version Used: NFSv4.2

Export options on Unraid:

Mount Options on Proxmox:

 

Any help or suggestions would be greatly appreciated!

I'm open to any troubleshooting steps or best practices that could help resolve this issue. If more information is needed, please let me know, and I'll provide it. I've been fighting with it for over a week now and nothing besides hard reset helps. 

i thought that i was the only one. i have had similar results. It just dies, until you reboot the system and then it's ticking bomb, until it dies again.

it was resolved by using nfsv4 and disabling hard link in unraid