Unsealed0019

Swapped the motherboard - that wasn't it. Swapped the CPU and could then boot anything under the sun again.

PassMark Memtest passed 100% no errors - booted without issue. Booting from a restored backup USB drive (brand new drive) runs into same boot failure (won't get past Loading /bzroot... OK) - USB devices turn off after and system hangs. New USB sticks show exact same symptoms (Unraid or Ubuntu). After removing all GPUs, reverting memory to some older sticks, unplugging power to hard drive cages, behavior is still about the same. After a CMOS reset, CSM would remain enabled - motherboard strangeness after bios update. Seems like I need a new motherboard or PSU but the PSU swap in the past didn't resolve the issue and only by tweaking some BIOS power settings did the issue temporarily go away (about a year).

I’ve hit a new wall in troubleshooting: Memory: I pulled 2 RAM sticks (down to 64 GB, 2×32 GB). With this setup, the system won’t boot at all. I cannot run the bundled Unraid memtest (it freezes at “Loading... /something OK”). Currently running PassMark MemTest86 instead (UEFI), since at least that one launches. Boot behavior: Attempting to boot Unraid from the flash now also freezes during startup (USB devices shut off, must hard reset). Safe mode resets on it's own. Normal mode hangs at the boot menu screen, which feels less like an OS/driver issue and more like a hardware or flash media issue. Flash drive suspicion: Plugging the Unraid USB stick into a Windows machine triggers “errors detected, scan and fix”. This makes me suspect the flash drive may be at least part of the problem, though as you mentioned this is also likely to happen from unclean shutdowns. BIOS: I restored to optimized defaults and only made a couple of edits: Typical Current Idle Enabled SVM and IOMMU Secure Boot off Will let memtest go for a while then will try booting the new USB drive. I do have some older smaller memory sticks I could revert back to just to outright rule out the current sticks.

If, when I plug it into a windows PC, it says 'errors detected, scan and fix' would that then indicate an issue and should I perform that action?

Thanks JorgeB - I will try that. I'm not sure how to tell which hardware component would be most likely to be having the problem. I'm leaning towards replacing the memory first but would likely get ECC this time which also requires a different CPU anyway. One other thing I forgot to mention is that when I had Tips and Tweaks installed, it did not show there being a CPU governor, which I wasn't sure if it was because of the BIOS tweaks or a software issue that could be resolved. This System Driver is not present on the crashing server: acpi_cpufreq (ACPI Processor P-States Driver) : cpufreq What is the likelihood this is a failing USB Flash drive - can I test for that?

Hi all, I'm at my whits end after spending an entire day troubleshooting this with ChatGPT and other resources. I’m running into recurring stability issues with my Unraid server that I can’t seem to pin down. The system will randomly crash or reset without leaving anything useful in the syslogs. Crashes can happen hours apart, days, or even weeks—it’s very inconsistent. I have another Unraid server set to receive the logs and it just cuts off prior to the crash. This started happening again about a week or two ago. Around that time, I had added a new Docker container and disabled another, but otherwise made very few changes. I already tried disabling the new container, but the crashes continue. A bit of history: About a year ago, I had the exact same problem (random crashing, no logs). I replaced the PSU at that time, but it didn’t solve it. I then adjusted some BIOS settings to avoid AMD CPU C-state and power idle issues, updated the BIOS, and added another fan to bring cooler air into the room. After making those changes, the problem went away for quite a while. Fast-forward to now, the random crashes are back. I swapped in another PSU again just to rule it out, but no improvement. So far, the system just drops dead with no warning and nothing in the logs. I’m trying to determine if this is hardware-related (CPU, RAM, motherboard, etc.), BIOS/firmware settings, or something within Unraid/Docker that’s triggering instability. BIOS Settings I’ve Adjusted (for stability) Since this seems to be hardware/firmware related, here are the BIOS settings I’ve already reviewed or changed while troubleshooting: CPU / Power Management: Disabled Global C-States and set Power Idle Control to a non-low power mode (to avoid Ryzen deep idle bugs). Disabled Core Performance Boost for stability (trading performance for consistency). Disabled CPU Watchdog Timer (server was crashing even with it off). Memory / DRAM: Enabled/disabled Power Down Enable (tried both). Left Gear Down Mode and Cmd2T at Auto for compatibility. Experimented with DRAM refresh modes and scrubber controls (kept mostly on Auto to avoid instability). XMP/DOCP profile currently disabled (running JEDEC defaults for max stability). Interconnect / DF Options: Tweaked Memory Interleaving and related settings (left at Auto eventually). Disabled DF Sync Flood Propagation (to prevent cascading errors). Left DF C-States mostly disabled for consistency. PCIe / GPU / IOMMU: PCIe ASPM Mode set to Disabled (to prevent link power management from destabilizing GPUs). Using multiple GPUs (WX5100 passthrough (VFIO), plus iGPU & WX2100 for Dockers) - stopped using WX2100 and only using iGPU, VM using WX5100 stopped UMA Frame Buffer Size set manually to avoid auto-adjustment issues (1G). Security / Virtualization: TSME (Transparent Secure Memory Encryption) disabled (or default, I forget but didn't matter). SEV / SEV-ES options left off. CSM (Compatibility Support Module) can’t be enabled anymore after BIOS update (seems forced UEFI). Secure Boot disabled. Misc / Timers: High Precision Event Timer (HPET): currently disabled. Power Loading: enabled to stabilize low-power PSU states (Typical Current). Spread Spectrum: disabled (to avoid clock modulation issues). Other Notes: System previously stable after disabling C-states and forcing power idle workarounds. Crashes now occur regardless of these settings. System Specs Motherboard: Gigabyte X570 AORUS ULTRA (latest BIOS, F39g) CPU: AMD Ryzen 7 5700G with Radeon Graphics @ 3800 MHz (with iGPU – Cezanne Vega) Memory: 128 GB non-ECC (running at JEDEC defaults, XMP off) GPUs: AMD Radeon WX 5100 (passthrough to VM) AMD Radeon WX 2100 (Docker use / compute) Integrated GPU (Cezanne Vega, used for Docker/host tasks) PSU: Recently replaced (Seasonic 800W) Cooling: Added extra intake fan for airflow in server room (which was part of the year-ago effort) Boot: Unraid on USB flash, UEFI-only (CSM disabled by firmware), Secure Boot off It may or may not be related to running a separate Ubuntu VM, which is currently shut down, and I am waiting to see if it crashes. Has anyone else experienced this pattern of random, log-less crashes on Unraid, especially with AMD platforms? What would be the most systematic way to isolate the cause? Thanks in advance for any ideas or troubleshooting steps I might have missed.

I'm running into the same general symptoms and issue (random crashing without anything useful in the syslogs - could be hours apart, days, or weeks)... Started happening out of the blue again about a week or two ago. I had added a new docker and turned off another, but other than that very few changes. Of course I tried disabling the new docker but that didn't make a difference. I already replaced the PSU and it made no difference. About a year ago, I was having the same issue, which is when I replaced the PSU but since that wasn't it, I also tweaked some BIOS settings to force the AMD CPU to not have c-states, or power idle issues, etc. I also updated the BIOS, and put in another fan for the room to pull in some cooler air. The issue went away after a number of changes but I know it crashed after the PSU swap.

I can try different USB ports but there's been a recent regression as far as I can tell. Several versions ago this wasn't an issue (or several versions of the plugin ago). Not the controller(s) but a few individual devices, yes. Those are successfully being passed and many preclears have been successful without this issue. I've been able to successfully preclear drives this with non-UASP enclosures a dozen times or so but the past several versions of UNRAID seem to have made this more problematic. Perhaps related but at least one time the unassigned devices plugin dropped nfs/smb shares and needed a reboot. The plugin itself may be part of the issue here.

Ok I got one with that feature and it's the same situation... any ideas?

Alright, confirmed both of mine do not have that.

Same issue as well, tried two different USB 3.0 external drive enclosures. Rebooting gets it working again but then eventually detection fails.

Thirded - it would be awesome if there was a simple way after doing "Stop All" to do "Start All" (which were set to auto start).

I did not! That was it, thanks

I'm on 6.12.3 but cannot see this option to launch remote-viewer - where are they located?

They are updated Ubuntu systems running v4 Checked with nfsstat -s & nfsstat -c Some extra info: UNRAID is NFS server Ubuntu bare metal is one client (haven't actually tested deleting from here, no cause of crash to my knowledge) Ubuntu UNRAID VM is another client (this seems to be where I see the issues the most with deletes and what not) I use UD SMB mounts (no NFS for UD) The bare metal system was seeing stale file handle issues (all NFS clients were) but that issue is passed now.

If possible I'd like to know where in the logs to look to see the crash event from this delete action (or pre-delete action). This was another: Jun 24 06:11:33 REDACTED smbd[12682]: [2023/06/24 06:11:33.404989, 0] ../../source3/smbd/close.c:1485(close_directory) Jun 24 06:11:33 REDACTED smbd[12682]: Could not close dir! fname=REDACTED_FOLDER_PATH, fd=37, err=116=Stale file handle Jun 24 06:11:33 REDACTED smbd[12682]: [2023/06/24 06:11:33.407654, 0] ../../source3/smbd/close.c:1485(close_directory) Jun 24 06:11:33 REDACTED smbd[12682]: Could not close dir! fname=REDACTED_FOLDER_PATH, fd=38, err=116=Stale file handle Jun 24 06:15:52 REDACTED kernel: ------------[ cut here ]------------ Jun 24 06:15:52 REDACTED kernel: nfsd: non-standard errno: -38 Jun 24 06:15:52 REDACTED kernel: WARNING: CPU: 10 PID: 9462 at fs/nfsd/nfsproc.c:889 nfserrno+0x45/0x51 [nfsd] Jun 24 06:15:52 REDACTED kernel: Modules linked in: rpcsec_gss_krb5 cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod tcp_diag inet_diag it87 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls amdgpu radeon gpu_sched drm_ttm_helper ttm btusb edac_mce_amd edac_core gigabyte_wmi wmi_bmof mxm_wmi kvm_amd kvm btrtl crct10dif_pclmul drm_display_helper crc32_pclmul crc32c_intel btbcm ghash_clmulni_intel aesni_intel drm_kms_helper crypto_simd btintel cryptd rapl bluetooth Jun 24 06:15:52 REDACTED kernel: drm igb i2c_piix4 nvme agpgart k10temp i2c_algo_bit ecdh_generic joydev ecc ahci nvme_core i2c_core libahci ccp syscopyarea sysfillrect sysimgblt fb_sys_fops tpm_crb tpm_tis thermal tpm_tis_core video tpm wmi backlight button acpi_cpufreq unix Jun 24 06:15:52 REDACTED kernel: CPU: 10 PID: 9462 Comm: nfsd Not tainted 5.19.17-Unraid #2 Jun 24 06:15:52 REDACTED kernel: Hardware name: MOTHERBOARD INFO Jun 24 06:15:52 REDACTED kernel: RIP: 0010:nfserrno+0x45/0x51 [nfsd] Jun 24 06:15:52 REDACTED kernel: Code: c3 cc cc cc cc 48 ff c0 48 83 f8 26 75 e0 80 3d bb 47 05 00 00 75 15 48 c7 c7 17 24 f0 a0 c6 05 ab 47 05 00 01 e8 42 87 94 e0 <0f> 0b b8 00 00 00 05 c3 cc cc cc cc 48 83 ec 18 31 c9 ba ff 07 00 Jun 24 06:15:52 REDACTED kernel: RSP: 0018:ffffc90002017dc0 EFLAGS: 00010282 Jun 24 06:15:52 REDACTED kernel: RAX: 0000000000000000 RBX: ffff888162b681a0 RCX: 0000000000000027 Jun 24 06:15:52 REDACTED kernel: RDX: 0000000000000001 RSI: ffffffff820d7be1 RDI: 00000000ffffffff Jun 24 06:15:52 REDACTED kernel: RBP: ffff888162b68030 R08: 0000000000000000 R09: ffffffff828653f0 Jun 24 06:15:52 REDACTED kernel: R10: 00003fffffffffff R11: ffff88a05e2bdcdd R12: ffff8882f77de6c0 Jun 24 06:15:52 REDACTED kernel: R13: ffff8881043f4000 R14: 00000000ffffffda R15: ffff8892bd198340 Jun 24 06:15:52 REDACTED kernel: FS: 0000000000000000(0000) GS:ffff889fde280000(0000) knlGS:0000000000000000 Jun 24 06:15:52 REDACTED kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 24 06:15:52 REDACTED kernel: CR2: 000000a98e47efb4 CR3: 0000000160f4c000 CR4: 0000000000750ee0 Jun 24 06:15:52 REDACTED kernel: PKRU: 55555554 Jun 24 06:15:52 REDACTED kernel: Call Trace: Jun 24 06:15:52 REDACTED kernel: <TASK> Jun 24 06:15:52 REDACTED kernel: nfsd_link+0x15f/0x1a5 [nfsd] Jun 24 06:15:52 REDACTED kernel: nfsd4_link+0x20/0x3e [nfsd] Jun 24 06:15:52 REDACTED kernel: nfsd4_proc_compound+0x437/0x56c [nfsd] Jun 24 06:15:52 REDACTED kernel: nfsd_dispatch+0x1a9/0x262 [nfsd] Jun 24 06:15:52 REDACTED kernel: svc_process+0x3f1/0x5d6 [sunrpc] Jun 24 06:15:52 REDACTED kernel: ? nfsd_svc+0x2b6/0x2b6 [nfsd] Jun 24 06:15:52 REDACTED kernel: ? nfsd_shutdown_threads+0x5b/0x5b [nfsd] Jun 24 06:15:52 REDACTED kernel: nfsd+0xd5/0x155 [nfsd] Jun 24 06:15:52 REDACTED kernel: kthread+0xe7/0xef Jun 24 06:15:52 REDACTED kernel: ? kthread_complete_and_exit+0x1b/0x1b Jun 24 06:15:52 REDACTED kernel: ret_from_fork+0x22/0x30 Jun 24 06:15:52 REDACTED kernel: </TASK> Jun 24 06:15:52 REDACTED kernel: ---[ end trace 0000000000000000 ]---

@dlandon Thanks for looking at it. I did recently upgrade the cache to be much larger and it's no longer getting anywhere close to full. The disks remain very full indeed, and I plan to get more but it's what it is for now... Allocation method is High-Water for all. Even with mostly full disks, a delete action shouldn't cause a complete system share meltdown. Min free space settings are all about 100GB or more, should all make sense. Good suggestions for NFS exports. I am no expert either but as UNRAID has limited characters for the rules, I've had to keep it shortened. Thankfully sec=sys should be default and unnecessary to add. I had stale file handle issues and am not sure what helped but removing insecure was one of the things I did. Is it a problem for UNRAID to have NFS mounts/shares only use the 'secure' port(s)? The clients are mounting the nfs shares with fstab options: rw,nosuid,noexec,hard,intr all_squash vs no_root_squash and all that confuses the hell out of me but my choice of `all_squash` to help normalize UID's shouldn't prevent delete from working or cause that to corrupt the UNRAID shares. Got another hard system share meltdown... I sat there wondering if things were stable enough to attempt a delete, started delete, got the popup asking if I was sure, and let it sit there because I was not sure... And there you have it, complete system share meltdown even without a delete being issued. I didn't proceed with the delete but somehow whatever NFS did prior to that prompt is what caused the issue, probably each of the three times this has occurred so far.

Created a file on an NFS share from a linux mount, then deleted the file and all unraid shares crashed and shut down basically. Must restart unraid after that. Issue maybe seems to stem from the permissions of an SMB user being forced into the file system as seen by the NFS mount point... even though I believe I've done all I can to force and normalize users/groups for the files... they just keep changing from nobody. Sounds like a major defect that needs some attention... Is general support not the right place for this?

With some changes, things have been stable comparatively however for the 2nd time, I have deleted a file on an NFS share and this immediately causes UNRAID to drop all shares (seem to all get corrupted in some form) This does not look good and appears to be a major defect of some sort: /mnt# ls -lh /bin/ls: cannot access 'user': Transport endpoint is not connected ... d????????? ? ? ? ? ? user/ ... I have to reboot to get them restored.

Got about a week and it occurred again when trying to delete a file through NFS. Delete failed, then all shares vanished when trying to delete again. Restarted UNRAID and they are back and then I deleted through SMB which worked ok. Seems like NFS is working better with IP(rw,all_squash,anonuid=99,anongid=100) but this issue still happened anyway... what can I do? @JorgeB@itimpi

Same recurring issue with 6.11.5 as well. What are the correct NFS export/mount parameters to avoid this issue? I have a Synology NAS as well with NFS exports and have never run into this issue. Occurs every day, several times a day with UNRAID.

I'd like to see this added as an option as well as I run into the same issues and have no idea how to 'properly' set permissions so SMB/NFS work together well. The New Permissions section always restores order but it would be less of a nuisance if it remained orderly instead.

Can no_subtree_check lead to more stale handles?

One of my NFS mount points went stale without the cache being filled up... Same system has two NFS mounts and the other is still fine. What might cause that? Everything has enough memory to spare, and space as far as I can tell.

So maybe 0 isn't very sensible then... I believe that was the default. Do I need to stop the array to change that value?