February 18, 20251 yr I had been on 6.11.5 forever and recently upgraded to 7.0 when my USB flash died. Aside from an issue with UNRAID CONNECT everything has been working well....but I noticed an intermittent call trace error relating to vmalloc Feb 18 14:10:55 CRUNCH kernel: warn_alloc: 1 callbacks suppressed Feb 18 14:10:55 CRUNCH kernel: lsof: vmalloc error: size 4194304, failed to allocated page array size 8192, mode:0x400cc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM), nodemask=(null),cpuset=/,mems_allowed=0-3 Feb 18 14:10:55 CRUNCH kernel: CPU: 12 PID: 3184246 Comm: lsof Tainted: P W O 6.6.68-Unraid #1 Feb 18 14:10:55 CRUNCH kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./EPYCD8-2T, BIOS P2.60 04/10/2020 Feb 18 14:10:55 CRUNCH kernel: Call Trace: Feb 18 14:10:55 CRUNCH kernel: <TASK> Feb 18 14:10:55 CRUNCH kernel: dump_stack_lvl+0x38/0x4a Feb 18 14:10:55 CRUNCH kernel: warn_alloc+0xfb/0x189 Feb 18 14:10:55 CRUNCH kernel: ? srso_return_thunk+0x5/0x5f Feb 18 14:10:55 CRUNCH kernel: __vmalloc_node_range+0x2bd/0x62b Feb 18 14:10:55 CRUNCH kernel: ? seq_read_iter+0x1de/0x353 Feb 18 14:10:55 CRUNCH kernel: kvmalloc_node+0xae/0xbc Feb 18 14:10:55 CRUNCH kernel: ? seq_read_iter+0x1de/0x353 Feb 18 14:10:55 CRUNCH kernel: seq_read_iter+0x1de/0x353 Feb 18 14:10:55 CRUNCH kernel: seq_read+0x91/0xbb Feb 18 14:10:55 CRUNCH kernel: vfs_read+0xa7/0x1d1 Feb 18 14:10:55 CRUNCH kernel: ? srso_return_thunk+0x5/0x5f Feb 18 14:10:55 CRUNCH kernel: ? __do_sys_newfstat+0x34/0x5c Feb 18 14:10:55 CRUNCH kernel: ksys_read+0x74/0xc0 Feb 18 14:10:55 CRUNCH kernel: do_syscall_64+0x57/0x7b Feb 18 14:10:55 CRUNCH kernel: entry_SYSCALL_64_after_hwframe+0x78/0xe2 Feb 18 14:10:55 CRUNCH kernel: RIP: 0033:0x14588a3106ed Feb 18 14:10:55 CRUNCH kernel: Code: 21 87 0e 00 f7 d8 64 89 02 b8 ff ff ff ff eb bb 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 80 3d 59 0b 0f 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec Feb 18 14:10:55 CRUNCH kernel: RSP: 002b:00007ffe7259dcc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 Feb 18 14:10:55 CRUNCH kernel: RAX: ffffffffffffffda RBX: 000000000043f600 RCX: 000014588a3106ed Feb 18 14:10:55 CRUNCH kernel: RDX: 0000000000000400 RSI: 0000000000448690 RDI: 0000000000000007 Feb 18 14:10:55 CRUNCH kernel: RBP: 000014588a3f81f0 R08: 0000000000000001 R09: 0000000000000000 Feb 18 14:10:55 CRUNCH kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014588a3f80a0 Feb 18 14:10:55 CRUNCH kernel: R13: 0000000000000000 R14: 0000000000001000 R15: 000000000043f600 Feb 18 14:10:55 CRUNCH kernel: </TASK> Feb 18 14:10:55 CRUNCH kernel: Mem-Info: Feb 18 14:10:55 CRUNCH kernel: active_anon:11403130 inactive_anon:2693854 isolated_anon:0 Feb 18 14:10:55 CRUNCH kernel: active_file:8358687 inactive_file:7968638 isolated_file:0 Feb 18 14:10:55 CRUNCH kernel: unevictable:836 dirty:30773 writeback:0 Feb 18 14:10:55 CRUNCH kernel: slab_reclaimable:886844 slab_unreclaimable:232381 Feb 18 14:10:55 CRUNCH kernel: mapped:395878 shmem:573300 pagetables:51966 Feb 18 14:10:55 CRUNCH kernel: sec_pagetables:195 bounce:0 Feb 18 14:10:55 CRUNCH kernel: kernel_misc_reclaimable:0 Feb 18 14:10:55 CRUNCH kernel: free:553088 free_pcp:41074 free_cma:0 Feb 18 14:10:55 CRUNCH kernel: Node 0 active_anon:16735288kB inactive_anon:1718592kB active_file:5798564kB inactive_file:6043816kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:313920kB dirty:20416kB writeback:0kB shmem:363020kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:17328128kB writeback_tmp:0kB kernel_stack:12820kB pagetables:54480kB sec_pagetables:176kB all_unreclaimable? no Feb 18 14:10:55 CRUNCH kernel: Node 1 active_anon:9206180kB inactive_anon:3987664kB active_file:9857148kB inactive_file:6922696kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:865512kB dirty:60296kB writeback:0kB shmem:1518372kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:9562112kB writeback_tmp:0kB kernel_stack:22916kB pagetables:46896kB sec_pagetables:264kB all_unreclaimable? no Feb 18 14:10:55 CRUNCH kernel: Node 2 active_anon:3228760kB inactive_anon:1957716kB active_file:11863852kB inactive_file:13841864kB unevictable:96kB isolated(anon):0kB isolated(file):0kB mapped:240120kB dirty:14400kB writeback:0kB shmem:210476kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:3741696kB writeback_tmp:0kB kernel_stack:12100kB pagetables:34144kB sec_pagetables:164kB all_unreclaimable? no Feb 18 14:10:55 CRUNCH kernel: Node 3 active_anon:16442292kB inactive_anon:3111444kB active_file:5915184kB inactive_file:5066176kB unevictable:3184kB isolated(anon):0kB isolated(file):0kB mapped:163960kB dirty:27980kB writeback:0kB shmem:201332kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:17127424kB writeback_tmp:0kB kernel_stack:18260kB pagetables:72344kB sec_pagetables:176kB all_unreclaimable? no Feb 18 14:10:55 CRUNCH kernel: Node 0 DMA free:15360kB boost:0kB min:12kB low:24kB high:36kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Feb 18 14:10:55 CRUNCH kernel: lowmem_reserve[]: 0 2831 31999 31999 31999 Feb 18 14:10:55 CRUNCH kernel: Node 0 DMA32 free:239148kB boost:0kB min:2968kB low:5864kB high:8760kB reserved_highatomic:0KB active_anon:124768kB inactive_anon:417300kB active_file:818600kB inactive_file:1064328kB unevictable:0kB writepending:40kB present:3027788kB managed:2935032kB mlocked:0kB bounce:0kB free_pcp:10604kB local_pcp:0kB free_cma:0kB Feb 18 14:10:55 CRUNCH kernel: lowmem_reserve[]: 0 0 29167 29167 29167 Feb 18 14:10:55 CRUNCH kernel: Node 0 Normal free:660640kB boost:32768kB min:63384kB low:93248kB high:123112kB reserved_highatomic:270336KB active_anon:16610520kB inactive_anon:1301292kB active_file:4979836kB inactive_file:4979072kB unevictable:32kB writepending:19872kB present:30395904kB managed:29867976kB mlocked:32kB bounce:0kB free_pcp:38796kB local_pcp:492kB free_cma:0kB Feb 18 14:10:55 CRUNCH kernel: lowmem_reserve[]: 0 0 0 0 0 Feb 18 14:10:55 CRUNCH kernel: Node 1 Normal free:125188kB boost:59392kB min:93244kB low:126268kB high:159292kB reserved_highatomic:2048KB active_anon:9206180kB inactive_anon:3987664kB active_file:9857148kB inactive_file:6922696kB unevictable:32kB writepending:60296kB present:33553920kB managed:33026000kB mlocked:32kB bounce:0kB free_pcp:34140kB local_pcp:3516kB free_cma:0kB Feb 18 14:10:55 CRUNCH kernel: lowmem_reserve[]: 0 0 0 0 0 Feb 18 14:10:55 CRUNCH kernel: Node 2 Normal free:1074724kB boost:114688kB min:148540kB low:181564kB high:214588kB reserved_highatomic:10240KB active_anon:3228760kB inactive_anon:1957716kB active_file:11863852kB inactive_file:13841864kB unevictable:96kB writepending:14400kB present:33553920kB managed:33026004kB mlocked:96kB bounce:0kB free_pcp:40912kB local_pcp:1552kB free_cma:0kB Feb 18 14:10:55 CRUNCH kernel: lowmem_reserve[]: 0 0 0 0 0 Feb 18 14:10:55 CRUNCH kernel: Node 3 Normal free:97796kB boost:0kB min:33848kB low:66868kB high:99888kB reserved_highatomic:2048KB active_anon:16442292kB inactive_anon:3111444kB active_file:5915184kB inactive_file:5066176kB unevictable:3184kB writepending:27980kB present:33553920kB managed:33022812kB mlocked:96kB bounce:0kB free_pcp:39252kB local_pcp:2512kB free_cma:0kB Feb 18 14:10:55 CRUNCH kernel: lowmem_reserve[]: 0 0 0 0 0 Feb 18 14:10:55 CRUNCH kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB Feb 18 14:10:55 CRUNCH kernel: Node 0 DMA32: 12435*4kB (UME) 3931*8kB (UME) 1486*16kB (UME) 582*32kB (UME) 182*64kB (UME) 177*128kB (UME) 34*256kB (UME) 25*512kB (UME) 56*1024kB (UME) 1*2048kB (U) 0*4096kB = 238788kB Feb 18 14:10:55 CRUNCH kernel: Node 0 Normal: 47593*4kB (UMEH) 11404*8kB (UMEH) 2380*16kB (UMEH) 1763*32kB (UMEH) 851*64kB (UMEH) 411*128kB (UMH) 88*256kB (H) 66*512kB (H) 36*1024kB (H) 37*2048kB (H) 2*4096kB (H) = 660324kB Feb 18 14:10:55 CRUNCH kernel: Node 1 Normal: 859*4kB (UME) 5383*8kB (UME) 3166*16kB (UME) 670*32kB (UM) 73*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (H) 0*4096kB = 125316kB Feb 18 14:10:55 CRUNCH kernel: Node 2 Normal: 24777*4kB (UME) 12657*8kB (UME) 11249*16kB (UME) 8873*32kB (UME) 5954*64kB (UME) 143*128kB (M) 3*256kB (M) 0*512kB 0*1024kB 5*2048kB (H) 0*4096kB = 1074652kB Feb 18 14:10:55 CRUNCH kernel: Node 3 Normal: 955*4kB (UME) 1415*8kB (UME) 671*16kB (UME) 1678*32kB (UME) 249*64kB (UME) 3*128kB (M) 0*256kB 0*512kB 0*1024kB 1*2048kB (H) 0*4096kB = 97940kB Feb 18 14:10:55 CRUNCH kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Feb 18 14:10:55 CRUNCH kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 18 14:10:55 CRUNCH kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Feb 18 14:10:55 CRUNCH kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 18 14:10:55 CRUNCH kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Feb 18 14:10:55 CRUNCH kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 18 14:10:55 CRUNCH kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Feb 18 14:10:55 CRUNCH kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 18 14:10:55 CRUNCH kernel: 16874061 total pagecache pages Feb 18 14:10:55 CRUNCH kernel: 0 pages in swap cache Feb 18 14:10:55 CRUNCH kernel: Free swap = 0kB Feb 18 14:10:55 CRUNCH kernel: Total swap = 0kB Feb 18 14:10:55 CRUNCH kernel: 33525359 pages RAM Feb 18 14:10:55 CRUNCH kernel: 0 pages HighMem/MovableOnly Feb 18 14:10:55 CRUNCH kernel: 552063 pages reserved Feb 18 14:10:55 CRUNCH kernel: 0 pages cma reserved I have not had any system hangs, or any evident operational issues -- diags attached I did read about setting /boot/config/share.cfg shareNOFILE="813384" Is this along the same route for remedy? crunch-diagnostics-20250218-1421.zip
February 19, 20251 yr Community Expert ? VM unraid? Review of Unraid Call Trace Error and Recommended Fix Your Unraid server is experiencing a kernel call trace error related to memory allocation, specifically within lsof. This issue appears to be linked to high memory usage and potential memory fragmentation, causing the __vmalloc_node_range() function to fail in allocating memory. Analysis of the Call Trace Key Functions Involved: __vmalloc_node_range(): Allocates virtual memory, which seems to be failing. kvmalloc_node(): A kernel memory allocation function. seq_read_iter(): A function related to sequential file reading (likely triggered by lsof). Possible Causes: Insufficient contiguous memory: Even though total memory usage seems fine, fragmentation might prevent large allocations. Memory exhaustion: The call trace shows significant memory usage, and Unraid appears to have no swap space configured, which could cause allocation failures. Kernel bug: Since Unraid uses a customized Linux kernel, this could be a kernel-related issue. Potential fixes: use a btrfs cache disk and install a swap plugin and make a swap file to help with over memory commitment issues... Additional unraid grub options may be needed based on this data... ?I think this is a limiation issues of you viratulizaing unraid... append isolcpus=12,14,16 mitigations=off iommu=pt nohz_full=12,14,16 rcu_nocbs=12,14,16 transparent_hugepage=never Explanation: transparent_hugepage=never: Disables huge pages, which might be causing fragmentation. iommu=pt: Ensures direct access to IOMMU without additional memory overhead. mitigations=off: Reduces kernel overhead caused by speculative execution mitigations.
February 19, 20251 yr Community Expert Review this if virtualizing with proxmox and the additional unraid and options that you may need to run:
February 19, 20251 yr Author just jumping in to say thanks for reviewing, but unraid is running baremetal and ive never needed swap before. Im going to digest your first post and see if that takes me anywheres
February 20, 20251 yr Author Solution @bmartino1 I've implemented a number of "fixes" and preliminary testing is positive: added to boot options: append mitigations=off iommu=pt transparent_hugepage=never installed swap plugin and configured a 8gb swapfile on nvme added shareNOFILE="813384" to /boot/config/share.cfg I still dont quite get why swap is necessary after all these years of not using + having a load of "free" memory
February 27, 20251 yr For the record, I am seeing the exact same issue. New intermittent vmalloc call traces since upgrade to 7.0, but no noticeable performance issues. 320gb of ram. I'll implement the above recommended steps and report back.
March 1, 20251 yr Author checking in so far i havent had any call traces since implementing my fixes i still dont quite understand why i would need to use swap after all these years of not having it, but i am hesitant to get rid of it since ive made headway
March 10, 20251 yr I'm experiencing this vmalloc issues as well ever since updating to unraid 7 from v6.12.14. I have 256 GB of ram and am using less than half of it. I will implement a swap file first and then the boot modifications. @therapistwhich swap file plugin did you use? The only one I see in the appstore is called "Swapfile for Unraid 6.9". Also, what does command "added shareNOFILE="813384" to /boot/config/share.cfg" do? Since upgrading to Unraid 7, this call trace started and my server has been hard crashing. I'm wondering if the two are related. Edited March 10, 20251 yr by msalad
March 10, 20251 yr Author Hey @msalad So I actually tested without the swap file plug-in and have not had any issues yet. With the swap file I was averaging 44 to 46% usage of RAM and went up to 56 to 57% after disabling which makes sense because the swap file had about 12GB used (10% of my 128GB) Thats the exact app I used though The added code changes "the number of fuse file descriptors" which ive never had to adjust before, but here we are. You dont actually have to add that line anymore...if you go to settings/global share settings you can set that parameter while array is stopped
March 12, 20251 yr On 3/10/2025 at 7:03 PM, therapist said: Hey @msalad So I actually tested without the swap file plug-in and have not had any issues yet. With the swap file I was averaging 44 to 46% usage of RAM and went up to 56 to 57% after disabling which makes sense because the swap file had about 12GB used (10% of my 128GB) Thats the exact app I used though The added code changes "the number of fuse file descriptors" which ive never had to adjust before, but here we are. You dont actually have to add that line anymore...if you go to settings/global share settings you can set that parameter while array is stopped Hey @therapist Thanks for the details. So I added the boot modifications iommu=pt and mitigations=off, I set the file descriptors to 813384, and I installed the swap file plugin. I didn't disable hugepages. For my swapfile, I put it on a nvme drive as an unassigned device, set the swapfile size to 32 GB, and swapiness=10. I have 215 GB of ram free right now but even with swapiness=10, I am seeing the swapfile being used - about 110 MB in the first ~30 min of enabling it. I do not understand what's going on. The swapfile shouldn't be seeing use until my free memory is < 10%... I reached out to unraid support before doing all of this but they simply said to try your fixes and didn't offer any additional insight. Not sure how to diagnose further but this is definitely concerning. I'll report back to unraid support with these results and see what they say
March 16, 20251 yr Any new findings on your end@therapist? With my swapfile set to 32GB and swapiness=10, 15GB was filled up in 48 hrs. Even though I have 200 GB of free ram right now. Makes no sense. Still no help from my ticket that's open with Unraid Support - they told me to "run some tests and let us know how it goes"... Not super helpful. I reconfigured the swapfile to 256 GB and swappiness=1. Gonna see how fast it fills up now
March 16, 20251 yr Community Expert 9 hours ago, msalad said: With my swapfile set to 32GB and swapiness=10, 15GB was filled up in 48 hrs. Even though I have 200 GB of free ram This is normal, if you have swap, it will be used, more or less depending on the settings, but it will still be used even if there's a lot of free RAM.
March 17, 20251 yr Author @msalad no news is good news i have had zero call traces since implementing the fixes. have been without swap active going on a little over a week and all seems OK. aside from an occasional hiccup/stutter with a w10 VM that ive had operational forever I am liking unraid 7.0
March 21, 20251 yr On 3/16/2025 at 5:40 AM, JorgeB said: This is normal, if you have swap, it will be used, more or less depending on the settings, but it will still be used even if there's a lot of free RAM. @JorgeBThank you for this info. I have since downgraded back to Unraid 6.12.15 and no longer experience this issue with nemory. I'm not sure what is going on with the Unraid 7.0/7.0.1 kernel but I no longer experience this call trace after downgrading and my server stopped hard crashing. On 3/17/2025 at 4:36 PM, therapist said: @msalad no news is good news i have had zero call traces since implementing the fixes. have been without swap active going on a little over a week and all seems OK. aside from an occasional hiccup/stutter with a w10 VM that ive had operational forever I am liking unraid 7.0 Unfortunately the fixes didn't help for me - my server kept crashing and that call trace kept occurring. I didn't disable hugepages like you did though. I have since downgraded to Unraid 6.12.15 and no longer have any issues. No call traces and no server crashes. Not sure what is going on with the Unraid 7.0/7.0.1 kernel but it was very unstable for me. Edited March 21, 20251 yr by msalad
March 29, 20251 yr I got the same error as this first user on this topic, trying just the increase of the fuse file descriptors from 40960 to 813384. If I get the error again ill try disabling hugepage files and see if that helps. but ive read that disabling that can brick your docker applications sql databases Edited March 30, 20251 yr by Kognac
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.