November 21, 20187 yr I upgraded to a TR 2950X last week after my last mobo died and rather than sourcing a poor X99 board from what was still retailing, I decided to upgrade. It's gone really well, except for high memory usage that just seems to grow in the first 24 hours of usage to the point where I run into OOM problems. Usually I can run 3x 8GB VMs, a 4GB VM and lots of dockers with memory to spare - now I can only run 1-2 VMs safely. I even installed the swap plugin last night and that's now being using 100%. I managed to track down that Jackett was using 10GB and I fixed that, but can I get help sourcing the problem please. Thanks highlander-diagnostics-20181121-2147.zip top - 21:39:54 up 1 day, 2:09, 2 users, load average: 15.82, 15.02, 16.51 Tasks: 823 total, 1 running, 820 sleeping, 0 stopped, 2 zombie %Cpu(s): 11.4 us, 4.4 sy, 0.6 ni, 71.2 id, 11.7 wa, 0.0 hi, 0.6 si, 0.0 st MiB Mem : 64302.1 total, 519.2 free, 46732.1 used, 17050.8 buff/cache MiB Swap: 12288.0 total, 0.0 free, 12288.0 used. 15748.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19054 root 20 0 4738240 4.2g 21528 S 215.2 6.7 3523:38 qemu-system-x86 46448 root 20 0 9747840 8.1g 20340 S 138.4 12.9 1881:50 qemu-system-x86 119982 nobody 20 0 13.6g 1.7g 0 S 75.2 2.7 856:25.16 java 31471 root 22 2 2907420 2.2g 18040 S 29.5 3.5 67:30.76 rcloneorig 14545 root 20 0 0 0 0 S 15.2 0.0 220:21.53 unraidd 44111 root 22 2 10136 1464 1128 D 10.3 0.0 1:12.58 rsync 44109 root 22 2 10244 2508 2056 S 9.6 0.0 1:10.03 rsync 15849 root 20 0 3537760 37916 19444 S 6.6 0.1 98:26.28 docker-containe 15138 root 20 0 1841844 18120 796 S 5.3 0.0 138:43.96 shfs 117826 nobody 20 0 5015844 283144 964 S 5.3 0.4 137:26.36 nzbget 61515 nobody 20 0 662320 482436 1728 S 4.0 0.7 198:43.13 transmission-da 117807 root 20 0 0 0 0 D 4.0 0.0 0:40.96 kworker/u256:17+flush-9:3 15828 root 20 0 3487504 86360 35216 S 3.6 0.1 61:51.55 dockerd 92800 root 20 0 47792 26860 20344 S 3.3 0.0 0:00.10 docker 1337 root 20 0 0 0 0 S 3.0 0.0 35:45.77 kswapd0 4959 root 20 0 34200 10672 7860 D 2.0 0.0 6:11.80 mc 56962 root 20 0 12.5g 122364 4904 S 2.0 0.2 19:47.04 python 59943 root 22 2 2316708 1.3g 18928 S 1.3 2.1 50:29.47 rcloneorig 2624 root 20 0 8200 4396 1276 S 0.7 0.0 10:13.00 haveged 10122 root 20 0 0 0 0 I 0.7 0.0 0:43.72 kworker/u256:28-btrfs-endio 13952 root 20 0 350436 2620 1936 S 0.7 0.0 5:19.77 emhttpd 28898 root 20 0 0 0 0 D 0.7 0.0 0:58.16 kworker/u256:13+flush-9:1 45858 102 20 0 44908 220 0 S 0.7 0.0 0:08.93 avahi-daemon 45859 nobody 20 0 1410632 130220 14344 S 0.7 0.2 121:18.42 Plex Media Serv 46461 root 20 0 0 0 0 S 0.7 0.0 14:18.54 vhost-46448 64360 nobody 20 0 3897776 1.8g 7412 S 0.7 2.9 89:34.27 mono 65113 root 20 0 0 0 0 I 0.7 0.0 0:35.27 kworker/u256:7-events_power_efficient 77244 root 20 0 0 0 0 I 0.7 0.0 0:02.20 kworker/u256:6-btrfs-endio 95933 root 20 0 0 0 0 I 0.7 0.0 0:48.71 kworker/u256:20-btrfs-endio 117805 root 20 0 0 0 0 I 0.7 0.0 0:14.43 kworker/u256:14-btrfs-endio 125302 root 20 0 0 0 0 I 0.7 0.0 0:41.51 kworker/u256:25-btrfs-endio 127782 root 20 0 0 0 0 I 0.7 0.0 1:00.62 kworker/u256:30-btrfs-endio 10 root 20 0 0 0 0 I 0.3 0.0 3:00.66 rcu_sched 101 root 20 0 0 0 0 S 0.3 0.0 1:21.29 ksoftirqd/18 7302 root 20 0 0 0 0 I 0.3 0.0 0:06.55 kworker/0:5-xfs-cil/md3 10757 root 22 2 160436 23840 18412 S 0.3 0.0 8:20.29 php 13980 root 20 0 84864 5228 3856 S 0.3 0.0 1:34.50 ttyd 19142 root 20 0 0 0 0 S 0.3 0.0 0:29.06 kvm-pit/19054 21594 nobody 20 0 2432084 59864 0 S 0.3 0.1 15:55.11 mysqld 22771 nobody 20 0 19.2g 334964 0 S 0.3 0.5 7:10.63 java 25191 nobody 20 0 904736 74272 1656 S 0.3 0.1 12:07.96 mongod 39379 nobody 20 0 1171124 76768 2812 S 0.3 0.1 0:09.05 Plex Script Hos 46469 root 20 0 0 0 0 S 0.3 0.0 1:53.26 kvm-pit/46448 60976 nobody 20 0 1367464 41744 2648 S 0.3 0.1 0:25.62 Plex Script Hos 63245 nobody 20 0 295316 79964 2928 S 0.3 0.1 4:24.19 python 64587 root 20 0 89180 9544 1812 S 0.3 0.0 4:06.66 fail2ban-client 91157 root 20 0 19840 4120 2548 R 0.3 0.0 0:00.13 top 124014 nobody 20 0 1033988 17212 2856 S 0.3 0.0 3:15.33 crashplan 125751 nobody 20 0 1033592 18392 5700 S 0.3 0.0 5:48.84 crashplan 1 root 20 0 2444 1672 1572 S 0.0 0.0 0:09.57 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.21 kthreadd 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kblockd 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq 9 root 20 0 0 0 0 S 0.0 0.0 0:36.73 ksoftirqd/0 11 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh 12 root rt 0 0 0 0 S 0.0 0.0 0:01.99 migration/0
November 22, 20187 yr Your syslog is full of this: Nov 20 19:36:52 Highlander kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_bmap_validate_extent+0x28/0xa8 [xfs], inode 0x9473ea5 xfs_iread_extents(2) Nov 20 19:36:52 Highlander kernel: 00000000ab79d81e: 90 b0 76 39 2c 15 00 00 00 af 45 b5 df 55 00 00 ..v9,.....E..U.. Nov 20 19:36:52 Highlander kernel: XFS (nvme0n1p1): Unmount and run xfs_repair I recommend you follow the advice in the third line. I don't know if that will fix your memory usage problem but it will make it easier to read your syslog
November 22, 20187 yr Out of interest, you say you noticed and fixed a Jackett issue - I've been having OOM errors that I'm looking into. How did you identify and fix the issue? And sorry for not helping...
November 22, 20187 yr Author 11 hours ago, John_M said: Your syslog is full of this: Nov 20 19:36:52 Highlander kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_bmap_validate_extent+0x28/0xa8 [xfs], inode 0x9473ea5 xfs_iread_extents(2) Nov 20 19:36:52 Highlander kernel: 00000000ab79d81e: 90 b0 76 39 2c 15 00 00 00 af 45 b5 df 55 00 00 ..v9,.....E..U.. Nov 20 19:36:52 Highlander kernel: XFS (nvme0n1p1): Unmount and run xfs_repair I recommend you follow the advice in the third line. I don't know if that will fix your memory usage problem but it will make it easier to read your syslog Sadly, I didn't spot this and my nvme drive lost all it's files. Nothing critical lost, just painful having to rebuild a few VMs today.
November 22, 20187 yr Author 2 hours ago, Cessquill said: Out of interest, you say you noticed and fixed a Jackett issue - I've been having OOM errors that I'm looking into. How did you identify and fix the issue? And sorry for not helping... I never read the full github page for Jackett, so tucked away at the bottom it says: Quote Disable autoupdates in the webui to prevent jackett crashing, the image is refreshed weekly so pick up updates that way. I hadn't and this causes the docker to go a bit bonkers and it's memory usage was 10GB...... https://hub.docker.com/r/linuxserver/jackett/
November 22, 20187 yr 15 minutes ago, DZMM said: I never read the full github page for Jackett, so tucked away at the bottom it says: I hadn't and this causes the docker to go a bit bonkers and it's memory usage was 10GB...... https://hub.docker.com/r/linuxserver/jackett/ Thanks for that. And damn - I have "disable auto update" checked already. Thought I might have got lucky there.
December 15, 20187 yr Author What's the best way to check memory usage? I'm still having problems and when I use top or advisor I can only account for about 50% when unraid is saying about 90%, which is stopping me launching my last VM? Thanks in advance
Archived
This topic is now archived and is closed to further replies.