Memory usage very HIGH? [Solved]


Recommended Posts

Hi All

 

I have 64 Gigs of RAM (Useable 62.8 GiB)

I have multiple Dockers (Not been a issue before adding new VM's)

image.png.d68666ce1d61f484e613ea773e5ecefc.png

 

My VM's

2 x Windows 10 = 6144 MB = 12G

1 x pfSense = 3072 MB = 3G

1 x Hassio = 2048 MB = 2G

1 x Xpenology = 8192 MB = 8G

Total: 25G

 

Docker 38% of 62.8G = 24G

 

Left = 13G

 

The Load:

image.png.bb3ea5052816a2059dea068c52bf5f0a.png

 

TOP:

image.thumb.png.06fdedb946277d491921dbe44c8745c1.png

 

Do I just need more RAM? (This system have a max of 64G 😞 )

Any tips to where I can cut memory usage?

I got the CPU usage down by selecting different cores for each system

 

As always your insight and expertise is much appreciated!

 

Edited by casperse
Link to comment
12 minutes ago, jonathanm said:

Keep in mind that you are excluding the host (containers included) from using any RAM you give to VM's. Many times it's better to give the VM absolute minimum required resources for performance and allow the host to manage everything you can give it.

Yes I Agree, but having:

My VM's

2 x Windows 10 = 3072 + 6144 MB = 9G

1 x pfSense = 3072 MB = 3G

1 x Hassio = 2048 MB = 2G

1 x Xpenology = 16384 MB = 16G (I found that I need to keep allocation = The orig. HW)

A Total of: 30G - out of a total of 64G?

 

I can't even start the last WIN 10 with 3G now without getting the :

image.png.3ccc21f255ff5d095899204688164cb6.png

 

Shouldn't this be possible and then letting the host to use the last 34 Gigs ?

What is the recommended memory value for a windows 10 for office work?

 

 

 

 

Link to comment

A long shot...

if changes was made to "dirty" memory, and there is a lot of unwritten cache, could that be the reason why the system can't free up enough memory to start the VM?

 

I know I asked it as a question. The reason is I am not 100% sure myself, how the system would react to this situation. But it is a possibility, I think.

 

/Alphahelix

Link to comment

This tweak has nothing to do with your RAM usage. The source of your problem must be something else (or you used a path, that targeted your RAM). Do you use transcoding to RAM and use the /tmp or /shm folder? Than this is your problem.

 

Next time you should investigate the RAM usage before restarting your server / containers. Use this command:

ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n

In addition check the sizes of your Ramdisks as follows:

df -h -t tmpfs

And finally you could check the size of your tmp folder which is even located in your RAM:

du -sh /tmp

 

Edited by mgutt
  • Like 2
  • Thanks 1
Link to comment
45 minutes ago, mgutt said:

This tweak has nothing to do with your RAM usage. The source of your problem must be something else (or you used a path, that targeted your RAM). Do you use transcoding to RAM and use the /tmp or /shm folder? Than this is your problem.

 

Next time you should investigate the RAM usage before restarting your server / containers. Use this command:


ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n

 

Thanks! much appreciated! -  I will try this next time I try the "cache" path again. (Yes I transcode to RAM but this was not the issue! - I had no trancodes and I even tried stopping PLEX)

 

Edited by casperse
Link to comment
58 minutes ago, mgutt said:

Next time you should investigate the RAM usage before restarting your server / containers. Use this command:

I don't get the error anymore but Memory is going up

 

image.png.1ccc33b68612073094c7c64818ebccb1.png

 

.....
0.00390625 MB           sh
0.0507812 MB            /usr/sbin/crond
0.0664062 MB            /bin/sh
0.0664062 MB            /bin/sh
0.0664062 MB            /usr/bin/tini
0.09375 MB              /usr/sbin/acpid
0.101562 MB             /usr/sbin/wsdd
0.109375 MB             /usr/sbin/avahi-dnsconfd
0.167969 MB             dbus-run-session
0.167969 MB             logger
0.214844 MB             /usr/sbin/crond
0.261719 MB             avahi-daemon:
0.289062 MB             xcompmgr
0.296875 MB             dbus-daemon
0.300781 MB             /bin/sh
0.316406 MB             /usr/bin/dbus-daemon
0.347656 MB             /usr/bin/tini
0.359375 MB             bash
0.367188 MB             bash
0.449219 MB             /usr/sbin/crond
0.464844 MB             /bin/bash
0.464844 MB             /usr/bin/tini
0.5625 MB               /bin/bash
0.679688 MB             sleep
0.695312 MB             sleep
0.710938 MB             sleep
0.714844 MB             sleep
0.726562 MB             /usr/local/sbin/shfs
0.734375 MB             /bin/timeout
0.738281 MB             sleep
0.753906 MB             sleep
0.984375 MB             nginx:
0.984375 MB             nginx:
0.984375 MB             nginx:
1.01953 MB              nginx:
1.1875 MB               nginx:
1.21875 MB              /usr/bin/privoxy
1.32812 MB              /usr/bin/openvpn
1.39453 MB              nginx:
1.39453 MB              nginx:
1.39453 MB              nginx:
1.46875 MB              /usr/sbin/atd
1.53906 MB              /bin/bash
1.55078 MB              /usr/sbin/inetd
1.64453 MB              init
1.67578 MB              /sbin/agetty
1.69531 MB              /sbin/agetty
1.74219 MB              /sbin/agetty
1.77734 MB              /sbin/agetty
1.78125 MB              /sbin/agetty
1.78516 MB              /sbin/agetty
1.78906 MB              sort
1.79297 MB              /usr/sbin/crond
2.01562 MB              /sbin/rpcbind
2.03906 MB              /bin/bash
2.04297 MB              /bin/bash
2.07812 MB              /usr/bin/dbus-daemon
2.14844 MB              nginx:
2.15234 MB              nginx:
2.17578 MB              /usr/sbin/rpc.mountd
2.19141 MB              /bin/bash
2.24609 MB              /usr/lib/plexmediaserver/Plex
2.26562 MB              /bin/bash
2.29297 MB              /usr/sbin/rsyslogd
2.36328 MB              /bin/bash
2.42188 MB              /usr/bin/privoxy
2.42578 MB              ps
2.51172 MB              /bin/bash
2.59766 MB              /usr/sbin/sshd
2.60156 MB              /usr/bin/openvpn
2.80469 MB              /bin/bash
2.84766 MB              /usr/bin/tint2
2.89453 MB              /bin/bash
2.93359 MB              find
3.05469 MB              awk
3.39844 MB              /bin/bash
3.57812 MB              avahi-daemon:
3.72266 MB              nginx:
3.73438 MB              /usr/bin/docker-proxy
3.73438 MB              /usr/bin/docker-proxy
3.73438 MB              /usr/bin/docker-proxy
3.73438 MB              /usr/bin/docker-proxy
3.76953 MB              /usr/bin/openbox
3.77344 MB              /usr/bin/docker-proxy
3.80469 MB              /usr/bin/docker-proxy
3.80469 MB              /usr/bin/docker-proxy
3.80469 MB              /usr/bin/docker-proxy
3.80469 MB              /usr/bin/docker-proxy
3.83594 MB              /usr/bin/docker-proxy
3.83594 MB              /usr/bin/docker-proxy
3.83594 MB              /usr/bin/docker-proxy
3.84375 MB              /usr/bin/docker-proxy
3.85156 MB              /usr/bin/docker-proxy
3.85156 MB              /usr/bin/docker-proxy
3.85938 MB              /usr/bin/docker-proxy
3.85938 MB              /usr/bin/docker-proxy
3.86719 MB              /usr/bin/docker-proxy
3.89062 MB              /usr/bin/docker-proxy
3.89062 MB              /usr/bin/docker-proxy
3.89062 MB              /usr/bin/docker-proxy
3.96094 MB              /usr/bin/docker-proxy
3.97266 MB              /sbin/udevd
3.98828 MB              /usr/bin/docker-proxy
3.98828 MB              /usr/sbin/virtlockd
4.01953 MB              -bash
4.07812 MB              /usr/local/sbin/emhttpd
4.08594 MB              /usr/bin/docker-proxy
4.08984 MB              /usr/bin/docker-proxy
4.21875 MB              /usr/bin/docker-proxy
4.29688 MB              /usr/bin/docker-proxy
4.85156 MB              /usr/sbin/ntpd
4.89844 MB              /bitwarden_rs
4.98438 MB              /usr/bin/docker-proxy
5.10938 MB              php-fpm:
5.16016 MB              ttyd
5.66797 MB              /sbin/rpc.statd
5.91797 MB              /usr/bin/docker-proxy
5.98047 MB              nginx:
6.10547 MB              /sbin/haveged
6.19531 MB              /usr/bin/docker-proxy
6.24609 MB              nginx:
6.24609 MB              nginx:
6.24609 MB              nginx:
6.90625 MB              /usr/sbin/nmbd
7.23047 MB              /usr/sbin/virtlogd
7.28125 MB              /usr/sbin/smbd
7.51953 MB              php-fpm:
7.90234 MB              php-fpm:
8.04297 MB              nginx:
8.08203 MB              /usr/sbin/smbd
8.08203 MB              nginx:
8.4375 MB               /usr/bin/slim
8.93359 MB              containerd-shim
8.96484 MB              containerd-shim
8.98828 MB              nginx:
9.29297 MB              containerd-shim
9.30859 MB              containerd-shim
9.34766 MB              /usr/local/emhttp/plugins/unbalance/unbalance
9.38281 MB              containerd-shim
9.39844 MB              containerd-shim
9.4375 MB               containerd-shim
9.48047 MB              containerd-shim
9.48828 MB              containerd-shim
9.51953 MB              containerd-shim
9.51953 MB              containerd-shim
9.51953 MB              containerd-shim
9.53516 MB              containerd-shim
9.54297 MB              containerd-shim
9.57812 MB              containerd-shim
9.58594 MB              containerd-shim
9.61328 MB              containerd-shim
9.61719 MB              containerd-shim
9.66406 MB              containerd-shim
9.82031 MB              containerd-shim
9.92969 MB              /usr/bin/python3
9.96484 MB              /usr/sbin/winbindd
10.7812 MB              php-fpm:
10.8203 MB              /usr/sbin/winbindd
11.0234 MB              /app/unpackerr
11.3438 MB              php-fpm:
11.5898 MB              containerd-shim
11.7188 MB              containerd-shim
11.793 MB               containerd-shim
11.9727 MB              php-fpm:
12.3555 MB              /usr/sbin/winbindd
12.7773 MB              php-fpm:
12.8008 MB              php-fpm:
13.3047 MB              php-fpm:
13.418 MB               /usr/local/emhttp/plugins/controlr/controlr
13.4375 MB              php-fpm:
13.4375 MB              php-fpm:
13.5039 MB              php-fpm:
13.5039 MB              php-fpm:
13.8086 MB              php-fpm:
14.9961 MB              /usr/sbin/smbd
15.0273 MB              /usr/sbin/smbd
15.418 MB               /usr/bin/python
16.5586 MB              /usr/bin/python
16.6602 MB              /usr/bin/python
17.1328 MB              /usr/sbin/smbd
17.6523 MB              /usr/bin/python
21.6797 MB              /usr/sbin/smbd
22.8477 MB              /usr/sbin/libvirtd
26.3164 MB              python
26.4609 MB              Plex
34 MB           docker
34.2461 MB              /usr/libexec/Xorg
35.1953 MB              /usr/sbin/Xvnc
36.5742 MB              krusader
38.4258 MB              Plex
39.2656 MB              /usr/bin/python
39.6094 MB              containerd
39.6484 MB              php-fpm:
39.7539 MB              php-fpm:
39.9492 MB              php-fpm:
41.4102 MB              deluge-web
43.1602 MB              Plex
51.0469 MB              /usr/sbin/python3
54.082 MB               python3
69.332 MB               python3
81.8398 MB              /usr/bin/dockerd
138.496 MB              /app/radarr/bin/Radarr
146.586 MB              /usr/local/sbin/shfs
158.039 MB              /opt/ombi/Ombi
166.699 MB              /usr/lib/plexmediaserver/Plex
167.602 MB              /app/Jackett/jackett
178.035 MB              /app/radarr/bin/Radarr
192.16 MB               mono
197.223 MB              mono
230.992 MB              rslsync
291.141 MB              mono
329.223 MB              system/EmbyServer
331.602 MB              /usr/sbin/mysqld
350.137 MB              java
2120.78 MB              /usr/bin/qemu-system-x86_64
3245.57 MB              /usr/bin/qemu-system-x86_64
4245.45 MB              /usr/bin/qemu-system-x86_64
4271.35 MB              /usr/bin/qemu-system-x86_64
16604 MB                /usr/bin/qemu-system-x86_64

The last 5 is the VM's with a total of 30 Gigs is there anything else above docker weise that stands out?

(I was surprised that Emby uses so much and that Plex have so many different processes)

Link to comment

It seems you did not setup a separate transcoding ramdisk. Do you use /dev/shm? Then this could be your problem. /dev/shm has a total size of 32GB and Plex will fill it over time. Not only for live transcoding, even for optimized versions and live tv streams etc. This is a general temp folder for all transcodings. Its better to define a separate folder with a specific size as described here:

https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460

 

I used 4GB in the past which works flawlessly up to 3 or 4 parallel running 4K transcodings. After I did a benchmark and hit the maximum of my iGPU, I changed it to 8GB as I received an "out of storage" error in Plex, but finally this shouldn't be needed in usual usage. 4GB is sufficient because Plex cleans up the folder automatically when it occupies >95% of the total size.

 

Plex uses multiple processes to load-balance them across multiple cores/threads. Maybe Emby has only one single-thread process?!

 

20 minutes ago, casperse said:

but Memory is going up

 

Repeat the command later, so we can compare the results.

 

Another command to check the size of your /tmp folder (which targets your RAM, too):

du -sh /tmp

 

Edited by mgutt
Link to comment
1 hour ago, mgutt said:

It seems you did not setup a separate transcoding ramdisk. Do you use /dev/shm? Then this could be your problem. /dev/shm has a total size of 32GB and Plex will fill it over time. Not only for live transcoding, even for optimized versions and live tv streams etc. This is a general temp folder for all transcodings. Its better to define a separate folder with a specific size as described here:

https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460

 

I used 4GB in the past which works flawlessly up to 3 or 4 parallel running 4K transcodings. After I did a benchmark and hit the maximum of my iGPU, I changed it to 8GB as I received an "out of storage" error in Plex, but finally this shouldn't be needed in usual usage. 4GB is sufficient because Plex cleans up the folder automatically when it occupies >95% of the total size.

 

Plex uses multiple processes to load-balance them across multiple cores/threads. Maybe Emby has only one single-thread process?!

 

 

Repeat the command later, so we can compare the results.

 

Another command to check the size of your /tmp folder (which targets your RAM, too):


du -sh /tmp

 

I used this guide when I first created my Plex server to use my RAM for transcoding:

Guess things have evolved since then ;-)

UPDATE: Did the script and its working

image.png.5d46d3850add63213ee46e62a661049d.png

One question should I schedule this to run in the User Scripts?

Edited by casperse
Link to comment
31 minutes ago, casperse said:

Guess things have evolved since then ;-)

Where would I set the script?

I would say the guide was not optimal since the beginning as "/tmp" uses (as far as I know) the complete RAM size.

 

In my last post you find the link to the full guide to create a ramdisk with a specific size. I asked @jonp to update his guide, but it seems, he didn't have the time for it yet.

Edited by mgutt
Link to comment
7 minutes ago, mgutt said:

I would say the guide was wrong since beginning. It was never clever to use the full ram size as transcoding path (which "/tmp" does as far as I know).

 

In my last post you find the link to the full guide to create a ramdisk with a specific size.

I have created the script in the "User Scripts" and updated the path (Thanks!) its working

Only thing I can't figure out is if I have to enable schedule for this script here?

image.thumb.png.b00f9d5bec4a45581085d3287aefae7b.png

 

UPDATE: Excellent guide thanks all is running now!

Edited by casperse
Link to comment

I went back and enabled the cache path for the docker.img :-)

Only activity I can see is Plex scanning libraries - no transcoding

So the CPU could be indexing and preview thumbnails?

 

But memory is back to 99%-100%

image.png.ad13cacb71308dd34a4525289c2676d6.png

 

du -sh /tmp

20G     /tmp

 

df -h -t tmpfs

Filesystem      Size  Used Avail Use% Mounted on
tmpfs            32M  632K   32M   2% /run
tmpfs            32G  8.0K   32G   1% /dev/shm
cgroup_root     8.0M     0  8.0M   0% /sys/fs/cgroup
tmpfs           128M  1.9M  127M   2% /var/log
tmpfs           1.0M     0  1.0M   0% /mnt/disks
tmpfs           8.0G     0  8.0G   0% /tmp/PlexRamScratch

 

(Yes I did set the PlexRamScratch to 8G just to be sure.... like you did I have a P2000 doing the trancodes and I don't want the RAM to be a bottleneck)

 

ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n
0.00390625 MB           sh
0.0546875 MB            /usr/sbin/crond
0.0546875 MB            /usr/sbin/crond
0.0625 MB               /bin/sh
0.0625 MB               /bin/sh
0.0625 MB               curl
0.0664062 MB            /bin/sh
0.0664062 MB            /bin/sh
0.0703125 MB            /bin/sh
0.0703125 MB            /bin/sh
0.0703125 MB            /bin/sh
0.0703125 MB            /bin/sh
0.0703125 MB            /bin/sh
0.0703125 MB            xargs
0.0742188 MB            /usr/bin/tini
0.0820312 MB            /bin/sh
0.0820312 MB            /usr/bin/tini
0.0820312 MB            /usr/bin/tini
0.0820312 MB            sleep
0.0859375 MB            /bin/sh
0.0859375 MB            /bin/sh
0.0859375 MB            grep
0.09375 MB              /bin/sh
0.09375 MB              /bin/sh
0.09375 MB              /bin/sh
0.09375 MB              /bin/sh
0.09375 MB              /bin/sh
0.09375 MB              /bin/sh
0.09375 MB              /usr/sbin/acpid
0.0976562 MB            /bin/sh
0.101562 MB             /bin/sh
0.101562 MB             /usr/sbin/wsdd
0.105469 MB             /bin/sh
0.109375 MB             /bin/sh
0.109375 MB             /usr/sbin/avahi-dnsconfd
0.117188 MB             /bin/sh
0.152344 MB             php7
0.15625 MB              /bin/sh
0.167969 MB             dbus-run-session
0.167969 MB             logger
0.171875 MB             pgrep
0.191406 MB             curl
0.195312 MB             bash
0.253906 MB             bash
0.257812 MB             grep
0.257812 MB             grep
0.261719 MB             avahi-daemon:
0.277344 MB             /usr/sbin/crond
0.320312 MB             /usr/bin/dbus-daemon
0.320312 MB             xcompmgr
0.339844 MB             dbus-daemon
0.339844 MB             drill
0.417969 MB             curl
0.417969 MB             curl
0.417969 MB             curl
0.425781 MB             curl
0.457031 MB             curl
0.457031 MB             curl
0.46875 MB              /bin/bash
0.46875 MB              /bin/bash
0.492188 MB             /bin/bash
0.492188 MB             /bin/bash
0.496094 MB             curl
0.53125 MB              /usr/bin/privoxy
0.570312 MB             /bin/bash
0.585938 MB             /bin/bash
0.59375 MB              /bin/bash
0.644531 MB             /bin/bash
0.730469 MB             /bin/timeout
0.734375 MB             /bin/timeout
0.734375 MB             /bin/timeout
0.734375 MB             /bin/timeout
0.738281 MB             /bin/timeout
0.738281 MB             sleep
0.742188 MB             sleep
0.753906 MB             /usr/local/sbin/shfs
0.757812 MB             Plex
0.792969 MB             /bin/timeout
0.792969 MB             /bin/timeout
0.792969 MB             /bin/timeout
0.796875 MB             /bin/timeout
0.800781 MB             /bin/timeout
0.800781 MB             /bin/timeout
0.824219 MB             nginx:
0.851562 MB             curl
0.851562 MB             curl
0.851562 MB             curl
0.855469 MB             curl
0.910156 MB             curl
0.945312 MB             curl
0.980469 MB             nginx:
0.980469 MB             nginx:
0.980469 MB             nginx:
1 MB            find
1.01562 MB              nginx:
1.17969 MB              /usr/bin/privoxy
1.1875 MB               nginx:
1.32031 MB              nginx:
1.32812 MB              /usr/bin/openvpn
1.33594 MB              /usr/bin/openvpn
1.39453 MB              nginx:
1.40234 MB              nginx:
1.40625 MB              /usr/bin/docker-proxy
1.40625 MB              /usr/bin/docker-proxy
1.40625 MB              /usr/bin/docker-proxy
1.40625 MB              /usr/bin/docker-proxy
1.41016 MB              /usr/bin/docker-proxy
1.41016 MB              /usr/bin/docker-proxy
1.41016 MB              /usr/bin/docker-proxy
1.41016 MB              /usr/bin/docker-proxy
1.41016 MB              /usr/bin/docker-proxy
1.41016 MB              /usr/bin/docker-proxy
1.46875 MB              /usr/sbin/atd
1.50781 MB              nginx:
1.55078 MB              /usr/sbin/inetd
1.64453 MB              init
1.67578 MB              /sbin/agetty
1.69531 MB              /sbin/agetty
1.71875 MB              /usr/lib/plexmediaserver/Plex
1.74219 MB              /sbin/agetty
1.76562 MB              sort
1.77734 MB              /sbin/agetty
1.78125 MB              /sbin/agetty
1.78516 MB              /sbin/agetty
1.79297 MB              /usr/sbin/crond
2.01562 MB              /sbin/rpcbind
2.01562 MB              kdeinit5:
2.07812 MB              /usr/bin/dbus-daemon
2.17578 MB              /usr/sbin/rpc.mountd
2.28125 MB              find
2.29297 MB              /usr/sbin/rsyslogd
2.30078 MB              /usr/bin/docker-proxy
2.32031 MB              /bin/bash
2.32422 MB              ps
2.36328 MB              /usr/bin/docker-proxy
2.36328 MB              /usr/bin/docker-proxy
2.36719 MB              find
2.37109 MB              /usr/bin/docker-proxy
2.37109 MB              /usr/bin/docker-proxy
2.39062 MB              find
2.39453 MB              /bin/bash
2.40625 MB              find
2.42578 MB              /usr/bin/docker-proxy
2.51562 MB              find
2.54688 MB              find
2.57422 MB              find
2.57812 MB              /bin/bash
2.57812 MB              /bin/bash
2.59766 MB              /usr/sbin/sshd
2.62891 MB              find
2.63672 MB              /bin/bash
2.63672 MB              find
2.64062 MB              /bin/bash
2.64062 MB              /bin/bash
2.64062 MB              /bin/bash
2.64062 MB              /bin/bash
2.64062 MB              /bin/bash
2.64062 MB              /bin/bash
2.64062 MB              /bin/bash
2.64062 MB              /bin/bash
2.79297 MB              /bin/bash
2.84375 MB              /usr/lib/kf5/klauncher
2.89062 MB              awk
2.92969 MB              find
3.02344 MB              /usr/bin/tint2
3.08594 MB              php-fpm:
3.24219 MB              /bin/bash
3.57812 MB              avahi-daemon:
3.71875 MB              /usr/bin/docker-proxy
3.72266 MB              nginx:
3.74219 MB              /usr/bin/docker-proxy
3.76562 MB              /usr/bin/docker-proxy
3.76562 MB              /usr/bin/docker-proxy
3.77344 MB              /usr/bin/docker-proxy
3.78906 MB              /usr/bin/docker-proxy
3.80469 MB              /usr/bin/docker-proxy
3.83594 MB              /usr/bin/docker-proxy
3.84375 MB              /usr/bin/docker-proxy
3.87891 MB              /usr/bin/docker-proxy
3.90625 MB              /usr/sbin/virtlockd
3.92188 MB              /usr/bin/docker-proxy
3.97266 MB              /sbin/udevd
4.01953 MB              -bash
4.08203 MB              /usr/local/sbin/emhttpd
4.26562 MB              /usr/bin/docker-proxy
4.30469 MB              /usr/bin/openbox
4.61719 MB              /usr/bin/docker-proxy
4.77344 MB              /usr/bin/docker-proxy
4.78125 MB              /app/unpackerr
4.85156 MB              /usr/sbin/ntpd
5.16797 MB              ttyd
5.66797 MB              /sbin/rpc.statd
5.98047 MB              containerd-shim
5.99219 MB              /bitwarden_rs
6.04688 MB              containerd-shim
6.05859 MB              nginx:
6.09766 MB              containerd-shim
6.10547 MB              /sbin/haveged
6.19922 MB              containerd-shim
6.21094 MB              containerd-shim
6.23438 MB              containerd-shim
6.30859 MB              containerd-shim
6.36328 MB              containerd-shim
6.44141 MB              nginx:
6.44922 MB              containerd-shim
6.44922 MB              nginx:
6.45312 MB              nginx:
6.46094 MB              containerd-shim
6.50781 MB              containerd-shim
6.5625 MB               containerd-shim
6.62109 MB              containerd-shim
6.62109 MB              containerd-shim
6.88672 MB              nginx:
6.98828 MB              nginx:
7.03125 MB              /usr/sbin/nmbd
7.28516 MB              php-fpm:
7.29297 MB              /usr/sbin/smbd
7.42578 MB              containerd-shim
7.52734 MB              /usr/sbin/virtlogd
7.86719 MB              php-fpm:
7.86719 MB              php-fpm:
8.125 MB                /usr/sbin/smbd
8.22656 MB              containerd-shim
8.42578 MB              containerd-shim
8.43359 MB              containerd-shim
8.4375 MB               /usr/bin/slim
8.79297 MB              containerd-shim
8.83594 MB              containerd-shim
8.96094 MB              containerd-shim
8.98828 MB              nginx:
9.08203 MB              containerd-shim
9.74609 MB              containerd-shim
9.79297 MB              /usr/sbin/winbindd
9.89062 MB              php-fpm:
10.1094 MB              php-fpm:
10.3555 MB              /usr/bin/python3
10.9336 MB              /usr/sbin/winbindd
11.3438 MB              php-fpm:
12.2266 MB              /usr/local/emhttp/plugins/unbalance/unbalance
12.5273 MB              /usr/sbin/winbindd
13.0664 MB              php-fpm:
13.0664 MB              php-fpm:
13.0742 MB              php-fpm:
13.3711 MB              php-fpm:
13.4844 MB              php-fpm:
13.5625 MB              php-fpm:
13.5625 MB              php-fpm:
13.5977 MB              /usr/local/emhttp/plugins/controlr/controlr
14.6719 MB              /usr/bin/python
14.7695 MB              php-fpm:
14.8789 MB              /usr/bin/python
14.9844 MB              /usr/bin/python
15.0859 MB              /usr/sbin/smbd
15.1484 MB              /usr/sbin/smbd
15.1992 MB              /usr/sbin/smbd
17.4922 MB              /usr/bin/python
17.9062 MB              /usr/sbin/smbd
20.4023 MB              /usr/lib/plexmediaserver/Plex
20.6523 MB              /usr/sbin/smbd
21.168 MB               /usr/bin/python
22.4844 MB              /usr/sbin/libvirtd
25.3398 MB              Plex
26.5938 MB              python
34.2461 MB              /usr/libexec/Xorg
34.8789 MB              Plex
35.5273 MB              docker
37.1523 MB              Plex
37.3242 MB              Plex
37.9258 MB              containerd
38.168 MB               /usr/bin/python
39.0625 MB              deluge-web
40.3359 MB              php7
43.6914 MB              php-fpm:
43.8945 MB              php-fpm:
45.5078 MB              php-fpm:
51.5078 MB              /usr/sbin/python3
52.168 MB               /usr/sbin/Xvnc
54.707 MB               python3
64.25 MB                Plex
79.7578 MB              /usr/bin/dockerd
104.973 MB              python3
110.766 MB              /app/radarr/bin/Radarr
136.883 MB              /usr/local/sbin/shfs
141.602 MB              /app/Jackett/jackett
144.734 MB              /opt/ombi/Ombi
146.723 MB              mono
167.867 MB              /app/radarr/bin/Radarr
188.879 MB              mono
280.812 MB              mono
293.574 MB              system/EmbyServer
332.719 MB              /usr/sbin/mysqld
359.27 MB               java
788.543 MB              rslsync
914.41 MB               /usr/lib/plexmediaserver/Plex
2107.17 MB              /usr/bin/qemu-system-x86_64
3276.16 MB              /usr/bin/qemu-system-x86_64
4268.11 MB              /usr/bin/qemu-system-x86_64
4302 MB         /usr/bin/qemu-system-x86_64
5088.28 MB              krusader
16605.5 MB              /usr/bin/qemu-system-x86_64

Again thanks for your help! much appreciated

Link to comment

Check whats inside your /tmp dir. The size is 20GB! It should be nearly nothing (or at most 8GB, but regarding your df command the PlexScratch dir is empty, so it should be nothing inside this dir).

 

Example:

du -h --max-depth=3 /tmp

 

Why does krusader need 5GB?!

 

Which Plex Container are you using? (Linuxserver, binhex, original...)

 

Did you set the docker.img in the general settings to /mnt/cache or only the Plex config path or both? If possible provide screenshots of those set paths.

Edited by mgutt
Link to comment
du -h --max-depth=3 /tmp

0       /tmp/mc-root
0       /tmp/PlexRamScratch/Transcode/Detection
0       /tmp/PlexRamScratch/Transcode/Sessions
0       /tmp/PlexRamScratch/Transcode
0       /tmp/PlexRamScratch
0       /tmp/CA_logs
0       /tmp/Transcode/Detection
0       /tmp/Transcode/Sessions
0       /tmp/Transcode
20G     /tmp/transcoding-temp     <-------------------------------- Dont know why this is here?
8.0K    /tmp/user.scripts/tmpScripts/PlexRamScratch
8.0K    /tmp/user.scripts/tmpScripts
0       /tmp/user.scripts/running
4.0K    /tmp/user.scripts/finished
20K     /tmp/user.scripts
7.0M    /tmp/fix.common.problems
0       /tmp/ca_notices
4.0K    /tmp/emhttp
0       /tmp/unassigned.devices/scripts
16K     /tmp/unassigned.devices/config
28K     /tmp/unassigned.devices
16K     /tmp/recycle.bin
0       /tmp/community.applications/tempFiles/templates-community-apps
7.9M    /tmp/community.applications/tempFiles
7.9M    /tmp/community.applications
0       /tmp/ca.backup2/tempFiles
0       /tmp/ca.backup2
4.0K    /tmp/nvidia
0       /tmp/.X11-unix
0       /tmp/.ICE-unix
536K    /tmp/plugins
20G     /tmp

The " /tmp/transcoding-temp" might be Emby? can I remove this folder from RAM  (Emby is just for test and no one is using this server)

I use the Plex container from Plexinc so far this has worked best for me (Have tried them all)

 

Don't know did the Krusader install from a spaceinvader video long time ago?

 

Docker:

image.png.1f53664caee8cf424702411d5f1896a1.png

 

Plex: I didn't think I needed to change this to cache, but I guess that would be best since I have all 800G in my cache NVMe

image.png.3294e4ac2f80f5e442f52b872c2f51d3.png

So I will change this now:

image.thumb.png.91c257cae1f5556d2d93bbbd806112b3.png

(I just changed this for Emby also, using the same transcoding dir in RAM)

 

 

 

Link to comment

Can I just delete this? No idea where its from?

image.png.56594fb4ba44900468512b88e42dd31a.png

 

Deleting these old transcode folders (Shouldnt they be delete automatically?)

Update looks like this is from Emby:

image.png.973a47f4200735437f22c4f10c222c75.png

 

After deleting this folder, and pointing EMBY transcode to the same "PlexRamscratch" 

I now get:

du -sh /tmp
183M    /tmp

 

And my memory is now: down to normal JAHUU!

image.png.bddc55672e0e03bde33f5cd8f7ee5e5d.png

 

I don't know enough about this, but isn't memory temporary and turning your PC of should flush the tmp right?

Edited by casperse
Link to comment
1 hour ago, casperse said:

The " /tmp/transcoding-temp" might be Emby?

Yes, but why does Emby write those files only if you change the path of the docker.img. Where did Emby wrote those .ts files before you changed the path to "PlexRamScratch"?

 

If you don't know it, use this command to find them:

find / -path '/proc/*' -prune -o -path '/sys/*' -prune -o -path '/mnt/disk*/*' -prune -o -path '/mnt/user*/*' -prune -o -iname '*.ts' -print

 

1 hour ago, casperse said:

I use the Plex container from Plexinc

Me too. Works flawlessly.

 

1 hour ago, casperse said:

Don't know did the Krusader install from a spaceinvader video long time ago?

Did you use Krusader between your first and last memory check? Because at the beginning it had a low usage:

36.5742 MB krusader

And now it reached 5GB. If you used it, maybe it caches files while transfers are running or similar. But if you didn't use it all, this sounds like a bug to me. P.S. You can limit the RAM usage through this extra docker parameter (advanced view):

--memory=1G

1715095977_2020-10-1221_34_42.png.575b26cca953341bc619851e6e7a195e.png

 

1 hour ago, casperse said:

I didn't think I needed to change this to cache, but I guess that would be best since I have all 800G in my cache NVMe

Ok, now things become clearer. My original tweak is only to change the plex config path to /mnt/cache and not the general docker.img path. This means you quoted my tweak in your "warning", but you even don't use it:

https://forums.unraid.net/topic/88999-unraid-tweaks-for-media-server-performance/?tab=comments#comment-901831

 

Changing the general docker.img path is a different tweak, but I didn't even write a guide for this. I use it, but wasn't able to test it with multiple dockers. So its more a "beta" thing. Maybe Emby has a problem with that. If I find the time, I will test that. Can you send me a screenshot of your Emby settings so I can test this with the same settings?

 

1 hour ago, casperse said:

I just changed this for Emby also, using the same transcoding dir in RAM

I have no experience if Emby is able to clean up the dir as Plex does it. I would suggest to add an additional ramdisk for Emby alone. The steps are the same (could be added to your existing script):

mkdir /tmp/EmbyRamScratch
chmod -R 777 /tmp/EmbyRamScratch
mount -t tmpfs -o size=8g tmpfs /tmp/EmbyRamScratch

Now set this path in your emby container as transcoding path. By that it would be also easier to disinguish between the container's ramdisk usage.

 

1 hour ago, casperse said:

Shouldnt they be delete automatically

No idea. Maybe you find the answer here:

https://emby.media/community/index.php?/topic/54676-how-often-is-transcoding-temp-cleaned/

 

1 hour ago, casperse said:

turning your PC of should flush the tmp right

Correct. If you want to delete them without rebooting the server, use this command:

rm -r /tmp/transcoding-temp

 

1 hour ago, casperse said:

Plex: I didn't think I needed to change this to cache, but I guess that would be best since I have all 800G in my cache NVMe

 

If it does, then you could do that. But do not forget. If your SSD becomes full, Plex is not able to write any more data (will probably crash). Because of that I wrote in my guide, that it is useful to set a "Min. free space" in the Global Share Settings > Cache Settings, so caching through usual shares will not fill up your NVMe. Of course it does not influence the Plex usage. If Plex alone fully utilizes the NVMe, then you should leave it to /mnt/user or buy a bigger NVMe.

 

Side note: I hope you set up a backup of your NVMe ;)

Edited by mgutt
  • Like 1
Link to comment
On 10/12/2020 at 9:33 PM, mgutt said:

Yes, but why does Emby write those files only if you change the path of the docker.img. Where did Emby wrote those .ts files before you changed the path to "PlexRamScratch"?

Sorry for the delay replying you!

 

I found that no matter what path I write in Emby it will create a subdir in the tmp/ram dir as "transcoding-temp"

I deleted this and used your script to create a EmbyRamScratch that it can create the "transcoding-temp" under 

image.png.ff7b685c455d8dda9e552712b3c362a0.png

So far no problems after this change! works great!

Quote

 

1715095977_2020-10-1221_34_42.png.575b26cca953341bc619851e6e7a195e.png

Thanks I also added this to my docker setting running Krusader looks ok now!

Quote

Ok, now things become clearer. My original tweak is only to change the plex config path to /mnt/cache and not the general docker.img path. This means you quoted my tweak in your "warning", but you even don't use it:

https://forums.unraid.net/topic/88999-unraid-tweaks-for-media-server-performance/?tab=comments#comment-901831

Sorry I have updated my "warning" in the topic for tweaks that I created - The changes have made the biggest speed improvement to my setup so far!

No the docker.img have just always been on the cache (Just not used the cache in the path, only the prefer on cache setting before)

Quote

Changing the general docker.img path is a different tweak, but I didn't even write a guide for this. I use it, but wasn't able to test it with multiple dockers. So its more a "beta" thing. Maybe Emby has a problem with that. If I find the time, I will test that. Can you send me a screenshot of your Emby settings so I can test this with the same settings?

 

Sure thing this really utilize the NVMe

(Only drawback is that I had to upgrade it to a Samsung MZ-V7S2T0BW 970 Evo Plus [2 TB] $$$ and my limit in PCIe slots I can only have 1 NVMe)

 

image.png.e1472844eabc4c2894ab68d4e7c96fc4.png

 

image.thumb.png.c917c3be3f6bce657d42dac863682ef7.png

Quote

 

I have no experience if Emby is able to clean up the dir as Plex does it. I would suggest to add an additional ramdisk for Emby alone. The steps are the same (could be added to your existing script):

Done! and thanks!

Quote

As stated before Emby creates a "transcoding-temp" dir (Must be hardcoded) but I have pointed that to the new "EmbyRamScratch/trancoding-temp/"

Quote

If it does, then you could do that. But do not forget. If your SSD becomes full, Plex is not able to write any more data (will probably crash). Because of that I wrote in my guide, that it is useful to set a "Min. free space" in the Global Share Settings > Cache Settings, so caching through usual shares will not fill up your NVMe. Of course it does not influence the Plex usage. If Plex alone fully utilizes the NVMe, then you should leave it to /mnt/user or buy a bigger NVMe.

No only one NVMe and so far this is okay. I have around 8-900G for cache a day and rest for VM/Docker/Appdata/docker.img

Quote

 

Side note: I hope you set up a backup of your NVMe ;)

 

Yes I have the excellent community tools that creates backup of the VM's/Appdata/docker.img

I have excluded the 800G Plex appdata folder, have a old yearly backup -  and worst case is just that it needs to do a rescan of the library

 

Again thanks for all your help! much appreciated! 🙏

 

BTW: I have created a local shared temp that I have as prefer on cache. So if I want to have something permanently on the cache drive I can put it there also used for special files on NeXTcloud that requires fast access and transfer speeds 🙂Would it also be good to change path to /cache/ here?

 

Edited by casperse
Link to comment
57 minutes ago, casperse said:

Yes I have the excellent community tools that creates backup of the VM's/Appdata/docker.img

I have excluded the 800G Plex appdata folder, have a old yearly backup -  and worst case is just that it needs to do a rescan of the library

You don't need to backup "docker.img":

https://forums.unraid.net/topic/58690-does-it-make-sense-to-backup-the-docker-image/

1 hour ago, casperse said:

I have excluded the 800G Plex appdata folder

And what about the database with your "watched status", settings etc? PS maybe you like to backup the appdata folder with this script as its incremental:

https://forums.unraid.net/topic/97958-incremental-backup-through-rsync/?tab=comments#comment-903782

 

Once per month (on the first of course) would be sufficient.

 

1 hour ago, casperse said:

I have created a local shared temp that I have as prefer on cache. So if I want to have something permanently on the cache drive I can put it there also used for special files on NeXTcloud that requires fast access and transfer speeds 🙂Would it also be good to change path to /cache/ here?

 

Of course. Every write which uses direct disk access (/mnt/cache) will be much faster than with shared disk access (/mnt/user).

 

PS You could enable disk share, disable all shared access of your disks except of your cache disk, mount a subfolder of the cache disk as a local network drive on your client and by that you even use direct disk access while transfering files to your server.

 

I described this here in my Guide in #3:

https://forums.unraid.net/topic/97165-smb-performance-tuning/

 

And here are some user experiences regarding this, too:

https://forums.unraid.net/topic/92282-solved-workaround-how-to-write-to-nvme-pcie-cache-at-full-1-gbs-with-10-gbe-nic/

 

Don't get confused as they are talking about 10G only. This "overhead" is a general issue for all transfers. It's only much more present with 10G.

Link to comment
7 hours ago, mgutt said:

You don't need to backup "docker.img":

https://forums.unraid.net/topic/58690-does-it-make-sense-to-backup-the-docker-image/

And what about the database with your "watched status", settings etc? PS maybe you like to backup the appdata folder with this script as its incremental:

https://forums.unraid.net/topic/97958-incremental-backup-through-rsync/?tab=comments#comment-903782

 

Once per month (on the first of course) would be sufficient.

 

 

Of course. Every write which uses direct disk access (/mnt/cache) will be much faster than with shared disk access (/mnt/user).

 

PS You could enable disk share, disable all shared access of your disks except of your cache disk, mount a subfolder of the cache disk as a local network drive on your client and by that you even use direct disk access while transfering files to your server.

 

I described this here in my Guide in #3:

https://forums.unraid.net/topic/97165-smb-performance-tuning/

 

And here are some user experiences regarding this, too:

https://forums.unraid.net/topic/92282-solved-workaround-how-to-write-to-nvme-pcie-cache-at-full-1-gbs-with-10-gbe-nic/

 

Don't get confused as they are talking about 10G only. This "overhead" is a general issue for all transfers. It's only much more present with 10G.

Thanks for all this information I have to read more to grasp it all

Not sure how you write directly to cache from a windows PC... (For Nextcloud I just added the path as cache to the docker temp folder) 

 

If I go to my \\IP\cache\temp I can't connect to it - do you have to modify the SMB config in order to make your cache drive directly accessible?

 

 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.