Another update and hopefully the solution for my issue:
As mentioned above, the issue reproducedable occured, when connecting with the Hyper Backup Client via rsync to Unraid.
I guess because of failing reverse lookups the avahi-daemon used all the ram and most of the CPU:
Feb 10 00:55:03 Dockerbox rsync[2997]: connect from 192.168.10.2 (192.168.10.2)
Feb 10 00:55:03 Dockerbox rsyncd[2997]: name lookup failed for 192.168.10.2: Name or service not known
Feb 10 00:55:03 Dockerbox rsyncd[2997]: connect from UNKNOWN (192.168.10.2)
Also Hyper Backup wasn't able to connect during that time. Eventually the system was out of memory killed a process (not always the same) and Hyper Backup connected.
Solution:
I added the line
reverse lookup = no
to my
/boot/custom/etc/rsyncd.conf
This results in no more failed name lookups, Hyper Backup connects instantly and Ram usage stays as it should.
I hope this was the main issue. Maybe this will help someone in the future.