I never used unRAID, but I know from others that my container works with:
- Bridge mode network (This is the simplest option. It doesn't require any advanced configuration.) The only downside is that you cannot use any ports that are already in use by unRAID. So if you want to enable the Samba Server in DSM it doesnt work because unRAID is already using the Samba ports.
- Custom br0 network in MACVLAN mode (This is the more difficult option. It requires DHCP=Y and the cgroup rules, etc.) The advantage is that DSM will get its own IP, so you will never have port conflicts.
As I said before, I think your br0 network is not in MACVLAN mode but in IPVLAN mode, which can never work with DHCP. From a quick Google search I see that you can change it in the Docker settings. So change the two settings in the screenshot below to macvlan and enabled.