Hello and welcome.
8GB is probably enough, but I'd almost always recommend putting 16GB into a new machine if you can. You could almost certainly get away with a 550w power supply in a mini-ITX build, I'd run a PS calculator. Regarding CPU choices, the 4600 is a Kaby Lake 1151 CPU and the 8100 is a Coffee Lake 1151 CPU - so keep in mind that even though they use the same socket they'll need different motherboards.
Regarding transcoding, if you really plan to support 3 transcoded streams the Pentium G4600 (5133 Passmarks) wouldn't be enough unless you also plan to use hardware transcoding. The rule of thumb is 2,000 Passmarks per 1080p stream, and don't forget to leave some capacity for unRAID itself. The Core i3-8100 is a better choice (8094 Passmarks) if you plan to support several transcoded streams. This is an inexact science, though - the source material, the capabilities of your players, and whether you are using hardware transcoding all affect transcoding.