You'll typically see smaller file sizes and some (minimal) quality improvements when encoding with Handbrake's software-based encoder, but it can be a resource hog. Depending on your use case, even a GTX970 can do a good job transcoding files. I'm using my old desktop machine with an i5-4690K and a GeForce GTX1070 as a transcoding setup until I finish my server rebuild -- 3 x 24GB 1080p h.264 mkvs can be transcoded to between 2-5GB h.265 files in about 15 mins. (depending on the film content). A 4K file is about 6GB and takes about 30 mins. on my system. I don't have a dedicated home theater, so a 1080p file at >2900Kbps looks perfectly fine on my 55" OLED (even from less than 7 ft. away). Any files that are blurry or pixelated can be re-ripped at a later date. I have thousands of DVDs, hundreds of Blu-Rays, and I've spent more than I care to remember on hard drives -- I'm to the "it looks good enough, so just play the movie" phase of my life where convenience means more than 100% fidelity.
As a suggestion, I would look into using Tdarr instead of Handbrake. It can handle bulk transcodes, or just setup a watch folder that will be automatically monitored. Plus, you can setup networked nodes to process files between multiple computers. Video decoding will still use your CPU, but the encoding will take place on the GPU(s) you specify. You'll want at least a GTX970 for h.265 encoding, but a GTX1650 Super seems like the sweet spot. Check the NVENC GPU support matrix here: https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new. SpaceInvader One has a good series of tutorials on Tdarr starting here:
If you're a perfectionist, you may just want to transcode only 4K files to save drive space, but I process high bitrate files from 720p and higher.
Hope this helps.
