I'm having a similar issue with QuickSync locking the system up hard. This is on a Supermicro X11SSH-TF with E3-1275v6, 16GB of memory. I've had it happen with Plex, Handbrake, and, lately, I've been using Tdarr, all of which can trigger the issue seemingly at random. I thought it might be a thermal issue, but after putting a giant CPU cooler on, it hasn't seemed to resolve the issue. Doesn't look like anything gets written to syslog, so I've had to do a hard boot via IPMI.
Would love to be resolve this so I can use transcoding consistently. It can work for relatively long periods, or fail a couple of files in. Is 16GB not enough memory maybe?