I can't reproduce this on my test machine.
It's a ASRock J3710-ITX with a NVIDIA T400 (driver version 510.60.02) and 8GB of RAM and I've now tried various files (4K HDR, 4K, 1080p, 720p) and tried various transcoding presets (720p 6MB, 720 4MB, 480 2MB) and they all work perfectly fine with NVENC.
Have you yet tried another container from another maintainer if it's the same there too?