Just adding this for anyone else who might have this problem in the future. I had upgraded my unraid server's GPU from a Quadro P400 to an RTX 4000 ada sff. It was showing up in the system devices but the nvidia driver was giving the "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver" error. I followed all of the troubleshooting in the OP of this thread and search several other places. I found on Nvidia's forums that you need to have Above 4G decoding enabled on your bios, and in my case I also had to disable CSM in my bios. I did also enable resizable bar, but I am unsure if that was required (it should be turned on, though). Hope that this can save someone some time in the future.