I figured out the issue. Filezilla kept uploading it as ASCII filetype instead of binary. it kept changing the file size ever so slightly and making the keyfile invalid. Got it working now!
So, if uploading via Filezilla, make sure to upload as binary.