Nothing jumps out why it's going so slow, you'll need to do some testing to find the problem, but it's harder to do that with a disabled disk.
You can cancel a rebuild at any time, but it can't be resume, i.e., it will always start from the beginning.
This is for virtualization, it shouldn't affect rebuild speed, you can check with lspci is the HBA is linking @ x8, but with only 4 disks connected it shouldn't make so much difference even if it wasn't.
lspci -vv -s 41:00.0
Look for LnkSta