SOLVED:
Here's the solution:
Went into Tools -> New Config -> Selected "Preserve current assignments" to "All" -> Checked the box that says "I want to do this" and clicked Apply.
When I went back to the Main tab, it now allowed me to add the 4th disk like I was adding it for the first time.
Basically looks like what it does is that if the disk has never been used by Unraid before then it will be partitioned to UnRAID standards. If it is previously used by Unraid then its contents are left intact. Parity is also set to be invalid and thus need rebuilding.
