Regular SMART tests were not really needed in the past with Unraid since a parity check is for all purposes the same as an extended test, cache devices are usually SSDs so also not really a need for SMART tests and SMART test monitoring, now with multiple pools it is some times good the run regular SMART tests (if using spinners) on those, it's possibly something that could be improved in the future, for now I have a script running an extend test for those drives once a month and it emails me the results.
No, you'll need to stop the array, also make sure the removed device is not still part of the pool, if it is and you added it back it will be wiped (there's a warning when assigned).