It's already removed from the pool, I didn't noticed it was still assigned, it should be OK to just unassign it, but to play it safer do this:
Stop the array, if Docker/VM services are using the cache pool disable them, unassign all cache devices, start array to make Unraid "forget" current cache config, stop array, reassign the smaller cache device only (there can't be an "All existing data on this device will be OVERWRITTEN when array is Started" warning), re-enable Docker/VMs if needed, start array.