Floating grounds are bad, mkay?
Plug the UPS into a switched outlet or power strip, then turn off the power to it without breaking the ground connection.
Chances are high you will get away with just yanking the cord, but the possibility is definitely there to create a very large voltage differential across cords still plugged in to the UPS, for example the USB lead. This could blow out the USB port.
Many years ago I personally fried a surge protector plugged into a UPS output when I yanked the supply out of the wall to demonstrate how good my UPS was. Rather embarrassing, but I was really glad the only damage was the surge protector, not the connected equipment.