On my commute and just driving around town, I leave the ESC on, however any time I take the ST out for a spirited drive or want to do a hard launch, I put it into Sport mode, no matter what the road conditions are. Sport mode greatly reduces wheel hop (by allowing the wheels to spin some) and lets you have a little more fun drifting in the twisties before the nannies kick in.
I've only completely disabled it twice to get a feel for it and found little, if any, difference between fully disabled and Sport mode. I won't hesitate to completely disable it when tracking or autocrossing, however I do like the extra safety cushion "Sport" mode gives you on the street.
13 ST3, Performance Blue
96 F-250 Powerstroke, 5-spd, 4x4