It's worth it to me, but only if it's smart shopping.
I've never once paid full price for anything. I'm an opportunist. When two of my rims got trashed from a pothole, my insurance sent me a check to replace them and I bought four custom rims with the money (2 stock rims = price of four custom rims). When I needed to replace my stock suspension cuz it wore out, I bought the SVT suspension kit for $50 more than the cost of off-the-shelf stock replacements.
Everything on my car is either used, or I bought it for an insane deal. Most of what I've bought has been because something needed replacing over the 110k miles i've owned my car (normal maintenance, accidents, etc.). I've gotten a lot from off the BST forum.
Other than nice tires and wheels, an SVT suspension, and an SVT intake, I don't have any performance mods either... and I'm not in any hurry to buy them. I don't race, and my manual zetec is plenty fast enough for me... although I do keep my eye out and will probably end up getting a full header/catback, but again... likely it will be used and dirt cheap.