I took the switch out and cleaned the connections. Still the fuse blew after doing it.
What I have noticed is that when I put on the fog lights, it seems to draw power away from the headlight on the drivers side.
As far as I can tell, they are on seperate circuits and the only place that they meet is in the light switch.
So I'm fast coming to the idea that a new switch is the answer.
I have ordered one with fancy led's from ebay.