I determined it has something to do with the rear brakes. The noise goes away when I brake.
I haven't had time to take the wheels off yet and take a look, hopefully I will be able to fix it when I swap on the good ol' snow tires.
2004 SVT : #55 STF (now racecar)
2003 Buell XB9S