Drilling sounds fine, as long as the difference in size isn't TOO large!
Sleeve just needs enough material to take the crush load of the bolt, so it tightens up solidly. As long as you aren't drilling it out to "paper thin" it should be OK.
How big a difference are we talking here? Is it a case of "almost" fits?