When I take the rear drums off to check the brakes, I always tighten them down to the highest setting my torque wrench will read, which is 150 ft-lbs. Yes, the manual does say 180-200 I think. However, I have yet to have any issues on either my 2000 or my wife's 2008 and she has over 90 on her car already.
Just hand tightening them is a no-go. The spec may be more, but 150 feels PLENTY tight enough. Going with the spec won't hurt either.
Just my .02.