So after changing the idler pulley, and the tensioner pulley, the problem is solved. They weren't horrible, but they clearly weren't operating smoothing like pulleys should. This had solved the rough idle issue for me when the A/C is on. My theory is that the car didn't have the power under idle in its old age to handle both the A/C and the crappy pulleys without getting bogged down. Also in this process I gave it a tune up (plug wires and plugs). The car runs significantly better and has no idle issue.
Lesson learned: Dont over think it and rule out the simple stuff. Hopefully this thread helps others. Thanks to everyone in the community that chimed in and offered their advice and experience.