Nice to see another person from Indiana on here.
I've taken my handles off a few times to paint, and replace the handles. To me, they were a pain in the butt, but not really that hard to do. I think my patience was wearing thin by the time i put them on.
I have power windows and locks on mine, and there wasn't a ton of room within the door cavity to get my hand up in there like i wanted to.
I ended up having someone help me with it for like 2 minutes to get everything lined up and back together.
You can remove them and paint with trim/bumper paint to return to the factory black look. You could also lightly sand them, primer and paint them to match the factory color. I've also seen some "paint pens" that are the trim black in color which would allow you to do the part on the car without removing it. Just be sure to mask off the area around it.