No, you can't take the panel off with out the door open. well you could but you'll have to break it. There are hex screws that run around the outside of the panel, 6 I believe. Even if you manage to take the two screws from behind the pull handle, one behind the door open handle you still have to put out then up to get it to unsnap but of course that's if you can get the 6 or so hex screws. My guess is the latch isn't releasing. You could try a coat hanger and try to catch on the door lock that connects to the latch but it would take lots of patients. But there is a good chance the latch is broken internal and I wouldn't even know how to approach that. I've taken the latch off to replace it before but mine was just the opposite the door wouldn't close. It's has three screws that hold it in place on the inside of the door frame. I'm gonna have to scratch my head on this one.
Do one thing every day that scares you. -Eleanor Roosevelt