That would, or could, sound like a vacuum leak which could easily be from the charcoal canister ... which happens to be where the fill tube vent line runs to.
I can't recall the name of the part off hand but, I replace some device on my canister due to an EVAP leak ... and I'd taken the part apart to try and repair it and broke it. But I'd found my leak prior to damaging the part.
There is a rubber boot on the part and that boot was a little loose and air was seeping past it. A little RTV and it was fixed.
It's the only thing with an electrical connector on the charcoal canister.
Now if that's your exact problem, I can't say. But it might be worth checking out if nothing else.