Does the car 'lurch' or jerk when you hear the noise?
IF not then:
My first thought is your bottom rear engine mount is gone or failed. (i am not certain those words are the exact right combination of words offically used, but it IS the engine mount, and it is at the bottom)
So the engine is 'loose' enough to clunk around.
(If the mount is broken or loose, the engine WILL move around no matter how easy or hard you shift/clutch.
((If it IS the mount, you should be able to hear the issue if in say first gear (stay in gear), running at say 15mph then let off gas for two seconds, then hit gas hard. CLUNK should follow. ))
So IMO your bottom/rear engine mount is loose or broken.
This is a known issue (so i have read, anyway)
It will not harm your car at all if you get it fixed.
So do not worry about if it caused some 'other' permanent damage.
But it should be looked at.