I’ve noticed you can use the shortcut key as a toggle - so if you hit M for the map, when you hit it again it closes back to the game - same with J (quest journal). I think this works as well even if you entered the menu from the ESC interface, but I could be wrong.