Well he did mention having taken a nap in that very office so perhaps mumbo jumbo sympathetic magic / known location / left behind a focus / etc. Yeah it is a major major stretch, but that is always a problem with comic book villainy : how to make the bad guy badass without making the defeat...