Even if you AREN'T playing on a potato, the current win states are such that on large and huge galaxies the difference between rolling up the last 10% of the habitable planets to win could be anywhere from 50 to 100 years compared to medium maps. Longer if you're unlucky enough to run into multiple pacifists who built dozens of habs. The Collosi don't really help either, since it takes them a few weeks to charge up and fire before destroying a single planet.
In my experience, as bad as late game performance is, it's easily overshadowed by the domination/alliance win requirements. It's even worse if the late-game crises don't snowball enough to wipe out a few people. The hyperlane-only fix has virtually eliminated late game lag for me, but closing up the endgame on larger maps is still tedious.