Don't have one yet, just been reading up on it at work and pob updated just today. You're probably farther along in theorycrafting and playtesting than I am.
But, yes, you are correct, that is a general downside of Ivory Tower. Arcane Cloak does help mitigate that.