Actually most invaders soul vessel into optimized builds and farm people in the water zones, in my experience. I walk around full human most of the time, so despite no sin on my game I get invaded a shit ton. Usually, it comes down to if you have PvP stuff on you and can slot it before they reach you (i.e. not in the middle of a mass trash fight). They all front load burst DPS, so if you can endure the initial barrage, you have a good shot at repelling an invasion. A lot of the better ones are doing dual fist weapons, though, which will stunlock you even if you have 80 poise. Those guys you can only Dark Vortex and hope they fuck up their timing.
As far as doing a caster build, my experiences as a full spectrum caster are that faith builds, with hexes, are the better way to go PvE wise. Sorcery is superior for PvP, because it has the best burst damage of anything in the game. Pyro is there for utility for people who don't want to dump points into Int and Faith. Dark Vortex is the single most useful spell for a number of situations, so you want at least 12 INT. If you want to do all the caster stuff, then pump it to 40 and swing one of the BK weapons or MLGS. Attunement should be at least 20, but 30 is good. Faith you can stop at 55 and just spice down the Sunlight Spear if you want it (it is pretty cool). The extra damage you gain for faith past 50 is minimal. You can spend most of the game with a mix of lightning spells and Dark Vortex slotted, while rocking Sunbro Ring, and never really need anything else, though the poison cloud hex is good for gimping certain areas.
