yeah, only the power stone fucks people up.
also, being able to hold it, is just a measure of being bad ass. powerful people can. hulk, thor, captain marvel, kree etc can just fine. hell, Gamora probably could.
Regular humans can't. Peter not dieing, was indication, he was not a regular...