Yea, I think that's more the exception. Even Thanos held that one, and the Time Stone was held by Strange, Hulk an crew. In my head I was thinking more on the lines of all of them once, with the obvious one being the Power Stone that we know goes kablooey.