Yeah early on, you can buy every gem you don't take as a reward, as well as some other gems you're not even offered in quest rewards for more "off class" builds, like say attack skills for Witch, from the various vendors in each act. Once you reach the library in act3, you can buy every gem...