So I think I've pretty much found the sweet spot for growth/food production, and now I'm getting fairly bored.
Seems like if you put down a Forrestry building, and then within that circle range where they replant trees constantly, throw a hunter lodge, gatherer, and herbalist, and it can pretty much self-sustain about ~30 people(adults,students,children). Add 1 fishery to that combo and now you're supporting about ~40 people indefinitely without even having to farm.
So I pretty much plunk down those 4 buildings + a fishery and I know I can build houses until my population gains by about 40. Once I hit that number, just do it again. I don't see any problems so far. With my town of like 90 adults, 25 students, and 30 kids it's still working flawlessly, my food slowly grows, I have about 10-15,000 food in storage at any given time.
I guess I just keep doing that until I run out of space and thats it :/