Pretty much. About half-way through, or maybe two thirds, it might start to click and become very much worth it. That Gardens was initially written as a screenplay well before any of the others might have something to do with its crudeness.
IMO, of the first 5 books, Memories is the only one...