Never been to the pier, but Venice beach is worth going to at least once, just for the freak show. After that though, not really worth going to again. It helps to go on a weekend during the summer. Not sure how it'd be in the "off season."
As far as driving, I haven't spent that much time in...