Yeah, this. Talk to them. Just say something like, "I'm thinking of replacing the fence. You don't have to do anything, I'll pay for it, and we'll both end up with a new fence."
I'd be willing to bet they'll be fine with it. If you don't have the conversation, they won't be fine with anything.