Talk to the person next door and tell them you're willing to do the work, make it a good, healthy tree that provides delicious fruit, and offer to let them pick half. If they aren't interested, they'll more than likely tell you to go ahead and deal with it and take the fruit. Win!