Do you have a survey plat map that shows where it is located in reference to the true property line?
You can always ask, even if it is yours - if they may be benefitting from it being there, too. Like if they have a dog or young children, or just like more privacy.
You could start with: I just wanted t let you know I'll be taking this old, rickety fence down later the month, so there will be some noise & people working. We'll be as quiet and neat as possible. The workers may need to step on your side to accomplish the work, OK?
At this point they may ask if you aren't putting up a new one.
You: I looked into it, but not sure it's in the budget right now.
Now, they may offer to help with the cost to replace it, rather than remove it. Always best if it is THEIR idea.
Don't be afraid to let the thought sit with them for a couple days while they discuss their privacy and finances with the spouse.
--68.229.xxx.xxx