Thanks for the replies, Scott.
I managed to stumble into a solution and I hope I am remembering this correctly since I didn’t record it: In my plugins menu, found PayPal Donation and there was an “edit” option. Not all plugins seem to have this, but the PayPal one did. Obviously if you want a plugin to work, don’t touch this. I went into edit, highlighted all the code and deleted it. The plugin disappeared from the plugin list and my site. It did leave behind a text tag [wp-48id something, something}. I found the footer this was located in and deleted. PayPal button gone 🙂
@scottpaterson
Same problem. I need to get this plugin off my site (nothing wrong with it – I’m just changing to a different donation solution) When I hit deactivate, I get the error message from WP.