Google is smart enough to make this distinction between categories and sub-directories, you don’t need to worry about it.
Remove /category/ from your WP won’t have any negative impact, you can use this plugin or any other like that without having any problem.
To see what might be causing a negative impact on your site, you would have to do a large number of tests, even new generated content on your site after you install the plugin can cause some good or bad effect. Moreover, it may be just a normal floating position in SERPs.
It has a more positive effect as google searches keywords inside the content of the topic and also matches that to the title and permalink..
The content has to be relevant to the title, I’m pretty sure inside your pages you don’t have category written all over your document, so its actually an improvement!
