Thanks for raising this and for the detailed explanation. Earlier in the plugin’s development, I debated whether additional rules should be applied when “Discourage search engines” is enabled. My hesitation was that if a crawler ignores a general Disallow, it’s likely to ignore more specific rules as well.
That said, this is a strong real-world example of why modifying robots.txt in this scenario still makes sense, even if it won’t stop every bot. Preventing unnecessary crawling by well-behaved bots is still a meaningful improvement, especially for staging environments.
I’ll work on adding this behavior and will try to push an update as soon as I can.
