Search engine issue

  • Resolved Dmitry Bychenko

    (@wpbluefox)


    10 months, 1 week ago

    Hello,

    I have received an email from search engine / webmaster tool that some page (home page) was added to search 3 more times with different queries like this: /?blackhole=gsdsyd7w3

    https://ibb.co/MxKpD8cg

    I think 2 rules that should be added to robots.txt file are dealing with this issue.

    That rules are added: https://ibb.co/MxYyC0L2

    Is this creating issue with particular page being indexed few times and competing with itself in search results?

    Or this issue doesn’t really matter much?

    If this is an issue, how to solve it?

Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Author Jeff Starr

    (@specialk)

    10 months, 1 week ago

    Glad to help.

    First thing is to make 100% sure that your site is not using any sort of cache plugin. Blackhole is not compatible with any sort of caching. Check the plugin homepage, documentation, readme.txt, forum sticky post, plugin settings, or help tab for more information about this.

    That said. If you are 100% sure that your site is not using any sort of caching, then it means the search engine is disobeying your site’s robots.txt rules (assuming the rules have been added correctly). So it is up to you, how to deal with disobedient search bots.

    The most important thing, is to make sure that the site is not running any sort of cache plugin.

    I hope this helps, let me know if I can provide any further information.

    Plugin Author Jeff Starr

    (@specialk)

    8 months, 1 week ago

    Hi Dmitry, just wanted to follow up with this. It’s been a while with no reply, so I hope the issue is resolved? Or if there is anything I can do to help, please let me know. Thank you.

    Plugin Author Jeff Starr

    (@specialk)

    6 months, 3 weeks ago

    Hey Dmitry, I hope you got this sorted. It’s been a while with no reply so gonna go ahead and mark this thread as resolved to help keep the forum organized. Feel free to post again with any further questions or feedback, Thank you.

    Thread Starter Dmitry Bychenko

    (@wpbluefox)

    6 months, 3 weeks ago

    Hey Jeff, sorry for not replying.

    Yes, it is written everywhere “no cache plugins”, but records in robots.txt..

    Seems like Yandex crawler dont care much about instructions in robots.txt.
    And seems like other search engines are fine with skipping page indexing according to robots.txt

    Yandex suggests solution for this issue: add
    Clean-param: blackhole
    parametr to robots.txt -Yandex section:

    User-agent: *
    …
    Disallow: /*blackhole
    Disallow: /?blackhole

    User-agent: Yandex
    Clean-param: customize_changeset_uuid
    Clean-param: blackhole

    What do you think about this?
    Still have to choose between not use cache plugin or not use BBB?

    Thank you for your plugins, especially for BBQ Firewall.

    Plugin Author Jeff Starr

    (@specialk)

    6 months, 3 weeks ago

    Yes that is correct: Do NOT use cache plugin together with Blackhole. The Yandex robots rules apply only to Yandex, so there lots of other search engines and bots to consider. Also, even more powerful/effective than “asking” for Yandex (or any other bot), is to just do it programmatically. I wrote a guide and quick plugin that can help clean things up for anyone who runs cache together with Blackhole. I hope it helps, let me know if I can provide any further information.

    Thread Starter Dmitry Bychenko

    (@wpbluefox)

    6 months, 3 weeks ago

    Thank you, issue solved.

Viewing 6 replies - 1 through 6 (of 6 total)

The topic ‘Search engine issue’ is closed to new replies.