Disallow: /wp-content/wp-cache can cause google crawl errors

  • James Revillini

    (@jrevillini)


    8 years ago

    Note that by default, you add this to robots.txt
    Disallow: /wp-content/wp-cache

    This can cause Google to view the site as NOT mobile friendly and can cause ranking issues as a result.

    I’d highly recommend removing that from your option to “create right robots.txt” as I cannot see it having any positive impact on SEO.

Viewing 2 replies - 1 through 2 (of 2 total)
  • webcraftic

    (@webcraftic)

    8 years ago

    Hi,
    Thank you for reporting this issue.

    Yes indeed, now this configuration file is already outdated. I need to better study the current configuration of robots.txt file.

    If you have interesting materials on how best to configure robots.txt in 2018, I’ll be glad to get acquainted with these materials.

    Best regards, Alex

    Thread Starter James Revillini

    (@jrevillini)

    8 years ago

    I did a little research and found that blocking wp-content/plugins and other things which may serve css and js is no longer desirable and the best option is probably the same as what yoast uses: https://yoast.com/robots.txt

    Here is their article: https://yoast.com/wordpress-robots-txt-example/

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Disallow: /wp-content/wp-cache can cause google crawl errors’ is closed to new replies.