• Resolved seventyone

    (@seventyone)


    Get this message from Google when indexing products – Update your robots.txt file to allow user-agents “Googlebot” and “Googlebot-Image” to crawl your site

    Have also had other issues with indexing / crawling. Plugin related or hosting?

    Any guidance would help – thanks!

    The page I need help with: [log in to see the link]

Viewing 3 replies - 1 through 3 (of 3 total)
  • Hello, pretty nice website you got there!

    I had a look at your https://triplebogey.com/robots.txt, and it is definitively not what The SEO Framework did. Here is what your robots.txt says:

    
    # Default robots file version:2
    User-agent: *
    Disallow: /calendar/action*
    Disallow: /events/action*
    Allow: /*.css
    Allow: /*.js
    Disallow: /*?
    Crawl-delay: 3
    

    Robots.txt file is used to give search engines and other services that crawl your website rules. I can explain what every line does if you want, but if you want the short and sweet version, just delete the file and TSF will make a new, virtual robots.txt file with magic. That file is in root of your WordPress installation, same location as folder wp-admin is. If it is not there, let me know.

    These 2 lines might be potentially important:

    
    Disallow: /calendar/action*
    Disallow: /events/action*
    

    But since I can’t access your folder structure, nor did I make the site, I have no idea if they are even needed for anything. This looks like a leftover from a previous website, maybe? Do you have a folder “calendar” or “events” in your root directory? If not, they are probably safe to ignore and the default robots.txt generated by TSF should be just fine.

    Let me know how it went, we will go from there.
    Cheers, Pierre.

    Thread Starter seventyone

    (@seventyone)

    thanks Pierre!

    the robots.txt is dropped in by hosting (no calendar / events folders that require any special treatment) – removed and now using TSF virtual – so we’ll give Google time to crawl and see – did test out a few URLS with their inspector and no longer seeing the errors

    appreciate the help / insight!

    / Steve

    My pleasure, please do not hesitate to talk to us with other issues you might encounter. As a shameless plug, consider rating us 5 stars here in WP directory, it helps a bit and makes me hate my job less.

    Cheers Steve, have a good one.

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Robots.txt issue Google’ is closed to new replies.