Googlebot cannot access CSS and JS files

jimmerd
(@jimmerd)

10 years, 10 months ago

I just got an email from Google saying:

Googlebot cannot access CSS and JS files

on my site.

Seems the robots.txt file is disallowing wp-admin/ and wp-includes/
which it has always done.

BUT there’s CSS and/or JS in there that Google wants to be able to see.

How to fix? Allow all?

Has the Googlebot lost its mind?

THANKS!

Viewing 15 replies - 1 through 15 (of 94 total)

1 2 3 … 5 6 7 →

ankurj
(@ankurj)

10 years, 10 months ago

Yes, Same here!
I also received warning about an hour back.

For some reason, Google bot wants to see some CSS and JS files. Not sure if it is a genuine warning we should care about?

allegroconsultants
(@allegroconsultants)

10 years, 10 months ago

you need to edit your robots.txt file. this seems to be something new for google to care about.
here is an example of the robots.txt you will need, just edit the one you have an put the allow lines in.

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/cache/
Allow: /wp-content/themes/
Allow: /wp-content/plugins/
Allow: /wp-content/uploads/
Allow: /wp-includes/css/
Allow: /wp-includes/js/
Allow: /wp-includes/images/

ankurj
(@ankurj)

10 years, 10 months ago

As far as I know, robots.txt doesn’t understand Allow tag.

Anything not under Disallow tag is automatically allowed.

g0rd
(@g0rd)

10 years, 10 months ago

That’s not accurate ankurj. stackoverflow has good information about using Disallow and Allow. As mentioned previously, allow the files that Google wants to see for its smartphone render.

Thread Starter jimmerd
(@jimmerd)

10 years, 10 months ago

Well, that mostly worked but it’s still returning an error for one PHP file/script, apparently due to the WordFence plugin:

http://MYSITE.us/site2014/wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=1717E62F164CDE77F5220B2D3B3B7AA4&r=0.03589733876287937

which is blocked because it’s in wp-admin/

ankurj
(@ankurj)

10 years, 10 months ago

@g0rd: Thanks for clarifying that. I stand corrected.

Did some more digging and found this from Google.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

soamz
(@soamz)

10 years, 10 months ago

So whats the solution now ?

Do as allegro consultants said ?

soamz
(@soamz)

10 years, 10 months ago

So add that to the robots.txt file as allegro consultants said ?

Thread Starter jimmerd
(@jimmerd)

10 years, 10 months ago

Yoast seems to think it’s best to simply allow all.

https://yoast.com/wordpress-robots-txt-example/

gavinr
(@gavinr)

10 years, 10 months ago

I’m getting these emails from Google today too, but seems to be only on my sites that are using Cloudflare – could that be related or is it definitely just a robots.txt issue?
gbox12
(@gbox12)

10 years, 10 months ago
I got this
```
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress
```
and I still get the message
Jim Birch
(@jimbirch)

10 years, 10 months ago
I would recommend:
```
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/cache/

User-agent: Googlebot
Disallow: /wp-admin/
```
Here is Google’s original post about the change:
http://googlewebmastercentral.blogspot.com/2014/10/updating-our-technical-webmaster.html
Matt Magi
(@matt94gt)

10 years, 10 months ago

I just got this email as well for about 20% of my websites.

Jimmbirch why would you disallow the cache?

Many of the sites that I got this notification from have this in their robots file:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /xmlrpc.php

lespo
(@lespo)

10 years, 10 months ago

It can’t be coincidence we’re all getting these emails from Google today – I wonder if it’s related to a recent WP update or plugin update. My robots.txt file is:
User-agent: *
Disallow: /wp-admin/
and I’ve not had problems until today.

Matt Magi
(@matt94gt)

10 years, 10 months ago

Apparently google just started sending out the JS/CSS warnings today (even though they have been using this method for some time) which would explain why we all are just finding out:
http://www.thesempost.com/new-google-search-console-warning-for-googlebot-cannot-access-css-and-js-files/

More info above.

I guess its OK to allow all now. As google is smart enough apparently not to index wp core files. They started targeting css/js a while back on their last update, but are now just sending out warnings. Here are some articles:
https://yoast.com/google-panda-robots-css-js/
https://yoast.com/wordpress-robots-txt-example/

********************************
******THE SOLUTION******
********************************

We should just keep all our robots.txt files blank from now on. There is no longer a need to block core wp directories.

I just updated my robots.txt. I removed my disallow of wp-includes, wp-admin etc. and now google is rendering my websites properly.

To test it, login to webmaster tools, and simply fetch and render as google. If your site appears in the screenshot to not be loading CSS then your robots.txt is blocking it somehow.