403 for google bot

Resolved gauravshukla
(@gauravshukla)

13 years ago

First about 403 errors with googlebot access

>>>>>>>>>>> 403 GET or Other Request Error Logged – May 20, 2013 – 2:18 am <<<<<<<<<<<
REMOTE_ADDR: 66.249.73.56
Host Name: crawl-66-249-73-56.googlebot.com
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /2011/02/samsung-galaxy-s-ii-on-pre-order-for-528-in-uk/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+androidosfeeds+(AndroidOS.in)
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Why are these pages getting 403 errors, these are just links with feedburner parameters. These are many of them, filling my errors log file within hours.. When I accessed the page in a browser, Bulletproof’s 403 error page is coming, as far as I know, normal post page should open.

Secondly, there are also 403 errors for site uptime tracking sites, which I use for monitoring where my blog is working fine for not..

>>>>>>>>>>> 403 GET or Other Request Error Logged – May 20, 2013 – 2:18 am <<<<<<<<<<<
REMOTE_ADDR: 74.86.158.107
Host Name: 74.86.158.107-static.reverse.softlayer.com
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

http://ww.wp.xz.cn/extend/plugins/bulletproof-security/

Viewing 8 replies - 1 through 8 (of 8 total)

Plugin Author

(@aitpro)

I believe this has do with with feedburner using these coding characters in URL’s “& amp ;” to render the & symbol instead of using the & symbol itself (which should be being used instead for good/safe coding practices) in Query Strings, which triggers this security filter below. Edit your root .htaccess file and…

RewriteCond %{QUERY_STRING} (;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]

…remove/delete the semi-colon ; and the pipe operator | after the semi-colon as shown below.

RewriteCond %{QUERY_STRING} (<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]

For the uptimerobot issue you have several options. See this forum post.
http://forum.ait-pro.com/forums/topic/split-uptimerobot-whitelist-uptimerobot-bot/

Thread Starter

gauravshukla

(@gauravshukla)

13 years ago

Even after removing the ; from the security filter, I am still getting the following errors:

>>>>>>>>>>> 403 GET or Other Request Error Logged - May 20, 2013 - 8:35 am <<<<<<<<<<<
REMOTE_ADDR: 66.249.73.56
Host Name: crawl-66-249-73-56.googlebot.com
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /2013/05/google-play-game-services-official-aims-to-bring-social-cloud-features-to-games/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+androidosfeeds+(AndroidOS.in)
QUERY_STRING:
HTTP_USER_AGENT: Mediapartners-Google

>>>>>>>>>>> 403 GET or Other Request Error Logged - May 20, 2013 - 8:37 am <<<<<<<<<<<
REMOTE_ADDR: 66.249.73.56
Host Name: crawl-66-249-73-56.googlebot.com
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /2013/03/htc-one-samsung-galaxy-s-4-to-be-sold-for-99-on-t-mobile/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+androidosfeeds+(AndroidOS.in)
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Plugin Author

AITpro

(@aitpro)

13 years ago

oops I did not catch the round bracket coding characters/parenthesis when I first looked at the Query string. The solution is here…

http://forum.ait-pro.com/forums/topic/403-from-feedburner-feed-in-feedly-imported-from-google-reader/

Thread Starter

gauravshukla

(@gauravshukla)

13 years ago

Thanks.. that worked, but I have identified more errors, which I think should not be there..

>>>>>>>>>>> 403 GET or Other Request Error Logged - May 20, 2013 - 12:16 pm <<<<<<<<<<<
REMOTE_ADDR: 59.161.254.21
Host Name: 59.161.254.21.static-mumbai.vsnl.net.in
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER: http://www.google.com/url?q=http://androidos.in/android-phones-prices/&sa=U&ei=a8aZUbKaF4K3rAeJ7oDoBA&ved=0CEUQFjAJ&usg=AFQjCNHTV9ybCqyMH4kDN8ZWf8FOwfYsBA
REQUEST_URI: /android-phones-prices/
QUERY_STRING:
HTTP_USER_AGENT: NokiaX2-02/2.0 (11.79) Profile/MIDP-2.1 Configuration/CLDC-1.1 UCWEB/2.0 (Java; U; MIDP-2.0; en-US; nokiax2-02) U2/1.0.0 UCBrowser/9.0.0.260 U2/1.0.0 Mobile

>>>>>>>>>>> 403 GET or Other Request Error Logged - May 20, 2013 - 12:17 pm <<<<<<<<<<<
REMOTE_ADDR: 125.16.95.11
Host Name: dsl-NCR-dynamic-011.95.16.125.airtelbroadband.in
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER: http://www.google.co.in/search?aq=f&hl=en&gl=in&tbm=nws&btnmeta_news_search=1&q=LG+Optimus+%27it%27+%28L-05E%29&oq=LG+Optimus+%27it%27+%28L-05E%29&gs_l=news-cc.3..43j43i53.133646.137208.0.138599.5.3.1.1.0.0.235.454.2-2.2.0...0.0...1ac.1.gS1J5XfWddo
REQUEST_URI: /2013/05/lg-optimus-it-l-05e-with-snapdragon-600-announced-in-japan/
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.0.3705; .NET4.0C; .NET4.0E; InfoPath.2)

Plugin Author

AITpro

(@aitpro)

13 years ago

“Java” is in the User Agent for the first error.

To allow this edit these security filters in your root .htaccess file…

RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]

…and remove java as shown below…

RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|winhttp|clshttp|loader) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]

The second error is caused because single quote coding characters/apostrophes are being used in the URL.

See this forum post for your options.
http://forum.ait-pro.com/forums/topic/feedburner-feed-header-causing-403-error/

Plugin Author

AITpro

(@aitpro)

13 years ago

If you still have additional questions please post them. Resolving.

Thread Starter

gauravshukla

(@gauravshukla)

13 years ago

The 403 related errors are resolved but I have noticed a significant spike in server load after installing BulletProof Security.

Does BulletProof Security add significant load on server resources?

Plugin Author

AITpro

(@aitpro)

13 years ago

Nope, as long as there are no conflicting rewriterules/redirects then BPS would not add any additional load on server resources.

Conflicting rewriterules will cause looping/infinite rewrite or redirect loops. To see if this problem is occurring check your Server log and see if you see this error – [error] [client 127.0.0.1] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use ‘LimitInternalRecursion’ to increase the limit if necessary. Use ‘LogLevel debug’ to get a backtrace

See this Forum Topic for possible causes/solutions
http://forum.ait-pro.com/forums/topic/request-exceeded-the-limit-of-10-internal-redirects/

Viewing 8 replies - 1 through 8 (of 8 total)

The topic ‘403 for google bot’ is closed to new replies.

Tags