Thread Starter
moutai
(@moutai)
Because my Google Search Console is sending a huge number of requests for each filter URL, about 20–30,000 per day, for example for URLs like:
GET /shop/?product-categories=acne,hair-damage,hair-dry,hair-kits,hair-masks,hair-serums,masky
GET /shop/?product-categories=face-creams,face-kits,hair-creams,hair-fatness,hair-foam,lips,spf
GET /shop/?product-categories=acne,hair-creams,hair-fatness,hair-foam,hair-loss,hair-masks,sale And since this puts a very heavy load on the site, I could really use some help as soon as possible.
-
This reply was modified 6 months, 4 weeks ago by
moutai.
Hello again Moutai,
Thank you for further elaborating on your issue. Your problem is discussed in the Managing crawling of faceted navigation URLs article of Google Search documentation. It describes multiple ways of addressing the overcrawling and excessive indexing issues, and while some of them are outside of the scope of a filtering plugin, you can currently use 2 options of the annasta Filters > SEO Settings tab to implement the solutions recommended by the above article:
- Enable the auto-insertion of canonical annotations with the help of the Add canonical annotations option.
- Enable the Add rel=”nofollow” attribute option in case you use presets with URL Filtering style. I would like to clear up that the “nofollow” attribute of the Add rel=”nofollow” attribute option is added to the links (<a> tags) generated by the filtering presets with the URL Filtering style, and not as page meta!
As for the “noindex” meta tag auto-insertion into the <head> section of filterable pages (I presume that’s what your initial question was about, wasn’t it?), we have decided against adding this as plugin option because it’s currently not the recommended way of dealing with multiple near-identical pages indexing issue. Besides, there always exists a relatively easy way to completely disallow the crawling of filtered URLs with the help of the robots.txt file.
Thread Starter
moutai
(@moutai)
May I ask you to send me the robots.txt rules that will help remove your plugin’s filtering from indexing? And also, as I understand it, in this case it’s better to enable Ajax filtering rather than URL-based filtering, right?
Hello again Moutai,
Unfortunately, there is no universal robots.txt file that I could share with all our users! A robots.txt file disallowing the Google crawling of your site’s faceted navigation will list the filtering parameters that are specific to the pages of your site, so if you have decided to go with this SEO strategy, you will need to implement it yourself.
As for the URL filtering style versus AJAX, if you wish to limit the exposure of your filtering parameters to search engine crawlers, then yes, AJAX filtering style should be a better fit!
Dear Moutai,
I will go ahead and close this inactive thread, please open a new topic if you ever need to ask additional questions about the work and functions of our filters!
