Facebook User Agent Being Blocked When Query String Present
-
I’m having an issue with the plugin, I can’t get any pages scraped by Facebook if a query string is present. I’m using the Sharing Debugger tool (https://developers.facebook.com/tools/debug/) to see what’s going wrong, and the scrape is timing out. Our site’s TTFB when a page is not cached is something preposterous like 13 seconds, but pages load fine when a caching plugin is installed.
The web page in question is https://www.cedrsolutions.com/healthcare/childcare-guidance/ I know by default WP Fastest Cache excludes the Facebook user agent from caching, but I’ve removed everything in the “# Start WPFC Exclude” section of the htaccess file to fix that. I have also added the “define(‘WPFC_CACHE_QUERYSTRING’, true);” line to the top of my wp-config file, but that shouldn’t matter for UTM variables.
If I scrape just that URL, everything works fine. However, if I try to scrape https://www.cedrsolutions.com/healthcare/childcare-guidance/?utm_source=facebook, the Facebook scraper times out. I’ve checked that url (https://www.cedrsolutions.com/healthcare/childcare-guidance/?utm_source=facebook) on gtmetrix and the speed shows the page is being cached for the gtmetrix crawler without issue. Twitter and LinkedIn’s scrapers also time out when I check them.
It’s almost as if the user agents for social media sites are still being excluded from viewing the cached version of the page, but I can’t figure out why if I’ve already removed them from the “# Start WPFC Exclude” section of the htaccess file.
-
I understand that the caching works for the URL when you just bring it up in a browser window. My problem is specifically that the Facebook crawler is being excluded from being served the cached version of the page. Again, I have removed the exclusion for the Facebook User Agent from the htaccess file, so it should not be being excluded from caching. However, if you check the URL in Facebook’s debugging tool (linked in original post), you’ll see the scraping process times out because the page takes so long to respond (indicating it is not cached)
I’m sorry if I’m being unclear. The screenshot you just sent me shows that the URL is able to be crawled normally by Facebook if there is no query string present. Facebook does not see the cached version of the page if there is a query string present.
If you try the exact same URL with a query string (like https://www.cedrsolutions.com/healthcare/childcare-guidance/?utm_source=facebook) the crawler will time out.
That’s the whole problem, WP Fastest Cache is excluding the Facebook crawler from seeing the cached page when there is a query string in the URL.
The topic ‘Facebook User Agent Being Blocked When Query String Present’ is closed to new replies.
