Title: 404 Errors Last modified: May 16, 2018 --- # 404 Errors * Resolved [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/) * Hey there * I am getting a bunch of 404 errors to comet-cache files and I can’t work out the settings that are incorrect! * I notice that most of these are coming from the Googlebot * The Source URLs as reported by my site look something like: * /wp-content/cache/comet-cache/htmlc/public/www-sitedomain-com/f/2/4/b/b/81de11d5393xxxxbb5730409 * I am running Comet Cache Pro v170220. * Thanks for your help. Viewing 15 replies - 1 through 15 (of 17 total) 1 [2](https://wordpress.org/support/topic/404-errors-91/page/2/?output_format=md) [→](https://wordpress.org/support/topic/404-errors-91/page/2/?output_format=md) * [Raam Dev](https://wordpress.org/support/users/raamdev/) * (@raamdev) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10288494) * Hi [@the-moose](https://wordpress.org/support/users/the-moose/), * There’s no reason that Google should be indexing files in the `wp-content/cache/` directory. The files in that directory need to remain web-accessible (especially the HTML Compressor files, which get linked to on various pages of your site by Comet Cache when you have the HTML Compressor enabled), but search engines should not be indexing anything there. * The files in the `wp-content/cache/` directory frequently change, get deleted, overwritten, etc., as the cache is regenerated, which happens regularly and at various times. The filenames and directory names in the cache directory will also change whenever this happens, so it’s not unusual for a link to a file in the cache directory to suddenly return a 404. Whenever a cache file is regenerated and the name changed, Comet Cache automatically starts using the new filename so that a 404 is never returned for a resource that is being used on a page. * It sounds like Google indexed files in the cache directory and then reported that those files were returning a 404 at a future date—this is entirely normal for files in that directory. * I recommend adding a line to your `robots.txt` file that excludes search engines from indexing everything in the `wp-content/cache/` directory. There’s an article here that explains how you can update your `robots.txt` file: [https://kinsta.com/blog/wordpress-robots-txt/](https://kinsta.com/blog/wordpress-robots-txt/) - This reply was modified 8 years ago by [Raam Dev](https://wordpress.org/support/users/raamdev/). * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10291733) * Thanks for your reply [@raamdev](https://wordpress.org/support/users/raamdev/). * Having looked at the Webmaster Tools I don’t see the files listed and they are not appearing in a Google search so I don’t think they have been indexed. * I assume that this means that when the Googlebot is crawling my website they are seeing a reference to the CSS file which is then 404-ing out? * I have run some Google PageSpeed Insights and very occasionally the rendering of the site is all screwed up (as if the CSS is not loading). Annoyingly it’s not easy to reproduce!!! * To me it looks like some references to old files are still on the live site for some reason. * [Raam Dev](https://wordpress.org/support/users/raamdev/) * (@raamdev) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10292480) * That would indicate to me that there’s some other layer of caching going on in addition to Comet Cache. If something other than Comet Cache is caching the page, then it could be that Google is finding the cached (and outdated) version of the page that contains now-invalid links to resources. * Comet Cache automatically clears the cache whenever a link to something changes to ensure that an out-of-date version of the page never gets served. * You might check with your web hosting company to see if they’re doing some other kind of server-side caching, or if you’re using something like Cloudflare or Sucuri, those could also be sources of other caching. * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10292601) * Thanks for the pointer. I have just spoken with my host and there is some server side caching enabled. * I have just disabled the server side caching and will monitor the 404 errors and see what happens! * Thanks for the tip! * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10292940) * Interesting. I have now disabled the server side caching and the issue is still occurring. * Any other ideas? * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10292969) * Just to add, I did block the cache folder through the robots.txt file for Googlebot, however it totally screwed the formatting that Google was seeing (in Search Console) and it wasn’t happy. * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10313469) * [@raamdev](https://wordpress.org/support/users/raamdev/) * Hi there * Any further thoughts or ideas on this please? * Thanks - This reply was modified 8 years ago by [The Moose](https://wordpress.org/support/users/the-moose/). * [Raam Dev](https://wordpress.org/support/users/raamdev/) * (@raamdev) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10317672) * [@the-moose](https://wordpress.org/support/users/the-moose/) The next thing that I would recommend is attempting to [reproduce the issue in a clean WordPress environment](https://cometcache.com/kb-article/testing-comet-cache-in-a-clean-wordpress-installation/) to rule out conflicts with other plugin/theme code. * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10319217) * [@raamdev](https://wordpress.org/support/users/raamdev/) * OK – whilst I understand that concept, how am I supposed to do this as it appears to be the search engine crawlers that are seeing the issue…and I’m not going to want to duplicate the site and see it get crawled? * Honestly, I had hoped that given I’ve paid for the Pro, the support would be more comprehensive! * [Raam Dev](https://wordpress.org/support/users/raamdev/) * (@raamdev) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10319399) * As explained in [this article](https://cometcache.com/kb-article/testing-comet-cache-in-a-clean-wordpress-installation/), we need to rule out a conflict with other plugin/theme code before we can determine if it’s a problem with Comet Cache. This is standard practice for WordPress plugins. * I am unable to reproduce this issue at all, and with 60,000+ other Comet Cache installations out there, I haven’t heard any other reports of this issue, so that further indicates to me that you’re experiencing this issue due to some combination of your hosting environment and/or the set of plugins/theme you’re running. * If you don’t want to make changes to your live site, then setting up another test site where you can attempt to reproduce the problem would be the only other option. * Without a way for me to reproduce the problem, we cannot diagnose what the problem might be. * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10319407) * [@raamdev](https://wordpress.org/support/users/raamdev/) * I understand the article you have linked to. Really, I do. However, how are we supposed to do that when the problem is with search engine crawlers? * [Raam Dev](https://wordpress.org/support/users/raamdev/) * (@raamdev) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10319424) * You would need to set up a test site with Comet Cache running using the same options as you have on the live site (you can export/import options with Comet Cache Pro in **Comet Cache → Plugin Options → Import/Export Options**) and then have the search engine crawl that site. * I know this is very inconvenient, but there really is no other way to be diagnosing the issue. * The other thing you could do is just disable the HTML Compressor feature of Comet Cache (since that’s where the 404 links seem to be reported from) and see if that solves the problem. If it does, you could try using a different HTML compressor, like [Autoptimize](https://wordpress.org/plugins/autoptimize/) and then see if you have the same issue with that plugin. * [Optimizing Matters](https://wordpress.org/support/users/optimizingmatters/) * (@optimizingmatters) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10319687) * Evening guys; Autoptimize can also be the (indirect) cause of 404’s. reason is googlebot uses it’s own cache to get pages and will then (for reasons no-one really knows 😉 ) try to fetch the resources that are linked in the HTML. if in the mean time AO’s cache has been cleared, those will obviously 404. There’s some more info on this topic [in this thread that was coincidentally opened today](https://wordpress.org/support/topic/404-autoptimize-errors-2/). * frank (ao dev) * Thread Starter [The Moose](https://wordpress.org/support/users/the-moose/) * (@the-moose) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10319921) * [@optimizingmatters](https://wordpress.org/support/users/optimizingmatters/) and [@raamdev](https://wordpress.org/support/users/raamdev/) * Thanks for your comments. * I think I am now understanding what is happening – Google store pages in their cache, and then when they try to load external resources that have since been moved (e.g. if Comet Cache Pro has updated the file name), it then just 404s ( hence what I’m seeing in my logs). * What prompted me to investigate this further was when I ran the Google Page Speed Insights test and occasionally it was telling me that my tap targets were too close together and it looked screwy on their rendering. It is now apparent that this was because Google wasn’t finding my CSS files (no idea why it was looking in their cache – I am not one to question our mighty overlords). * I presume everyone else is actually experiencing the same thing – I guess I am more anal than most! * The fact Google was occasionally reporting that my tap targets were not far enough apart (and therefore it was not a mobile friendly design) was my concern – I know they are pushing mobile first so if they are not seeing the site as mobile friendly, then I have a problem. * It seems to me that there are 2 solutions: * 1. Comet Cache Pro doesn’t cache the css/js files (because realistically they don’t change often) and instead use browser caching to handle them * 2. Comet Cache Pro to include the functionality to automatically create a 301 redirect for old file names to the newest location. * All hail our Google overlords. * [Raam Dev](https://wordpress.org/support/users/raamdev/) * (@raamdev) * [8 years ago](https://wordpress.org/support/topic/404-errors-91/#post-10324749) * > 1. Comet Cache Pro doesn’t cache the css/js files (because realistically they > don’t change often) and instead use browser caching to handle them * The whole point of the HTML Compression feature in Comet Cache Pro is to compress and cache the CSS/JS, so if you don’t want that to happen then disabling the HTML Compressor feature (as I mentioned earlier) would be the way to go if that’s what you’d like to do. * > 2. Comet Cache Pro to include the functionality to automatically create a 301 > redirect for old file names to the newest location. * This is not feasible, as it would require keeping track of a very large number of now-expired cache files and doing a lookup whenever a request to a non-existent file comes in, which would slow down the entire site, which is the exact opposite of what Comet Cache is designed for. * You’d also need to be trying to catch _every_ 404 request, which isn’t even possible from within WordPress without special web server configuration, as the web server will serve the 404 response for static files before it even reaches WordPress, which is where Comet Cache lives. So without a special web server configuration that forwards every 404 request to WordPress, Comet Cache wouldn’t even be able know that a 404 response to a now-expired Comet Cache file was made. Viewing 15 replies - 1 through 15 (of 17 total) 1 [2](https://wordpress.org/support/topic/404-errors-91/page/2/?output_format=md) [→](https://wordpress.org/support/topic/404-errors-91/page/2/?output_format=md) The topic ‘404 Errors’ is closed to new replies. * ![](https://ps.w.org/comet-cache/assets/icon-256x256.png?rev=1348602) * [Comet Cache](https://wordpress.org/plugins/comet-cache/) * [Frequently Asked Questions](https://wordpress.org/plugins/comet-cache/#faq) * [Support Threads](https://wordpress.org/support/plugin/comet-cache/) * [Active Topics](https://wordpress.org/support/plugin/comet-cache/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/comet-cache/unresolved/) * [Reviews](https://wordpress.org/support/plugin/comet-cache/reviews/) * 17 replies * 3 participants * Last reply from: [Raam Dev](https://wordpress.org/support/users/raamdev/) * Last activity: [8 years ago](https://wordpress.org/support/topic/404-errors-91/page/2/#post-10324824) * Status: resolved