Cache rebuild does not work – nginx preload mode
-
I have the following settings:
nginx (no apache) on http3, custom rules to serve the cached https html files (works)
preload mode running every 10 hours (works)
Expiry Time & Garbage Collection – Scheduler – Timer 600s
Cache rebuild enabled – does _not_ work.
Problem
Upon requesting a URL that is cached on the server (verified, the file exists but was created 600+s ago), wp super cache refuses to serve that file. Rather, it blocks the process, re-renders the page, and serves the newly rendered file. Verified by both changing timestamp on the html cached file, and the html content footer.
Expected
If Cache rebuild is enabled, the static html file must be served and asynchronously regenerate the html file.
-
Hi @adamh007.
Have you enabled debugging in the plugin and checked what the plugin says it’s doing when it’s serving a page that should be cached? If a file is older than the cache timer of 600 seconds (in your case), it won’t be served. It’s too old and considered obsolete.
Cache rebuild is for when a cache file was supposed to deleted by an operation like leaving a comment on a page, but instead, the next visitor gets the previous cache page, while WordPress creates the new cache file.
Donncha
Hi @donncha thanks for checking this and responding. What you say makes sense, but that’s not what the plugin is telling me:
Cache rebuild. Serve a supercache file to anonymous users while a new file is being generated. (Recommended)
From this most people would assume, including me, that a stale file can be served while the new one is being generated. I can see based on your answer this is not the case, but this is not trivial and caused a lot of unexpected delays in responses for us.
We since then increased the cache timeout below to the pregen time, but I would really expect the “Cache rebuild” to work as it is stated.
I’m wondering, under what circumstances is it better not to serve a stale file when I asked to “Serve a supercache […] while a new file is being generated”?
Alternatively, what’s the point of async pregen every x hours if the cache timeout is less than x hours and all of the pregen files would be thrown away?
I’m hoping you can see the two timeouts and the checkbox creating a very non-intuitive situation, where the server is not serving cached files, even though I asked it to do so, and I even run a periodic pregen process.
Thank you again for looking into this!
The “cache rebuild” feature probably should have a “learn more” link next to it because it’s a complicated feature.
I’m wondering, under what circumstances is it better not to serve a stale file when I asked to “Serve a supercache […] while a new file is being generated”?
When the cached page is considered stale and normal operations of the plugin have caused the cache file to be expired. It’s presumed that your server will generate a new page quickly. On the other hand, if the page has been invalidated by someone leaving a comment, for example, it shows external activity on the site, and there might be more pressure on the server to handle a large amount of incoming traffic. In those cases, the rebuild feature really helps to reduce the server load.
There’s no right or wrong way about this, as there are good arguments on both sides.
Alternatively, what’s the point of async pregen every x hours if the cache timeout is less than x hours and all of the pregen files would be thrown away?
You can enable “preload mode” which disables garbage collection. Otherwise, it’s up to you to set the garbage collection times according to how fresh you want your cached files.
I’m hoping you can see the two timeouts and the checkbox creating a very non-intuitive situation, where the server is not serving cached files, even though I asked it to do so, and I even run a periodic pregen process.
Bots and other visitors to your site will regenerate the cache files too. There are other plugins that create a static copy of your site, which might be what you want, but the aim of this plugin is to speed up popular pages so that most of the visitors hitting them get a cached copy. The first visitor might get a fresh copy, but it shouldn’t be too much slower if the rest of the site is cached and the server is doing little.
Yes, the garbage collection settings could be laid out better. If you have ideas or a patch to contribute, you can add it to the Jetpack repo at https://github.com/Automattic/jetpack/, and make sure to specify the “Super Cache” plugin.
hi @donncha thanks for your patience working through this. I enabled debug logging, which pointed to a different(?) problem, namely the home index file gets trashed several times during preload (which I might have perceived first as files are getting timed-out despite preload).
Given the preload takes several minutes with reasonable CPU load on the server, and the index gets deleted several, users get a huge spike in response time when trying to access the non-existent home during preload.
I checked the source, and it does indeed call an actual deletion, no further checks or skips: wp-super-cache/wp-cache-phase2.php at 961ab356684208c40065464ea0ac331ac1f9f829 · Automattic/wp-super-cache
Here are the snippets from the debug log, showing the repeated deletion of root file. I’m hoping I stripped all sensitive data from it…
The wp-cache-phase2 file is 3.5k lines long -eeeek- so I don’t pretend to fully understand yet why this behaviour exists. However, this does not make sense, neither on face value, nor thinking about it more – could you please confirm this is a bug?
17:01:40 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wp_cron_preload_cache: fetched https://myexample.com/blog/currentpage1
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wp_cron_preload_cache: scheduling the next preload in 3 seconds.
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: pagesample
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: about
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: faq
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: sitemap_index.xml
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: feedbackpageisgoodforyou
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: terms
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: pagesample2
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: services
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: index-https.html
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/index-https.html
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: resources
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: anotherpage
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: ..
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: blog
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: .
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: contact
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: reading files: somanypages
17:01:41 66527 /wp-cron.php?doing_wp_cron=1732813277.6925950050354003906250 wpsc_delete_files: remove directory ABSPATH/wp-content/cache/supercache/myexample.com/
…
17:06:44 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wp_cron_preload_cache: fetched https://myexample.com/terms
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wp_cron_preload_cache: scheduling the next preload in 3 seconds.
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: pagesample
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: about
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: faq
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: sitemap_index.xml
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: feedbackpageisgoodforyou
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: terms
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: pagesample2
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: services
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: index-https.html
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/index-https.html
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: resources
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: anotherpage
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: ..
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: blog
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: .
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: contact
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: reading files: somanypages
17:06:45 66390 /wp-cron.php?doing_wp_cron=1732813590.5101430416107177734375 wpsc_delete_files: remove directory ABSPATH/wp-content/cache/supercache/myexample.com/
…
17:09:28 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wp_cron_preload_cache: fetched https://myexample.com/blog/currentpage2
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wp_cron_preload_cache: scheduling the next preload in 3 seconds.
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: pagesample
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: about
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: faq
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: sitemap_index.xml
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: feedbackpageisgoodforyou
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: terms
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: pagesample2
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: services
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: index-https.html
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/index-https.html
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: resources
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: anotherpage
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: ..
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: blog
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: .
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: contact
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: reading files: somanypages
17:09:29 67725 /wp-cron.php?doing_wp_cron=1732813753.8266520500183105468750 wpsc_delete_files: remove directory ABSPATH/wp-content/cache/supercache/myexample.com/
…
17:11:10 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wp_cron_preload_cache: fetched https://myexample.com/services/unicornpetting
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wp_cron_preload_cache: scheduling the next preload in 3 seconds.
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: pagesample
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: about
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: faq
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: sitemap_index.xml
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: feedbackpageisgoodforyou
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: terms
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: pagesample2
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: services
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: index-https.html
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/index-https.html
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: resources
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: anotherpage
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: ..
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: blog
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: .
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: contact
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: reading files: somanypages
17:11:11 66527 /wp-cron.php?doing_wp_cron=1732813855.2376370429992675781250 wpsc_delete_files: remove directory ABSPATH/wp-content/cache/supercache/myexample.com/
…
17:13:12 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wp_cron_preload_cache: fetched https://myexample.com/about
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wp_cron_preload_cache: scheduling the next preload in 3 seconds.
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: pagesample
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: about
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: faq
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: sitemap_index.xml
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: feedbackpageisgoodforyou
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: terms
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: pagesample2
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: services
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: index-https.html
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: deleting ABSPATH/wp-content/cache/supercache/myexample.com/index-https.html
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: resources
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: anotherpage
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: ..
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: blog
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: .
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: contact
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: reading files: somanypages
17:13:13 67962 /wp-cron.php?doing_wp_cron=1732813983.6998479366302490234375 wpsc_delete_files: remove directory ABSPATH/wp-content/cache/supercache/myexample.com/digging into the code further, @donncha it seems like the bug is due to a call to
wpsc_delete_files( get_supercache_dir() );
This happens after every batch of 10 posts is processed (WPSC_PRELOAD_POST_COUNT), and the preload reschedules itself. However, incorrectly, deletes the root folder after every batch.
Making the matter worse, wp-cron will not fire unless there is a visitor to the side, so the preload process just hangs, waiting for someone to hit the uncached home page – then the preload cron will slow down generating the cachefile, adding extra delay to serving the home index.
This seems a rather big problem in wpsc, not sure how this has never been noticed before 🤔
It’s not as bad as you think it is, but when it was first implemented, the preload system handled 100 pages at a time, rather than 10. 10 was picked during the last update because some sites on very low powered hosting had problems. (In hindsight, they should have upgraded their hosting, as their server could barely cope with a 2 pages loading in parallel. They eventually did upgrade.)
Set WPSC_PRELOAD_POST_COUNT to your own limit, and the homepage won’t be deleted so often.
WP Cron operates async, with a request to wp-cron.php to run the preload, so the front page of your site will be served uncached as quickly as it ever will be.
If you want, you can disable WP Cron by setting “DISABLE_WP_CRON” and running it from the system cron. That may be more reliable if you have a low traffic website.
Preload could be improved. It’s not perfect, but we don’t have plans to change it in the near future.
It’s not as bad as you think it is
It’s actually much worse than you think it is. It is severely bugged, leading to massive spikes in response times. Look at the chart below. This is your bug, refusing to fix, causing spikes up to 4 seconds (!) everywhere around the world.
EDIT: images are bugged out in the post, here is a link to them: https://imgur.com/a/V0s2vCF
And another bug..
I found another bug, terrifyingly. GC runs anyway, regardless of preload mode. This is likely due to setting cache_max_time somewhere to preload mode time.
My settings:
My log:
wp_cron_preload_cache: doing taxonomy preload.
15:57:10 75004 /wp-cron.php?doing_wp_cron=1732895829.8098969459533691406250 wp_cron_preload_cache: no more posts to get. Limit (58) reached.
15:57:10 75004 /wp-cron.php?doing_wp_cron=1732895829.8098969459533691406250 wp_cron_preload_cache: no more posts. scheduling next preload in 600 minutes.
15:57:10 75004 /wp-cron.php?doing_wp_cron=1732895829.8098969459533691406250 wp_cron_preload_cache: clean expired cache files older than 36000 seconds.This is clearly not what the code was intended to do in wp-cache.php:
if ( $cache_max_time > 0 ) { // GC is NOT disabled
wp_cache_debug( "wp_cron_preload_cache: clean expired cache files older than $cache_max_time seconds.", 5 );
wp_cache_phase2_clean_expired( $file_prefix, true ); // force cleanup of old files.
}But in the same file you made a mistake and set cache_max_time to nonzero:
$cache_max_time = (int)$wp_cache_preload_interval * 60; // fool the GC into expiring really old filesI understand you are hesitant to touch these files, contaning over 4k lines each*, but this is so buggy and dysfunctional that I find your position indefensible.
*how did you even get to this point? Methods several pages long, files thousands of lines long. I understand it grew organically, but this is not a hobby project, it’s a for-profit organization.
The fixes you’d need to implement
- Don’t delete home index after every post-batch – the line I mentioned in my previous post can be safely deleted, no reason to wipe out home index after a preload run, it’s clearly a bug
- Don’t set cache_max_time to preload time – the code already caters for disabled GC – it’s also clearly a bug
If in doubt, again, please do look at the chart above – your cache plugin causing massive spikes, despite every setting on the plugin indicating this would never happen.
-
This reply was modified 1 year, 6 months ago by
adamh007.
It’s not as bad as you think it is
Following up on this, @donncha , based on the chart below, try to guess when I fixed the bugs, that were “not as bad”
Thanks, @adamh007 for the detailed reply. We will look into those issues, and get back to you.
BTW, further updates will be on the GitHub issue, so you can follow that. I’ll mark this issue resolved.
The topic ‘Cache rebuild does not work – nginx preload mode’ is closed to new replies.