Title: Background processing: time exceeded Last modified: March 26, 2023 --- # Background processing: time exceeded * Resolved [jkburges](https://wordpress.org/support/users/jkburges/) * (@jkburges) * [3 years, 2 months ago](https://wordpress.org/support/topic/background-processing-time-exceeded/) * Hi, * I get the following error message after about 20mins and ~30% of the way through crawling during a publication: * > Background processing: time exceeded. * At that point, it seems as though no more pages can be successfully crawled. * This did not happen in a test environment. One thing that is different is that this env (production) is also serving production traffic, so there’d be more load on the servers, although looking at CPU and memory usage, it seems like there’s plenty of capacity. * The other difference (from testing) is that there are 2x servers on the backend, not sure if that is affecting anything. The uploads directing is shared between the two servers, so they should be seeing the same data (and same MySQL DB). - This topic was modified 3 years, 2 months ago by [jkburges](https://wordpress.org/support/users/jkburges/). - This topic was modified 3 years, 2 months ago by [jkburges](https://wordpress.org/support/users/jkburges/). - This topic was modified 3 years, 2 months ago by [jkburges](https://wordpress.org/support/users/jkburges/). Viewing 8 replies - 1 through 8 (of 8 total) * Thread Starter [jkburges](https://wordpress.org/support/users/jkburges/) * (@jkburges) * [3 years, 2 months ago](https://wordpress.org/support/topic/background-processing-time-exceeded/#post-16596643) * Seems like I was able to get it processing again by hitting wp-cron, i.e.: * > curl -v [http://localhost:80/wp-cron.php](http://localhost:80/wp-cron.php) * I have disabled WP_CRON via: * > define(“DISABLE_WP_CRON”, true); * so I guess that’s why I have to manually request wp-cron. * Still be good to understand why things are timing out here though compared to in my test environment. * Plugin Author [Team Staatic](https://wordpress.org/support/users/staatic/) * (@staatic) * [3 years, 2 months ago](https://wordpress.org/support/topic/background-processing-time-exceeded/#post-16598335) * [@jkburges](https://wordpress.org/support/users/jkburges/), first of all, thanks for your interest in Staatic to improve the speed and security of your WordPress site. * Staatic uses the WP Background Processing by Delicious Brains to allow the publication process to run in the background. The crawler component processes a specific number of pages/assets per batch, to prevent exceeding memory or time limits set by PHP or the host. Depending on the ability of Staatic to increase the PHP time limit, it will use different batch sizes. * It is very possible that the live traffic, combined with the selected batch size,( slightly) exceeds the time limit on your production environment. If this is the case, the publication sub-process may end and wait for WP-Cron to restart the process. In case WP-Cron is disabled, it is not able to resume automatically. * Note that Staatic will only try to increase the PHP time limit with 120 seconds, to prevent time-outs from occurring in other places of the chain, like the web server or load balancer. * What you can try is to create a simple mu-plugin file to lower the batch sizes set by Staatic, e.g. “/wp-content/mu-plugins/staatic_batch_sizes.php”, with the following content: * ```wp-block-code Background processing: time exceeded, consider lowering the batch size. * FWIW, I have tried enabled wp-cron, but I always seem to get an error message near the start: * > Publication failed during initialize_crawler task > RuntimeException: Unable to enqueue crawl url ‘[https://origin.www-production.biteable.com/wp-content/uploads/2022/11/Gradient-Header-Homepage-Original-round-corner-700×644.png](https://origin.www-production.biteable.com/wp-content/uploads/2022/11/Gradient-Header-Homepage-Original-round-corner-700×644.png): > Duplicate entry ‘\x18R\xC3\x0DEOPN\x97\xFBN\x96\xFD?\x10`’ for key ‘eRwwa_staatic_crawl_queue. > uuid’ in /var/www/html/wp-content/plugins/staatic/src/Bridge/CrawlQueue.php: > 64 > Stack trace:#0 /var/www/html/wp-content/plugins/staatic/vendor/staatic/crawler/ > src/Crawler.php(207): Staatic\WordPress\Bridge\CrawlQueue->enqueue(Object(Staatic\ > Crawler\CrawlUrl), 60)#1 /var/www/html/wp-content/plugins/staatic/vendor/staatic/ > crawler/src/Crawler.php(116): Staatic\Crawler\Crawler->addToCrawlQueue(Object( > Staatic\Crawler\CrawlUrl))#2 /var/www/html/wp-content/plugins/staatic/vendor/ > staatic/crawler/src/Crawler.php(100): Staatic\Crawler\Crawler->enqueueProvidedCrawlUrls( > Object(Staatic\Crawler\CrawlUrlProvider\AdditionalPathCrawlUrlProvider))#3 / > var/www/html/wp-content/plugins/staatic/vendor/staatic/framework/src/StaticGenerator. > php(86): Staatic\Crawler\Crawler->initialize(Object(Staatic\Crawler\CrawlUrlProvider\ > CrawlUrlProviderCollection))#4 /var/www/html/wp-content/plugins/staatic/src/ > Publication/Task/InitializeCrawlerTask.php(54): Staatic\Framework\StaticGenerator- > >initializeCrawler(Object(Staatic\Framework\Build), Object(Staatic\Crawler\ > CrawlUrlProvider\CrawlUrlProviderCollection))#5 /var/www/html/wp-content/plugins/ > staatic/src/Publication/BackgroundPublisher.php(212): Staatic\WordPress\Publication\ > Task\InitializeCrawlerTask->execute(Object(Staatic\WordPress\Publication\Publication), > true)#6 /var/www/html/wp-content/plugins/staatic/vendor/deliciousbrains/wp- > background-processing/classes/wp-background-process.php(127): Staatic\WordPress\ > Publication\BackgroundPublisher->task(‘Staatic\\WordPre…’)#7 /var/www/html/ > wp-content/plugins/staatic/src/Publication/BackgroundPublisher.php(137): Staatic\ > Vendor\WP_Background_Process->handle() > #8 /var/www/html/wp-content/plugins/staatic/vendor/deliciousbrains/wp-background- > processing/classes/wp-background-process.php(204): Staatic\WordPress\Publication\ > BackgroundPublisher->handle() > #9 /var/www/html/wp-includes/class-wp-hook.php( > 308): Staatic\Vendor\WP_Background_Process->handle_cron_healthcheck()#10 /var/ > www/html/wp-includes/class-wp-hook.php(332): WP_Hook->apply_filters(”, Array)# > 11 /var/www/html/wp-includes/plugin.php(565): WP_Hook->do_action(Array)#12 / > var/www/html/wp-cron.php(188): do_action_ref_array(‘staatic_backgro…’, Array)# > 13 {main} * The only way I’ve been able to get a successful publication all the way through is to have wp-cron disabled, then once publication gets to the crawling stage and I start seeing the background processing errors, I start polling /wp-cron. php using curl, which restarts the crawling after each background processing error. * Plugin Author [Team Staatic](https://wordpress.org/support/users/staatic/) * (@staatic) * [3 years, 1 month ago](https://wordpress.org/support/topic/background-processing-time-exceeded/#post-16617212) * Thanks for the additional information and we’re sorry to hear that the development version did not resolve the issue for you. * The “Unable to enqueue crawl url” error indicates that the crawler initiation task failed because of an earlier task failure, either due to an unknown error or timeout. This causes subsequent retries to fail immediately as a result of the previous midway failure. In this case, we expect that it is caused by the initiation task failing to complete within one of the following time limits: * 1) the time limit of 120 seconds, set by Staatic using `set_time_limit` (most common); 2) any limit enforced by your PHP configuration (e.g. PHP `max_execution_time` and/or PHP-FPM `request_terminate_timeout`);3) any limit enforced by the web server configuration (Apache/Nginx/Caddy/etc., request timeout). * The crawler initiation task is primarily responsible for enqueuing site URLs, which are to be processed by the crawler component during subsequent tasks. If you have a large uploads directory and configured “Additional paths” to include this directory in the build, it can take quite some time to scan everything. Especially when the directory is on an external (network) file system, as appears to be the case on your production environment. On your testing environment, scanning the uploads directory is probably much faster due to local storage, causing the task to complete without timing out. * If you would have used the `wp staatic publish` WP-CLI command, it would probably have worked without any issues, since this is (in most cases) not limited by these constraints. * We did make further adjustments to the background processing, to improve the reliability of publications triggered from WP-Admin. Also the default time limit has been increased to 300 seconds (5 minutes), and has been made configurable using the “Publication Task Timeout” setting under Staatic > Settings > Advanced. These changes have been included in a new beta version (1.4.2-beta2), which is now available from the WordPress plugin directory as the current Development Version. * Please go ahead and see if this version resolves the issue. If it doesn’t, can you please provide us with the Staatic diagnostics JSON file? This file can be generated by going to /wp-admin/admin.php?staatic=diagnostics and may be sent to [contact@staatic.com](https://wordpress.org/support/topic/background-processing-time-exceeded/contact@staatic.com?output_format=md). It contains some environment information that helps us further troubleshoot the issue (feel free to open it in a text editor to verify). * Looking forward to your feedback. * Thread Starter [jkburges](https://wordpress.org/support/users/jkburges/) * (@jkburges) * [3 years, 1 month ago](https://wordpress.org/support/topic/background-processing-time-exceeded/#post-16619650) * > used the `wp staatic publish` WP-CLI command * I gave this a try, and it worked. I will just go with this, since it’s what I’ll need to do to automate the publication process anyway. * Thanks for your support. - This reply was modified 3 years, 1 month ago by [jkburges](https://wordpress.org/support/users/jkburges/). * Plugin Author [Team Staatic](https://wordpress.org/support/users/staatic/) * (@staatic) * [3 years, 1 month ago](https://wordpress.org/support/topic/background-processing-time-exceeded/#post-16619951) * Sure, for automation that makes sense. Thanks for your patience. If you have any other questions, feel free to get back in touch. Viewing 8 replies - 1 through 8 (of 8 total) The topic ‘Background processing: time exceeded’ is closed to new replies. * ![](https://ps.w.org/staatic/assets/icon.svg?rev=2550278) * [Staatic - Static Site Generator for WordPress](https://wordpress.org/plugins/staatic/) * [Frequently Asked Questions](https://wordpress.org/plugins/staatic/#faq) * [Support Threads](https://wordpress.org/support/plugin/staatic/) * [Active Topics](https://wordpress.org/support/plugin/staatic/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/staatic/unresolved/) * [Reviews](https://wordpress.org/support/plugin/staatic/reviews/) * 8 replies * 2 participants * Last reply from: [Team Staatic](https://wordpress.org/support/users/staatic/) * Last activity: [3 years, 1 month ago](https://wordpress.org/support/topic/background-processing-time-exceeded/#post-16619951) * Status: resolved