Title: Crawler Threads & Server Load Limit Last modified: November 27, 2022 --- # Crawler Threads & Server Load Limit * Resolved [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/) * Hi, * Our webserver is a very fast 4 core (AMD) machine with plenty or RAM, but when a crawler runs, the frontend becomes very unresponsive (from less than an sec serving a page to 30-50 sec). * Settings for Treads are 4 and Server Load Limit is 3. * We have thousands of pages and as you can see in this [image](https://worldofsuperyachts.com/wp-content/uploads/2022/11/litespeed-crawler.jpg), the runtime for all 8 crawlers with 4 treads goes up to 75513 sec (about 21 hours), meaning the frontend are quite slugish a lot of the time over a day. We have daily updates to many pages, so we need to run the crawlers daily. * How do we setup litespeed better? * See all crawler [settings here](https://worldofsuperyachts.com/wp-content/uploads/2022/11/litespeed-crawler-settings.jpg). * Thanks! Viewing 15 replies - 1 through 15 (of 30 total) 1 [2](https://wordpress.org/support/topic/crawler-threads-server-load-limit/page/2/?output_format=md) [→](https://wordpress.org/support/topic/crawler-threads-server-load-limit/page/2/?output_format=md) * [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * (@serpentdriver) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16238754) * Do you have access to command line interface? (SSH) * Thread Starter [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16238799) * Yes, our techie has access. What would you have him do? * [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * (@serpentdriver) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16238808) * Run “uptime” command while the crawler is running and post the result. - This reply was modified 3 years, 6 months ago by [serpentdriver](https://wordpress.org/support/users/serpentdriver/). * Thread Starter [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16238825) * He will do it tomorrow. * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16240848) * what’s your env ? * any control panel in place ? * the default max php concurrency is 10 , this might be the bottleneck * [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * (@serpentdriver) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16240874) * > the default max php concurrency is 10 , this might be the bottleneck * If [@webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) complains that the frontend seems to have glitches, then I suspect that his server is not designed for the amount of articles. The specified values for the CPU cores and RAM then seem to be more vCores and vRam and he runs shared hosting. * Thread Starter [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241782) * Uptime result (while crawler 8 is running): 192958 up 16 days, 23:59, 1 user, load average: 1.74, 1.77, 1.83 * Thread Starter [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241794) * It’s a cloud solution with 4 vcores and 16bg vram. Once the crawlers are done, the site is lightning fast. A page (post) not crawled is about 10 sec to load, and up til 50 sec while any crawler is running (if that page is not in cached). * [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * (@serpentdriver) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241796) * I can’t say with absolute certainty because more information would be needed, but it looks like your server is chronically overloaded and you should consider switching to a more powerful server. In any case, you should reduce the number of threads to a maximum of 3. * Thread Starter [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241800) * Crawler Status: 28 Nov 2022 19:38:15, Size: 5541, Crawler: #8, Position: 2238, Threads: 4, Status: crawling, updated position * [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * (@serpentdriver) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241811) * > It’s a cloud solution with 4 vcores and 16bg vram. * I had already assumed that these are virtual resources and that you have to share these resources with others. Given your high item count and the fact that WooCommerce is a performance killer, you should consider switching to a dedicated server. * Thread Starter [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241812) * > you should reduce the number of threads to a maximum of 3. * Means crawling will take more that 24 hours. * We will switch to a dedicated 12 core server soon. Still would like to dial in the perfect crawler settings before this happens. * Thread Starter [webarkitekterne](https://wordpress.org/support/users/webarkitekterne/) * (@webarkitekterne) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241817) * > the fact that WooCommerce is a performance killer * Not really using woo much, but are heavy on ACF posts. * [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * (@serpentdriver) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241820) * Just to give you an example. I have developed an alternative crawler that crawls around **90.000** URLs on a non-WP site in just under 5 hours. **On a Shared Hosting.** * [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * (@serpentdriver) * [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/#post-16241824) * > Not really using woo much, but are heavy on ACF posts. * That doesn’t matter. WordPress + WooCommerce + Plugins kills performance. Viewing 15 replies - 1 through 15 (of 30 total) 1 [2](https://wordpress.org/support/topic/crawler-threads-server-load-limit/page/2/?output_format=md) [→](https://wordpress.org/support/topic/crawler-threads-server-load-limit/page/2/?output_format=md) The topic ‘Crawler Threads & Server Load Limit’ is closed to new replies. * ![](https://ps.w.org/litespeed-cache/assets/icon-256x256.png?rev=2554181) * [LiteSpeed Cache](https://wordpress.org/plugins/litespeed-cache/) * [Frequently Asked Questions](https://wordpress.org/plugins/litespeed-cache/#faq) * [Support Threads](https://wordpress.org/support/plugin/litespeed-cache/) * [Active Topics](https://wordpress.org/support/plugin/litespeed-cache/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/litespeed-cache/unresolved/) * [Reviews](https://wordpress.org/support/plugin/litespeed-cache/reviews/) ## Tags * [crawler](https://wordpress.org/support/topic-tag/crawler/) * [frontend](https://wordpress.org/support/topic-tag/frontend/) * [settings](https://wordpress.org/support/topic-tag/settings/) * 30 replies * 3 participants * Last reply from: [serpentdriver](https://wordpress.org/support/users/serpentdriver/) * Last activity: [3 years, 6 months ago](https://wordpress.org/support/topic/crawler-threads-server-load-limit/page/2/#post-16260331) * Status: resolved