Title: Sync throughput Last modified: February 22, 2021 --- # Sync throughput * Resolved [esowers](https://wordpress.org/support/users/esowers/) * (@esowers) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/) * I’m setting up a new site to sync records. On the salesforce side we have about 4,000 records that get updated frequently by salesforce tasks. This initiates a sync to WordPress. However I’m running into an issue with the WordPress Scheduled Actions not keeping up with the pull_process_records tasks. I currently have about 6500 pending actions and that’s not going down at all. * I see the options in the plugin for changing pull query limit, batch size and concurrent batches. However when I increase those it doesn’t seem to allow more than 20-40 records to be processed at a time. I read the documentation on Action Scheduler’s website and they said turning concurrent batches to 0 would process everything, however it doesn’t look like that works in this instance. * Can you give any pointers on what I might be missing? I have the wp-cron scheduled to run every 5 minutes as a server scheduled task. * Thanks! - This topic was modified 5 years, 3 months ago by [esowers](https://wordpress.org/support/users/esowers/). Viewing 15 replies - 1 through 15 (of 19 total) 1 [2](https://wordpress.org/support/topic/sync-throughput/page/2/?output_format=md) [→](https://wordpress.org/support/topic/sync-throughput/page/2/?output_format=md) * Plugin Author [Jonathan Stegall](https://wordpress.org/support/users/jonathanstegall/) * (@jonathanstegall) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14086396) * Hm, I’m not sure. The Action Scheduler version that we run is a major version behind, but I don’t think that is likely to be the reason for this issue. * When you say these 4000 records get updated frequently, how frequently are they updated? Is the plugin failing to get through the records before they get updated again? * I’m also curious if you are seeing anything in the plugin logs. I don’t normally think it’s a good idea to turn debug mode on in a production environment (it creates a lot of entries) but it might be worth it for a short period of time to see if those logs show anything useful. I think if I were you I’d be curious about the SOQL query that the plugin is running and if/how it gets incremented on each run between when the records get updated in Salesforce. * I’m not entirely sure what to do after that, in a production environment. I think if none of that is helpful you may need to try to get a large dataset into a sandbox and work with it in a local WordPress so you can do more error logging. * Thread Starter [esowers](https://wordpress.org/support/users/esowers/) * (@esowers) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14107998) * so I think I’m narrowing down our problem and just wanted to circle back with some follow-up questions. * Our total salesforce contact records are around 300k. The record type we’re trying to sync is around 4k records. When we set the field mapping, we’ve selected the specific record type we want, however Action Scheduler > pending shows processes for records outside of that record type. Those records are filling up the Action Scheduler queue and take a while to process. The problem is with those record types there are no logs or any other information on that because it looks like they just get ignored because they’re the wrong record type? * So my question. 1) When the salesforce query is run, is it supposed to pull all records first, then let action scheduler process through the batch for the ones that match record types? Or is the pull really only supposed to be the record type selected in the fieldmap? * 2) From my test we’re processing about 53 records in an hour. That seems quite slow as my batch size is 100 and concurrent batches is 5 running every 4 minutes. Could there be a PHP limit that it’s hitting or something? Adjusting batch size and concurrent batches doesn’t seem to do anything. * Thanks for any help you can provide. * Plugin Author [Jonathan Stegall](https://wordpress.org/support/users/jonathanstegall/) * (@jonathanstegall) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14108147) * Hm, well I really think it would help you if you can get the plugin to log the full SOQL query that it is running each time. Either locally or in a staging environment (which would be ideal!) or in production if you can do it quickly. Then you can find out exactly what query it is sending to Salesforce when it runs and if there’s something wrong. * To answer the questions: * 1. The query is supposed to only pull records that match record types for each fieldmap. That is, if there’s a record type value for a given fieldmap it should limit that fieldmap’s query to the respective RecordTypeId. 2. It could be hitting a limit, but that’s beyond what I can speak to as it probably depends on your server’s capabilities. * Thread Starter [esowers](https://wordpress.org/support/users/esowers/) * (@esowers) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14108659) * Thanks. I did turn on the debug and it looks like the query doesn’t include the record type. This is pulling from the Salesforce accounts. Am I missing something? * Debug: on Salesforce API HTTP Request to URL: [https://xxx.salesforce.com/services/data/v50.0/query?q=SELECT+Id%2C+Name%2C+field0__c%2C+field1__c%2C+field2__c%2C+field3__c%2C+field4__c%2C+field5__c%2C+Phone%2C+field6__c%2C+Website%2C+field7__c%2C+field8__c%2C+field9__c%2C+LastModifiedDate%2C+CreatedDate+FROM+Account+WHERE+LastModifiedDate+%3E+2021-02-26T22%3A18%3A36Z+ORDER+BY+LastModifiedDate+ASC+LIMIT+100](https://xxx.salesforce.com/services/data/v50.0/query?q=SELECT+Id%2C+Name%2C+field0__c%2C+field1__c%2C+field2__c%2C+field3__c%2C+field4__c%2C+field5__c%2C+Phone%2C+field6__c%2C+Website%2C+field7__c%2C+field8__c%2C+field9__c%2C+LastModifiedDate%2C+CreatedDate+FROM+Account+WHERE+LastModifiedDate+%3E+2021-02-26T22%3A18%3A36Z+ORDER+BY+LastModifiedDate+ASC+LIMIT+100). * Plugin Author [Jonathan Stegall](https://wordpress.org/support/users/jonathanstegall/) * (@jonathanstegall) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14123614) * I’m thinking there’s a bug here, after looking deeper into what is happening. It doesn’t seem to be using the record types on the query that actually gets sent to Salesforce. Clearly it seems like it should, but it’s not. I’m not sure if that was intentionally thought out (probably not?) but I do think it can be fixed in a next release. * In the meantime, it would be possible to filter the query with the `object_sync_for_salesforce_pull_query_modify` hook in a way that won’t be overridden when this gets fixed. * Plugin Author [Jonathan Stegall](https://wordpress.org/support/users/jonathanstegall/) * (@jonathanstegall) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14128824) * I’m going to track this bug and whenever it gets resolved [here](https://github.com/MinnPost/object-sync-for-salesforce/issues/383). * Plugin Author [Jonathan Stegall](https://wordpress.org/support/users/jonathanstegall/) * (@jonathanstegall) * [5 years, 3 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14156758) * [@esowers](https://wordpress.org/support/users/esowers/) wondering if you’d be willing to give this fix a test, if your setup allows you to do that. It’s been working well in my local testing so far. The full fix is in the classes/salesforce_pull. php file, which you can find here: [https://github.com/MinnPost/object-sync-for-salesforce/blob/383-fix-record-type-conditions/classes/salesforce_pull.php](https://github.com/MinnPost/object-sync-for-salesforce/blob/383-fix-record-type-conditions/classes/salesforce_pull.php). There’s a [pull request](https://github.com/MinnPost/object-sync-for-salesforce/pull/384) as well. * I’ll try to get it released soon either way, but a test would be helpful if you’ve got the capacity. * Plugin Author [Jonathan Stegall](https://wordpress.org/support/users/jonathanstegall/) * (@jonathanstegall) * [5 years, 2 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14191234) * I’m going to release this as a fix as soon as I can get the release ready, hopefully this week. Going to mark this as resolved, but we can return to it if the new release isn’t sufficient. * Thread Starter [esowers](https://wordpress.org/support/users/esowers/) * (@esowers) * [5 years, 2 months ago](https://wordpress.org/support/topic/sync-throughput/#post-14191283) * Awesome, thanks! * [Tim Nolte](https://wordpress.org/support/users/tnolte/) * (@tnolte) * [5 years ago](https://wordpress.org/support/topic/sync-throughput/#post-14453146) * [@esowers](https://wordpress.org/support/users/esowers/) I don’t see it mentioned but did you ever change the number of records to process that is present on the first settings tab? There are 2 settings on that first tabe which also play into the additional settings in the Schedule tab. * Thread Starter [esowers](https://wordpress.org/support/users/esowers/) * (@esowers) * [5 years ago](https://wordpress.org/support/topic/sync-throughput/#post-14453243) * [@tnolte](https://wordpress.org/support/users/tnolte/) Yea, I’ve tried a number of different settings with the query record limit and the pull throttle, however, neither of those seemed to speed up our syncing process. The part I’m stuck on is I’m not sure where the bottleneck is. I’ve tried high query limits and low throttles and low throttles and high query limits. Neither of which seemed to help. * I landed on it maybe just being a limitation of the server we’re on as I know actionscheduler is supposed to be able to process a lot more records than what we’re sending at it. I’ve just run out of ideas. * For example I did a refresh of 1,000 records yesterday and it took about 10 hours to finish the sync in the WordPress site. Here are the settings at the time Query Limit: 200 Pull Throttle: 5 seconds Batch size: 100 Concurrent Batches: 5 Run Schedule every 5 min. * Not sure if those can go higher without timing out the cron job. Normally this isn’t a problem as one-off salesforce updates come over within a couple of minutes, it’s just the large refreshes that take a while. * [Tim Nolte](https://wordpress.org/support/users/tnolte/) * (@tnolte) * [5 years ago](https://wordpress.org/support/topic/sync-throughput/#post-14453480) * [@esowers](https://wordpress.org/support/users/esowers/) OK, that is interesting to see. We will be rolling out our new WordPress Salesforce portal and we are also looking to bulk load thousands of objects across 6 object types, up to 1500 of 1 object type. We are going to be trying again to get them all bulk loaded in our Development encourage for a test as we are hoping to get all of the data preloaded. * [Tim Nolte](https://wordpress.org/support/users/tnolte/) * (@tnolte) * [5 years ago](https://wordpress.org/support/topic/sync-throughput/#post-14453481) * Oh, I will say that in one of last attempts we did manage to load about 350 records/ objects in about 15 minutes but then it just stopped. * Plugin Author [Jonathan Stegall](https://wordpress.org/support/users/jonathanstegall/) * (@jonathanstegall) * [5 years ago](https://wordpress.org/support/topic/sync-throughput/#post-14453573) * One thing I haven’t done is test ActionScheduler 3.0 (or 3.1) with this plugin. It’s on the list of tasks, but my assumption is that it would require some devoted time because it’s such a different library than the 2.x version. I do wonder if it would help some of these issues. * Thread Starter [esowers](https://wordpress.org/support/users/esowers/) * (@esowers) * [5 years ago](https://wordpress.org/support/topic/sync-throughput/#post-14453985) * We are using the plugin on 2 separate WordPress installed. In our first install we did the initial refresh after we mapped all the fields to bring over about 2-3000 records which took a little while, then after that individual record updates really only take about 15min to come over (which isn’t bad). We have 4 different Salesforce objects mapped through the plugin, so it gets some good use as our staff are regularly updating salesforce records (as well as process builder and workflow processes doing their thing). * As for our second install, that’s where we’re running into more of a bottleneck. For some reason the processing seems to go slower on the second install. However, that’s on a server with our other sites, so it may just be server resources are limited. The update [@jonathanstegall](https://wordpress.org/support/users/jonathanstegall/) did to limit record types helped limit the queries, however it does still take some time to run. * [@jonathanstegall](https://wordpress.org/support/users/jonathanstegall/) That would be awesome if 3.1 would help speed that up. I’m happy to help test any changes, however my coding skills are still fairly limited. =) Thanks! Viewing 15 replies - 1 through 15 (of 19 total) 1 [2](https://wordpress.org/support/topic/sync-throughput/page/2/?output_format=md) [→](https://wordpress.org/support/topic/sync-throughput/page/2/?output_format=md) The topic ‘Sync throughput’ is closed to new replies. * ![](https://s.w.org/plugins/geopattern-icon/object-sync-for-salesforce.svg) * [Object Sync for Salesforce](https://wordpress.org/plugins/object-sync-for-salesforce/) * [Frequently Asked Questions](https://wordpress.org/plugins/object-sync-for-salesforce/#faq) * [Support Threads](https://wordpress.org/support/plugin/object-sync-for-salesforce/) * [Active Topics](https://wordpress.org/support/plugin/object-sync-for-salesforce/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/object-sync-for-salesforce/unresolved/) * [Reviews](https://wordpress.org/support/plugin/object-sync-for-salesforce/reviews/) * 19 replies * 3 participants * Last reply from: [Tim Nolte](https://wordpress.org/support/users/tnolte/) * Last activity: [5 years ago](https://wordpress.org/support/topic/sync-throughput/page/2/#post-14459548) * Status: resolved