Sync Loops – constant update pulls

  • Resolved davidrivierafjorge

    (@davidrivierafjorge)


    4 years, 5 months ago

    I have a site with version 2.1.1 that pulls from SF->WP on Create/Update/Delete. It seems after the recent 2.1.0/2.1.1 update something happened, and all records seem to be re-syncing continuously. I was able to throttle it down, but the pending re-sync queue continues to build up. We temporarily disconnected the plugin to stop updates. There are ~4000 records in SF, about 615 should only sync to the website. We have a field “Add to Web” that is set to yes for those 615, but it does need to touch each of those 4000 records to get the “Add to Web” value. This issue resulted in over 500,000 API calls in one week.

    When we reconnected, the update syncs resumed at the rapid pace.

    This happened just shortly before a bulk update was done to all SF records on January 1. I suspect it began on December 28 when I upgraded from 2.0.3 -> 2.1.1.

    Is there a way to flush any pending update syncs and only pull future triggers? It seems to be in an update sync loop. If there was any change in 2.1.0/2.1.1 that changed how record sync status is set, we may need to adjust our custom code to reflect that. I cleared all pending tasks in Action Scheduler, but it seemed to rebuild pending tasks when the plugin was reconnected to SF. [We pull one primary Object from SF, then if it is “Add to Web”, we pull other related SF objects with SOQL.] I am reviewing our code to see exactly what is happening step by step.

    Could we update wp_object_sync_sf_object_map columns object_updated and last_sync to the current time so everything is up-to-date? Basically, how can we tell the website and plugin that there are no pending sync actions needed and only sync future triggers?

    Does turning on logging or debugging in the Object Sync plugin display any feedback on the live webpages, or just on the plugin admin page?

    Any insight is useful at this point.

Viewing 12 replies - 1 through 12 (of 12 total)
  • Plugin Author Jonathan Stegall

    (@jonathanstegall)

    4 years, 5 months ago

    Hm, here are a couple of things to try:

    1. You can certainly turn on logging/debugging. It does not do anything on live webpages; it creates entries of the Log post type that are only accessible in the admin to user roles that can modify Salesforce settings. It can create a lot of entries, but it seems like it would be worth it.
    2. You might try clearing the plugin cache. It clears some of the wp_options values that could be causing problems.
    3. If that doesn’t work and the logs are not helpful, you might also try deactivating the plugin and reactivating it.
    4. It doesn’t sound to me like your custom code would be related, but it’s possible.

    I’m not sure what would be causing this, but I’d like to know if there’s a bug with it. The main change to the 2.1 version that could be related is how the plugin sets and reads its option and transient values, both of which are stored in wp_options. It certainly seems like it should have been a seamless update, but that is possibly where the problem is.

    Thread Starter davidrivierafjorge

    (@davidrivierafjorge)

    4 years, 5 months ago

    Plugin cache was cleared in an attempt to resolve this.

    One note about how the plugin was updated: we update plugins on local dev code base, then git push it to WPEngine. Were there any database changes that happen during the upgrade that were missed if we didn’t use the WP Dashboard Update process? Would deactivating and reactivating correct those missed db updates?

    The site owner is very concerned about any downtime or maintenance page on the front end when doing core/plugin updates.

    Plugin Author Jonathan Stegall

    (@jonathanstegall)

    4 years, 5 months ago

    It’s possible that the upgrade process missed a change to how the options are saved, but I don’t think it would be because of using that upgrade process. Certainly this plugin isn’t written to affect anything on the front end; it’s intended to only sync data that then WordPress is able to work with on its own.

    Thread Starter davidrivierafjorge

    (@davidrivierafjorge)

    4 years, 5 months ago

    Our current status:

    4000 objects in SF. Only 615 should be on the website using a field in the SF object. However, this issue appeared with the plugin update (or the ACF plugin, we use custom fields; nothing in our code seems suspicious at this time.)

    A bulk update was made to all 4000 objects, meaning we need to at least pull to check the Add_To_Web__c field. If Yes, we process more pulls using the SFAPI and SOQL through the plugin.

    Our main question: where is the queue of update triggers stored? Is in SF? Is it in the wp_object_sync_sf_object_map table? Can we bypass all triggers? When we reconnected, it appeared to run through everything again.

    Could we update wp_object_sync_sf_object_map columns object_updated and last_sync to the current time so everything appears to be up-to-date? Will this bypass any update triggers that have a mod date less than today’s date?

    Thanks in advance for your insight. We will be reconnecting soon to enable logging.

    Plugin Author Jonathan Stegall

    (@jonathanstegall)

    4 years, 5 months ago

    The queue runs through the Action Scheduler library, which has its own custom tables. I’m not sure that would be helpful for you, but you could look into it.

    The wp_object_sync_sf_object_map isn’t used to decide when updates are run, it’s basically just a status tracker. So you can update those values, but they’ll be overridden the next time it runs.

    The functionality that determines whether a record needs to be updated (basically it tries to see if a changed record was updated already or if the change is new and needs to be used by the plugin) is in the wp_options table. The transients expire when they’re used. The option values sometimes depend on which object is being loaded and which fieldmap it is, but their names start with object_sync_for_salesforce_pull_.

    Thread Starter davidrivierafjorge

    (@davidrivierafjorge)

    4 years, 5 months ago

    When we reconnected the plugin to Salesforce this time, it was not syncing. A series of saving options or the field map within the plugin resumed syncing for about 10 records, then syncing stopped.

    Logging (with Debug) doesn’t show any configuration errors. It does log the SOQL calls we make.

    I was able to get the plugin to start syncing again by an unknown combo of saving plugin settings, deactivating/reactivating the plugin, and clearing the cache. It stopped after 11 records again.

    I’m asking the site owner to check if Salesforce is rate limiting, or if the web host is rate limiting.

    Any insight?

    Plugin Author Jonathan Stegall

    (@jonathanstegall)

    4 years, 5 months ago

    Since the plugin is logging the SOQL calls that you’re making, I think try to compare the queries and see if they show why it would only sync a few records. The way the query is supposed to work is:

    1. Run the query with the LastModifiedDate set based on the last time it ran.
    2. Load the first batch of records, and detect whether there are additional batches to process.
    3. Process those additional batches, if any.
    4. Increment the LastModifiedDate and cache the query so it can run the next time.

    Thread Starter davidrivierafjorge

    (@davidrivierafjorge)

    4 years, 5 months ago

    On my local dev I rolled back to version 2.0.3 and it is still very intermittent. At this point, I’m starting to look at something on SF’s side. I’m waiting for the site/SF account owner to get feedback from SF.

    I’m growing more confident that it’s not something with the plugin.

    Plugin Author Jonathan Stegall

    (@jonathanstegall)

    4 years, 5 months ago

    To make sure that it is SF, maybe you can try this on your local dev:

    1. Make a backup of your wp_options table (I do this just in case even if it’s local)
    2. delete from wp_options where option_name LIKE 'object_sync_for_salesforce_pull_%'

    These values don’t get cleared by the cache clear (maybe they should, but that has always felt aggressive), although they do get cleared if you uninstall the plugin and clear its data. In any case, this should essentially start the process that checks for data changes over from scratch. If you’re running 2.0.3 and you do this, maybe you can see whether it’s still intermittent or whether it behaves like it should have been all along.

    Plugin Author Jonathan Stegall

    (@jonathanstegall)

    4 years, 4 months ago

    I’ve just released a version 2.1.1, and I’d like to find out whether that release improves or resolves, or in any other ways changes, this.

    Thread Starter davidrivierafjorge

    (@davidrivierafjorge)

    4 years, 4 months ago

    I didn’t follow up after your Jan 6 message. We uninstalled the plugin, deleting all data, and reinstalled it, then restored the backed up settings. This process resolved our continuing sync issue.

    I’ll keep an eye out for version 2.1.1 and let you know if there are any issues or concerns.

    Thanks for your work on this!

    Plugin Author Jonathan Stegall

    (@jonathanstegall)

    4 years, 4 months ago

    Sorry, I meant to say that 2.1.2 was just released, not 2.1.1. But in any case, thank you for your reply. I’m going to mark this one as resolved for now.

Viewing 12 replies - 1 through 12 (of 12 total)

The topic ‘Sync Loops – constant update pulls’ is closed to new replies.