Zettle Webhook calls killing server

  • Resolved markisu72

    (@markisu72)


    4 years, 1 month ago

    Hi all,

    when synching with Zettle, the webhook /wp-json/zettle/v1/webhook/listen gets called quite often (sometimes it seems 20+ requests per second).

    This creates brutal load on my machine (2 CPUs capped) resulting in not only timeouts calling the webhook (10s) but only for customers (30s) in certain circumstances.

    We experienced this twice today in production and I am able to reproduce it in a testing environment as well.

    To be clear: the load by Zettle produces NGINX timeouts for our webshop customers.
    This is absolutely horrible.

    Is there a way to reduce the aggressiveness the webhook gets called?
    Can I create some db indexes to help speeding the calls up?

    In general, the calls have very bad response times 3-5 seconds in average, I’d say but sometimes (e.g. during initial sync) dozens of subsequent PusherRequests run into a 10s web service timeout.

    What can I do, to reduce load, speed up the calls and avoid Zettle sync calls to affect availability of our server?

    Thanks
    Markus

    • This topic was modified 4 years, 1 month ago by markisu72.
    • This topic was modified 4 years, 1 month ago by markisu72.
Viewing 8 replies - 1 through 8 (of 8 total)
  • Thread Starter markisu72

    (@markisu72)

    4 years, 1 month ago

    Some example from my logs

    
52.215.24.2 - - [20/Apr/2022:23:15:02 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:02 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.032 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.030 1 NGINX - "POST REQUEST" responsive
HTTP/1.1" 302 138 "-" "curl/7.58.0" 0.000 0 NGINX - "" responsive
52.215.24.2 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.029 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:01 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.030 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.031 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:15:00 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:59 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.029 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:59 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.029 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:59 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.041 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:59 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:59 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.032 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.030 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.033 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.040 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.029 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.032 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.034 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.028 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.033 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:58 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:57 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:57 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:57 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:57 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:57 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:57 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:56 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:56 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.040 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:56 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.037 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:56 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:56 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.040 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.040 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.030 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.040 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.030 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.035 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.038 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.034 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:55 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.032 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:54 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:54 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.029 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:54 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.036 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:54 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.032 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:54 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.039 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:54 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.032 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:53 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.035 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:53 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.034 1 NGINX - "POST REQUEST" responsive
52.215.24.2 - - [20/Apr/2022:23:14:53 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.040 1 NGINX - "POST REQUEST" responsive
34.240.78.7 - - [20/Apr/2022:23:14:53 +0200] "POST /wp-json/zettle/v1/webhook/listen HTTP/1.1" 499 0 "-" "PusherApplication (WebHookPushClient)" 10.040 1 NGINX - "POST REQUEST" responsive
    • This reply was modified 4 years, 1 month ago by markisu72.
    Plugin Support Syde Joost

    (@joostvandevijver)

    4 years, 1 month ago

    Hello @markisu72

    thank you for reaching out to us, we are here to help.
    I think I know what is causing this and we have a bug open to get this corrected. The calls are made for retrieving the Zettle status for the product page overview. This can be turned off but you won’t have the Zettle status showing if the products are properly synchronized to Zettle.
    You can do this in the product page overview under: Screen Option > PayPal Zettle Status (deactivate).
    This should avoid this phpflood from occurring.

    Let me know if this helps you further. I have linked this thread to the bug we have open for this, so you should get an update once this is corrected.

    Kind regards,
    Joost

    Thread Starter markisu72

    (@markisu72)

    4 years, 1 month ago

    Hi Joost,

    I know that issue with the product overview but I can confirm that this is different.

    The log excerpt I attached was from a test system, to which nobody except me has access and I only had the initial Zettle sync screen open – not the product overview.

    These calls occur during initial sync (and as I surprisingly observed yesterday on prod) also occasionally later.
    Also, it might be interesting to know, that we have not started to work with Zettle in the store yet. We’re just about to set it up. (on my test system there is no inventory movement anyways)

    Best
    Markus

    Plugin Support Syde Joost

    (@joostvandevijver)

    4 years, 1 month ago

    Hello @markisu72

    I asked the developers to take a look. They told me that multiple webhooks per second look strange (unless you are selling like crazy). There once was an issue where webhooks could be registered multiple times, but you will need to find out if there are multiple webhook receivers registered with the same URL.
    We believe there are measures implemented to prevent this, but maybe you found a loophole.

    Are there more logs to this that you can share with us to take a further look into this? If you don’t feel comfortable sharing those details on a public forum, please open a ticket with our service desk. You can find the instruction for this here: https://zettle.inpsyde.com/docs/request-support/
    (please add the URL of this thread in the ticket as well)

    Kind regards,
    Joost

    Thread Starter markisu72

    (@markisu72)

    4 years, 1 month ago

    Hi Joost,

    will do on Sunday or maybe beginning of the week with lots of logs.

    Best
    Markus

    Thread Starter markisu72

    (@markisu72)

    4 years, 1 month ago

    Done

    Plugin Support Syde Joost

    (@joostvandevijver)

    4 years, 1 month ago

    Hello @markisu72

    as informed in the service desk ticket, we have opened a bug for this issue and are pending approval to get developers to fix and release this solution. Since there is not much more we can do on this thread, I will mark it as resolved. Once the bug has been closed, we will update this thread as well.

    If you have any other questions or issues, please don’t hesitate to open a new thread.

    Kind regards,
    Joost

    Plugin Support Syde Joost

    (@joostvandevijver)

    4 years ago

    Hello @markisu72

    we have a beta version available that should reduce the PHP request significantly. If you want to test this version and give us some feedback on it, you can download it here.

    Kind regards,
    Joost

Viewing 8 replies - 1 through 8 (of 8 total)

The topic ‘Zettle Webhook calls killing server’ is closed to new replies.