Webinar.participant_joined webhooks are sometimes missing

API and Webhooks
1

Description
Although the participants actually joined the webinars, I have been successfully receiving the webinar.participant_joined webhook most of the time, and the webinars’ attendee reports shows join time, webinar.participant_joined webhook are sometimes missing.
I have looked into my server logs and also the Webhook Only Logs in App Marketplace, but there were not a log of webinar.participant_joined for the specific webinars. (possibly webinar.participant_joined logs of other webinars are partially missing too. )

Error
webinar.participant_joined webhook are sometimes missing

Which App Type (OAuth / Chatbot / JWT / Webhook)?
Webhook Only App

Which Endpoint/s?
webinar.participant_joined

Infomation for research
Account number: 54706753
One of the webinars ID: ***********
Time when the webhooks were supposed to be sent: Apr 19, 2020 15:23 ~ 16:00 JST

Screenshots (If applicable)
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

*This post has been edited to remove any meeting / webinar IDs

1 Like
2

Hey @yuya.uwatoko,

We will look into this! (ZOOM-154736)

Thanks,
Tommy

3

Hi @tommy,

Thanks for the response.

Please look into this.
We and our clients are facing this problem and are in trouble.

4

Hey @yuya.uwatoko,

Engineering is looking into this to find the root cause. I will keep you updated.

Thanks,
Tommy

5

Any updates on this?
It would help me a lot if i could know if it’s actually an issue on the Webhook or not.

6

Hey @yuya.uwatoko,

Looks like it is an issue on our end. Are you still seeing the webhooks not send?

Just to make sure, send a 200 ok response back each time after you receive a webhook.

Thanks,
Tommy

7

@tommy,

I believe I am having the same issue with meeting.participant_joined events. My OAuth app relies on timely webhook events. I’m seeing webhook delays of as little as 3.5 minutes and as large as 30 minutes.

Here is an example webhook:

MeetingID: ***********
WebhookId: Webhook_5f7c7e36c7784ac18d1a9dc2d5f4663c

The eventual webhook body says that the event time was “2020-04-29T17:14:37Z”. I did not receive the webhook until “2020-04-29T17:17:23Z”.

*This post has been edited to remove any meeting / webinar IDs

1 Like
8

Hi @tommy,

Thanks for your reply.
I’m pretty sure we are sending back 200 response, but I’m not sure if we are still missing some webhooks.
I’ll need to compare all the logs in our side and your side to see if there are missing webhooks.
Can you confirm if it’s working fine or not?

Also I have posted related but another critical issue here https://devforum.zoom.us/t/webinar-webhooks-are-sent-as-meeting-events/15172/3.
Please reply to this as soon as possible.

Thanks,
Yuya

9

Hey @yuya.uwatoko, @recordedspeed,

Are you still seeing the webhook delays?

Thanks,
Tommy

10

@tommy

I’m still seeing delays for my webhook events and I’m able to get back to you now with more information.

I added logging a few days ago to track the severity and occurrences of meeting.participant_joined event delays. I measure the delay as the difference between participant object’s “join_time” and the time on my server (both UTC). Below are the results.

This graph shows the maximum (purple line) and average (red line) delay in seconds per timebucket (1 hour) over the last 2 days.
The x-axis time is in Pacific Time. Even just over 2 days I’m starting to see a pattern: the delays occur between 6am and 10am PT. The largest delay is more than 45 minutes.

Note: I don’t log when the time difference is less than 10 seconds. This is why there are gaps in the data.
These measurements are for thousands of events per hour.

Screen Shot 2020-05-08 at 12.53.54 AM
Screen Shot 2020-05-08 at 12.53.54 AM2698×1588 325 KB

Please let me know if you need any more information from me. I can try to collect any additional information you need to help resolve the issue.

1 Like
11

@tommy

I continue to see the same pattern M-F daily of delays up to 45 minutes. The errors occur 6AM-10AM PT every day and include weekends as well (although on weekends the delays are usually in the 100 second range)

@shrijana.g I’m tagging you as well to get your thoughts on the matter. From what I can tell, you are a domain authority on Zoom webhooks.
referencing your medium post: https://medium.com/zoom-developer-blog/automate-workflows-with-zoom-webhooks-334600f8c99

Let me know if you need more information from me.

12

I can provide you with an example webhook callback from today which was delayed 2,458 seconds

My server received this webhook on: 2020-05-11 14:49:04.087 UTC

{
  "payload":{
    "account_id":"XXXXXXX",
    "object":{
      "uuid":"mTUeW0xBRN+kTfUgLR4u+A==",
      "participant":{
        "user_id":"XXXXXXX",
        "user_name":"XXXXXXX",
        "id":"XXXXXXX",
        "join_time":"2020-05-11T14:08:06Z"
      },
      "id":"XXXXXXXXX",
      "type":2,
      "topic":"Zoom Meeting",
      "host_id":"XXXXXXX",
      "duration":60,
      "start_time":"2020-05-11T14:09:09Z",
      "timezone":"America/Chicago"
    }
  },
  "event":"meeting.participant_joined"
}
13

Hey @recordedspeed,

We are currently experiencing downgraded performance on our Report and Dashboard systems, which is causing the webhook delays.

Stay updated here: http://status.zoom.us/

Apologies for the inconvenience,
Tommy