Meeting.participant_* webhooks have stopped

Description
Starting around 9 AM PDT this morning, we stopped receiving almost all meeting.participant_* webhooks. We are still receiving other meeting webhooks.

Error
No meeting.participant_* webhooks.

Which App Type (OAuth / Chatbot / JWT / Webhook)?
OAuth app client ID YrJ5gQkpSVikdYkHxwoACw

As of 11:21 PDT, the webhooks have started again.

Actually it seems this is still an issue. Some webhooks are coming through but seemingly quite delayed.

For example, for meeting Ig/rO+CrQBqlj77CDb5qFA== which started at 2020-03-12 17:00:34 and ended at 2020-03-12 17:13:55 UTC, we got meeting.participant_joined webhooks starting at 2020-03-12 18:21:53 UTC.

Hey @ryan,

We will look into this.

Can you confirm you are sending a 200 OK response back after receiving each webhook?

Thanks,
Tommy

Confirmed, for example:

Mar 12 11:21:53 grain-staging heroku/router at=info method=POST path="/_/zoom_webhook" host=example.grain.co request_id=91a63b89-8120-4fc0-b07a-9f8091574034 fwd="3.211.241.119" dyno=web.1 connect=0ms service=27ms status=200 bytes=308 protocol=https
Mar 12 11:21:53 grain-staging app/web.1 18:21:53.083 request_id=91a63b89-8120-4fc0-b07a-9f8091574034 [info] POST /_/zoom_webhook
Mar 12 11:21:53 grain-staging app/web.1 18:21:53.083 request_id=91a63b89-8120-4fc0-b07a-9f8091574034 [info] Received "meeting.participant_joined" for meeting uuid: Ig/rO+CrQBqlj77CDb5qFA==
Mar 12 11:21:53 grain-staging app/web.1 18:21:53.107 request_id=91a63b89-8120-4fc0-b07a-9f8091574034 [info] Sent 200 in 24ms

Hey @ryan,

Thanks for confirming. We are looking into the issue and will get back to you. (ZOOM-143828)

Thanks,
Tommy

Still seeing ~40 minute delays to participant webhooks, e.g.:

Mar 12 14:06:10 grain-next app/web.1:  21:06:10.381 request_id=aa3c9025-5617-4b50-8101-334ba186a073 [info] Received "meeting.started" for meeting: HebCR/hPSeqOZzc8MUXQvg==
Mar 12 15:02:55 grain-next app/web.1:  22:02:55.249 request_id=8045270a-d0c1-4678-aab9-7a9682e1a45e [info] Received "meeting.ended" for meeting: HebCR/hPSeqOZzc8MUXQvg==
Mar 12 15:23:55 grain-next app/web.1:  22:23:55.575 request_id=e1186307-d71f-4fac-9d4f-43e294cbb5ce [info] Received "meeting.participant_joined" for meeting uuid: HebCR/hPSeqOZzc8MUXQvg==
Mar 12 15:24:20 grain-next app/web.1:  22:24:20.732 request_id=1b0b1c9e-aceb-4af8-b5f8-fc36c3a2aa33 [info] Received "meeting.participant_joined" for meeting uuid: HebCR/hPSeqOZzc8MUXQvg==
Mar 12 15:24:38 grain-next app/web.1:  22:24:37.830 request_id=8c10b4e9-3729-4877-8b08-18c7dc2c483e [info] Received "meeting.participant_left" for meeting uuid: HebCR/hPSeqOZzc8MUXQvg==

The event received at 22:23:55 occurred at 21:45:11 per the payload.

Thanks, for the additional info @ryan, we are looking into this.

-Tommy

It appears the large delay in meeting.participant_* webhooks was resolved yesterday ~23:00 UTC and they continue to be timely as of this morning.

I’ll update this thread if we see this issue start popping up again. I’ll also use this opportunity to make another vote for an API/Platform status page.

1 Like

Thanks for the update @ryan.

Yes, please let us know if you see your webhooks getting delayed again.

And yes, we are working on an API status page :slight_smile:

Thanks,
Tommy

Hey Tommy! Just wanted to jump in and say we are still seeing sporadic issues with meeting.participant_* webhooks, they are sometimes incredibly reliable an other times not sent for hours.

We have recently observed that meeting.ended webhooks are now trailing in much later than the actual ended time.

Hey @tommy just checking in, its been two hours since I posted, and we still have yet to see any response. If there is an issue please inform us so we can take precautionary measures and inform users that they will be facing a degraded experience when using our app.

Hey @matt,

The devforum is very busy today, so I am still getting to everyones questions. Typically we have a 24 hour SLA on weekdays, so you can expect your post to be replied to in that timeframe.

We did have downgraded performance on our meeting.participant_* webhooks, last week due to the significant increase in Zoom usage, and have since increased the number of servers and are monitoring closely. I suggest you also monitor closely, and inform your users.

Thanks,
Tommy

Thanks for the updates @Tommy, we appreciate the work everyone is doing under the increased load we’re all experiencing. We’ll update this thread if there are any new significant changes on our end.

1 Like

Happy to help @ryan! :slight_smile:

Thanks, please do update us!

-Tommy

We’re still getting those webhooks delayed (sometimes no webhook for join/left events) which is breaking our business and we’re loosing money where we trusted zoom and constructed whole meeting system on zoom SDK. I guess we need to take a look to other alternatives because it’s not acceptable to not getting any solution in a week!

Hey @hardc0der,

We are aware of some webhook performance degrading due to the high volume of traffic we are experiencing. Our engineering team has been actively working on the resolution.

Stay updated here:

Thanks for your patience,
Tommy