Webhook MeetingEvents Are Broken

Description
The Zoom Webhook is failing to report nearly all meeting.participant_joined / meeting.participant_left events (possibly other event types too).

Error
No error, the webHook is just not sending data. We are seeing events for meeting.created, meeting.updated, meeting.started, meeting.ended, but very little data is coming through related to the actual join/leave events.

Sample MeetingID - 97037773408

The events are not coming through on either the WebSDK implementation of the native Zoom client. We noticed some new errors specifically on the WebSDK site that may be related - 404 errors to the following URLs are constant today.
https://rwcam.zoom.us/wc/ping/97037773408?ts=1589826095318&auth=V7STFFYUJ6qbPIMM3-UNnqMhSvypwgKiCrACZl7LHOk&rwcToken=V7STFFYUJ6qbPIMM3-UNnqMhSvypwgKiCrACZl7LHOk&dmz=1&_=1589826093120

Which App Type (OAuth / Chatbot / JWT / Webhook)?
WebHook

How To Reproduce (If applicable)
Steps to reproduce the behavior:

  1. Request URL / Headers (without credentials) / Body
  2. See error

Screenshots (If applicable)
image

Well, as of 12:20pm PST, we apparently started receiving a backlog of delayed meeting events. I can’t tell if it’s all caught up or not, but the notifications we were expecting this morning are coming through, just a few hours late.

We first noticed issues May 4th. We’ve been sampling webhook delays for meeting start/participant joined/participant left/meeting ended by comparing the time we received to the relevant timestamp in the event.

We’ve been seeing significant delays on most days during peak business hours. By significant I mean greater than a couple minutes but has gone up to 1 hour frequently.

Saw the largest delays today starting around 11:30 EST, with a peak around 3pm EST of 3.5 hours. We saw webhook delays return to normalcy around 5pm EST.

we suspect it’s related to dashboard “degraded performance” showing on status.zoom.us for over a week now.

see tommy’s reply in Looking for API Endpoint to toggle on/off webhooks

Thank you @taylor.hurt, we too are in the situation that receiving delayed webhooks is worse than not receiving them at all. I didn’t realize this had been going on for several weeks.

Hey @jmauck,

Apologies for the webhook delays. Currently we are seeing degraded performance for our reports and dashboard systems which are affecting some webhooks.

Our engineering team is working to fix this issue.

You can stay updated here: http://status.zoom.us/

Thanks,
Tommy