Uptick in recording.started webhook delays

meeting.started, participant.joined and recording.started are critical

1 Like

looks like we topped out around 30m delay for recording.started this past hour and we’re slowly trending back down, though unsurprisingly we see a spike at the top of every hour so we’ll see what happens here.

Confirmed across the board, the past 3 days have been getting progressively worse. with 30 minutes being the norm today.

Hey @BenS, @ryan, @jimig,

Thanks for the info! We are currently investigating the issue and will update you when we have fixed the webhook delays.

-Tommy

@tommy you’ve been saying that for months?

I am surprised this is not a pinned post. I would strongly advise any developer building an application depending on webhooks to reconsider their approach and not consider them reliable in terms of timeliness nor sequence. This has been an ongoing issue since March and only getting worse.

@tommy we need some commitment from Zoom that the webhooks are usable in the very near-term. Are you able to provide that before the weekend? If not, we will begin to re-architect our systems to not use webhooks on Monday. This has been one of the worst DX I’ve encountered and expected more from Zoom.

Hey @jimig,

I am sorry to hear you have not had the best experience with the Zoom webhooks.

We have just improved the webhook response times. Please let us know if you continue to see extended delays.

Thanks,
Tommy

1 Like

@tommy I have to share the sentiment a bit, I had to add heuristic logic to account for the 1 to 3 minutes delay and today weathered a functionality meltdown due to the 30 minutes. This issue voids the very promise of webhooks.

It would be helpful to hear more details about what’s going on on your side and what you are doing to address it. You alluded to a resources constraint earlier, is it deeper than that maybe? Do you have any sort of monitoring on this? It seems like it’s up to us to bark on this thread for Zoom to realize this is happening. Is the webhooks team paying attention? I know you guys deal with a lot, I’m hoping to stress the importance and our willingness to help.

Thank you.

Hi @tommy,

Hope you are well and sorry about that but we are all frustrated here. Our clients depend on this.

Anyway, I would like to let you know that during the past three days we started to receive more and more complaints about this. We had a call with 23 minutes delay. This is far more than critical. If this continues in the next week or two, I am afraid that we are gonna lose some of our clients that we won with hard work and determination. I don’t know if you guys understand how important this is for all the businesses who are working with Zoom.

Also, I have to agree here with @BenS. We need more information on what you are actually doing to locate, investigate and fix this issue. We need detailed information on what are you doing on your side and probably we will reconsider how our products are working with your integration.

We will all appreciate it if you can share detailed information for the steps that you took for the past FEW MONTHS since we are waiting for a fix from the beginning of the summer. Sorry about that but I am sure you can understand how disappointed we are.

Best,
Plamen

We saw no improvement on Friday to recording.* events.

@tommy I get the feeling there is no monitoring on your side of webhook delivery performance? This is wrong- you should not rely on developers telling you about issues, you should be aware and be more pro-activate as an organization.

Today we have started to remove all dependencies on recording events from our application. Whilst this changes our application functionality for the worst, it will provide a consistent user experience - a tradeoff we are willing to make.

I wish everyone the best with these hooks. Please pass this feedback on to your platform team- their failure to address this issue after 6 months is very disappointing.

Hey everyone!

I know this can be and is frustrating. However, this is our current top priority. I can’t give many details publicly, but just know that we are working hard to strengthen all aspects of our platform and will make sure we deliver web hooks fast in the future.

We’ll provide more concrete updates as we have them, but for now, just know we are working very hard to make sure we provide you with the reliability you deserve.

2 Likes

Tim, that’s fair, thank you for coming to Tommy’s rescue and letting us know there’s movement :).

Do you have any sense of a timeline by any chance? Do you guys have the issue well qualified at this point? Is it helpful when we bring up a peak here?

Today is another bad day on this front by the way :). But hopefully this isn’t news at this point.

Anyway, good luck getting to the bottom of it and please do reach out if we can help. We obviously all have a vested interest in seeing this resolved :). Take care.

Hard to nail down exactly because there are so many moving parts and they all get improved incrementally, but it won’t last long.

Absolutely. Marching towards reliability as fast as we can.

Examples are always helpful but we have good enough metrics that we know when it’s happening.


Thank you so much for being understanding! We’re here to support you and will make sure this gets resolved!

1 Like

Reporting in that things appear to be much more stable in the last couple weeks. We appreciate the effort that went in to improving stability here. Thanks!

1 Like

Hey @ryan,

Thanks for the shoutout—and thanks for your help in raising the priority on this. We’re glad you’re seeing improvements already! :100:

Best,
Will

1 Like

@will.zoom whilst we’ve removed dependencies on webhooks our monitoring showed awful delays.

We’d love to use them someday and are following - here is what we saw:

I read you’ve made improvements, is there any chance the developer community can be more involved in the issues so we can manage customer expectations? Over a 3h delay makes the hooks useless for most applications, we may as well just poll the APIs.

Yes, I should have knocked on wood before my last post :joy:. But, we are very grateful for the update to https://status.zoom.us to include webhooks issues. Today’s incident report was very helpful in helping us communicate issues to our customers.

Hey @ryan @jimig,

Our apologies for the delays—we had an incident yesterday that has since been resolved. That said, we’re working on taking your feedback to heart and have added webhook delays to our status page, as you noted.

As always, thank you for your diligence in reporting any issues to us—we’re working hard to continue improving this experience.

Thanks,
Will