Description
We have an app in the marketplace and for some events, but not all of them, we receive duplicate events.
Error
It’s a duplicate event.
Which App?
Zapier
How To Reproduce (If applicable)
Steps to reproduce the behavior:
Install App
Create a New Recording for one account, or add a new Webinar Registrant for another
Note that you get two events.
Screenshots (If applicable)
If applicable, add screenshots to help explain your problem.
Additional context
The events come through with the same authorization header, so they’re clearly coming from the same app
The events all go to the same URL, so I know they’re duplicate production (not staging/dev) events
The events have the same X-Zm-Trackingid header, but they have different X-Request-Id headers
I have opened ticket 11140441 on this, but they said they couldn’t help me.
There is also an internal Jira ZOOM-277107 on this, but their last suggestion, to publish the app, doesn’t make sense since (i) it’s already in a Published state, not Draft, and (ii) this isn’t happening for all events, just some, which makes me not believe it has anything to do with the app’s state
Double check you are sending a 200 OK response after receiving the webhook event. If we do not receive this, then we will continue to send the event as we assume the request failed.
Yes, we send 200 every time, and I checked on these duplicate requests as well. Furthermore, the events come in with 100 ms of each other and only ever come once or twice (never more), so I don’t think it’s that we’re not sending a 200 back. These seem to be duplicate events
Follow-up question, @tommy … if we were to try to roll our own solution to this problem, can you confirm that we can use the X-Zm-Trackingid to de-duplicate? I’ve only done a little bit of sampling of the data, but it appears to be unique except in the case of these duplicate events. Can you please confirm that would work? Is there documentation on that header somewhere?
UPDATE: This page says of X-Zm-Trackingid, “Unique identifier that Zoom uses to identify the request.” so that seems like a pretty good thing to use for de-duplication if we can get that going. (Would still love to see Zoom fix the issue tho)
If you’re receiving the duplicate events within milliseconds of each other, can you please double check that you don’t have another event subscription under this or another app under your account?
A few places to look:
Confirm that you’re not receiving development and production events at the same URL (only applicable for an OAuth app)
Double check that you don’t have another webhook, jwt, or oauth app that is subscribed to the same event
That’s correct, we have neither of this things happening. We know it’s from the same Marketplace app since they come through with an identical verification token. And we only have the one published app anyway.
And our development and production events come to another URL.
Yes. Every field is exactly the same, except for X-Request-Id.
A week or so ago I was on a very long call with developers on the Marketplace team from China. They said they had diagnosed the issue as something being different between the draft app and the published app. This was concerning for a couple of reasons: First, that I couldn’t see the settings for the published app. Second, that the way they said the published app was set up did not match what I had ever published in my memory.
Finally, they assured me that if I published the app as they were seeing it on my screen that the issue would stop. Unfortunately, they were wrong and it continues.
We have a half-fix in place on our end to detect duplicates and only act on the first one, but unfortunately if they come in too close to each other even that fix will not work. So it’s still something we’d really love for the team to fix.
I can also tell you that the Jira ticket internally with the Marketplace team for this is ZOOM-277107. I also have a request with T2 (#11140441) that is currently with Graeme who is trying to schedule another call with me (though I’m not sure what additional information could be needed.)
Thank you for sharing these details, and my apologies, as I did not realize this was actively being investigated by our team already.
In taking a look, I can see that this investigation is still underway internally and our Engineering team is working with the agent in the ticket you’ve referenced. Given this, I will let them continue with communications there, and they should have an update for you shortly.