API Endpoint(s) and/or Zoom API Event(s)
recording.completed Webhook Event
Description
I am trying to update our verification flow to use the new “verify with zooms header” verification flow described in the link below. My endpoint is receiving the recording.completed webhook as expected, and currently works with the verification token flow.
However, I am in the process of updating this to using the new verification flow, but the required headers (x-zm-signature) is missing. In my marketplace app settings, I have enabled a value for Secret Token and Verification Token (Retires in October 2023).
Any ideas why this is and how I can fix this? https://developers.zoom.us/docs/api/rest/webhook-reference/#verify-with-zooms-header
Error?
There’s internal logging that gets triggered when this expected header is missing.
Hi Gianni, Thank you for the response, apologies for the late reply.
I ended up changing this verification flow to use the custom header flow because we were facing a couple other issues - however, I’m noticing this new custom header is now absent.
Marketplace settings
In the zoom marketplace app settings, I have set the configuration to send custom headers and clicked ‘saved’. When I refresh the page, these changes are persisted. Our new custom header is using the standard x-prefix custom header (ie To confirm, is this sufficient, or are there further steps required for these headers to be set (eg republishing the app or something?)? Screenshot attached for where in the UI this was updated with placeholder values for screenshot only.
Our Application
We have an internal upstream system that has an allow-list of headers and strips out unallowed ones. I have updated this system, and with postman, verified this new header reaches our application (I copied a real recording.completed webhook and set this new custom header). I am not really comfortable sending our webhooks to an external tool, because these webhooks are for recording.completed, and contains PII / is used in prod (we’re just updating the verification flow on an existing production flow). Would it be possible to verify on Zoom’s end that these headers are being sent? Sample screenshot of real webhook with the zoom trace ID attached (note the expected new custom header is absent).
Sample screenshot of the place in the marketplace App UI that was changed to set the custom header (the actual header/value are different).
Thank you for the quick response! I updated the development URL (I ran this app locally with ngrok) but I’ m noticing that webhooks don’t seem to be sent to this development URL. This leads me to wonder if the changes aren’t being applied even though they are saved in the zoom marketplace UI?
In the documentation I noticed it mentions Zoom requires Webhook validation for all webhooks created or modified after Oct 23, 2022. All webhooks created prior to this date do not require validation unless you modify them.. I noticed the endpoint in the marketplace UI is marked as ‘validated’ but I’m noticing the production endpoint doesn’t actually send the validation response. I’m fairly certain this webhook event has been in production since 2021 (ie before Oct 2022).
Would this prevent the changes from being applied correctly? i.e. If the endpoint does not return the expected webhook validation response body, do the change saved in the marketplace manager UI not get applied to the sent webooks? Does zoom have some intermediate state between the changes being saved, and the changes being applied that isn’t visible via the UI?
My main source of confusion here is that when I save the changes in the marketplace app manager UI, it shows a success message, there’s no errors shown, and the url shows a ‘validated’ checkmark (which is always there). When I refresh the page, the changes persist, which suggests everything saved successfully. However, the changes don’t actually seem to be applied. If I’m understanding this correctly, should I be receiving some sort of error message instead of a successful saved response?
These are all really good points. See my responses below.
Does this mean you did not do the process I suggested? You just updated the development URL? Nonetheless, the url update should be sufficient to count as an update to the event itself:
The changes are expected to apply to webhook events added before October 2022 if the event is updated:
That’s correct, I did not remove the event - I was concerned that there could be issues in re-adding the event, which would cause issues for our production users. I tried the testing step you suggested with the sample app.
I suspect what might be going on is related to that message you screenshotted - “All webhooks created prior to this date do not require validation unless you modify them”. My endpoint doesn’t send a validation response (it is shown as ‘validated’ but I suspect this status is just shown because it was grandfathered in). I’m wondering if I need to implement this validation step , before any of these other changes (eg updating the dev URL, updating the custom header sent, etc) can be applied.
Either way, i’ll take a look in my inbox, thank you!!
They don’t observe the header being sent and it’s suspected that this is likely due to this being a legacy app (pre-verification).
I shared that despite it being a legacy app, the documentation relays the impression that if the events are updated, it will be responsive to the new verification/validation process. I’m waiting for clarification on this part.
Additionally, I want to confirm, all testing with the webhooks is being done with the development url? You haven’t done any with production and re-published the app?
Hi @mokutsu , the latest recommendation from service engineering is one I made earlier: to deactivate and re-activate your app before running through the verification flow again.
Hi Gianni,
Happy new year, thanks so much for looking into this.
I don’t see a button to deactivate the app in the marketplace UI nor in the documnetation, would it be possible to get more information on how to deactivate and reactivate my app? Do I need to go through the entire app approval process again - Is there a way this can be done without disabling the app / minimal downtime for our customers?
if the changes aren’t being applied because this is a legacy app, do I need to update the app to use the newer verification method at all (ie will the legacy support be sustained or is this also being deprecated)?
I forgot to respond to this, I have been clicking “save” in the marketplace app manager UI. Does this not get applied??
Unfortunately, our development environment is reachable via a cookie, not a URL, so the zoom developer URL doesn’t work for our setup. The development URL was set to the same URL as production by whomever setup the application in zoom, so I have left this as is. For context, I have tried setting up a local development server exposed with ngrok that receives the webhook and attaches the required testing environment cookie and forwards it to the prod URL, but this gets rejected by some internal application + routing layers we have that sit in front of our actual application that handles the zoom webhook.
I have therefore been testing changes in prod with extra logging and exception handling and postman.
If you deactivate your app, you do risk downtime for your customers. To deactivate, you click the ‘Activate’ tab and un-install the app. The app would need to be approved again to be published, but I’m going to reach out to the marketplace team to get more input. I do not advise doing this at this time.
Additionally, legacy support is sustained and you do not have to make further changes for verification.
Ok, sorry just so I’m understanding this correctly, are the changes to the Verification Headers not applicable to legacy applications? Ie are the changes I’m making to my legacy application unnecessary?
I noticed the warning in the documentation for validations indicates an exception is made for legacy apps, but I see no similar warning for verification header changes.
Based on that, my understanding was that:
all apps, legacy and non-legacy, need to migrate to using the new webhook verification flow (via x-zm-signature or a new custom header)
new apps and apps being modified need to implement the webhook endpoint validation response
in order for new changes to be applied (ie changing the webhook verification header), the endpoint validation needs to be implemented, per point 2. As a result, even if I have a legacy application, I need to implement the new header + the validation response by February, 2024. In addition to these changes, I may need to deactivate/reactivate my app (hold off on this for now, until I hear back).
Is this understanding correct?
Screenshots of the warnings I’m referencing are below.
Thank you so much!
Misato
No worries at all. This is an interesting and helpful case for all of us to learn. Re-reading the conversation with our service engineering team, it’s not that the changes are not applicable or unnecessary to legacy applications per se, but that this is likely the reason for the behavior and persisting validation we’ve observed with your app.
If that doesn’t work, app deactivation may be required, but I will work with you and our marketplace team to resolve. I’m seeking their counsel about that scenario right now.
Hi Gianni, thanks so much for the response! So we were hoping to use the custom header verification because there’s some adiditonal issues with the x-zm-signature heading method. Is there a way this can be done without switching to the x-zm-signature heading?
This was because we noticed there’s a mismatch between the documentation for the body between the recording.completed webhook and the one in the webhook verification documentation. The webhook verification documentation includes event, payload, and event_ts. However, the recording.completed webhook contains an additional download_token field. My concern here is that if we deserialize the webhook in order to omit this download_token field, we can’t guarantee the order of the JSON key-value pairs, so when it’s converted back into a string to construct the signature, it may cause issues.
If we use the x-zm-signature header, should I be using the webhook body as-is when constructing the signature (ie including the extra download_token field)?
@mokutsu , after discussing internally , we’ve clarified that verification is optional, but encouraged. The documentation will be updated to address this confusion (DEVELOPERS-4436). If you would still like to complete the verification process, please re-generate the secret token and attempt with custom header again.
Additionally, I was told webhook updates do not constitute marketplace submission review unless you were increasing data access (new scope), but waiting for more detail about if you deactivated and re-activated your app.
Validation of the endpoint is mandatory and I know it says in the app UI that the endpoint is already validated, but we can investigate this part of the behavior you observed as well:
