Meeting Participant Join / Left events and multiple deliveries

I’m building an online classroom and I’d like to check class attendance via Zoom Webhooks.

Everything works quite smoothly with the meeting.participant_joined and meeting.participant_left events, but I ran into the following edge case:

In chronological order:

  • User A Joins - Webhooks gets delivered correctly, I mark A as “joined”
  • User A Leaves - Webhooks get delivered but my app is slow and does not return within 3 seconds. On my side the action is executed correctly and I mark A as “left”
  • User A Joins again - Webhhoks get delivered, I mark A as “joined” again
  • Zoom redelivers the User A meeting.participant_left, I mark A as “left” but in fact she is still on the conference

At this point I need to detect that the last delivery is a duplicate and discard it.

Few options I thought about:

  • Dedupe using the x-zm-trackingid. The documentation say that it is the “Unique identifier that Zoom uses to identify the request.” but I’m not sure if that would be the same across different delivery attempts. Anyone knows?
  • Look at the “left_time” value in the payload and do not mark A as “left” since the “join_time” is more recent
  • Dedupe based on event_ts, though it could be error prone under heavy load

Anyone else dealing with the same problem? How did you solve it?

Hi @startupkid ,

Happy to look into this. Can you tell me a little bit more about your developer environment (i.e. stack you’re using) so I have more context for what you’re seeing with the webhooks?

Perhaps a code sample would be helpful as well.

Gianni

Hi Gianni,

I’m using a Rails application to receive the webhooks, I have a specific zoom controller with a Callback method that executes some actions when a web hook come in.

Everything works very well, the only issue I’m having is with deduping and sequencing events, as I tried to describe above.

Here is some code to handle the “participant_joined” event, not sure how it helps though

  def callback
    # Log the event
    z = ZoomEvent.create(
      source: 'webhook',
      name: params[:event],
      ts: params[:ts],
      payload: params.to_yaml
    )

    case params[:event]
    when 'meeting.participant_joined'
      a = nil
      s = Schedule.find_by(zoom_meeting: params[:payload][:object][:id])
      zri = params[:payload][:object][:participant][:registrant_id]
      ze = params[:payload][:object][:participant][:email]
      if zri
        # First, registrant_id lookup
        a = Attendance.find_by(zoom_registrant_id: zri, schedule: s)
      elsif ze
        # Second, email lookup
        a = Attendance.find_by(email: ze, schedule: s)
      end
      if a.nil?
        # If everything fails, create new attendance record with the data we have
        u = User.find_by(email: ze)
        a = Attendance.new(user: u, schedule: s, email: ze, zoom_registrant_id: zri, duration: 0)
      end
      a.last_event = 'meeting.participant_joined'
      a.join_time = DateTime.now
      a.save
    end

    respond_to do |format|
        format.html { render plain: 'OK' }
    end

  end

Hi @startupkid ,

Thank you for this. I asked because I’ve seen environment specific behavior.

Regarding the x-zm-tracking id, it is in fact unique and can be used to dedupe here. I also like option two (looking at the “left_time” and “join_time” values.

Let me know how it goes.

Gianni

Thanks Gianni,

unfortunately there seems to be other issues.

Here is what I see in the logs for this specific user:

meeting.participant_joined @ join_time = 2021-07-13T22:00:23Z
meeting.participant_joined @ join_time = 2021-07-13T22:54:48Z
meeting.participant_left @ left_time = 2021-07-13T22:54:56Z

I joined the meeting and the user was currently in the meeting

What could be the reason of the second join event coming before the third left event?
Could there be a glitch in the time coming from the API?

Thanks!

Hi @startupkid ,

I don’t fully understand, could you please address these questions:

Did you intend for you or the user to leave the meeting? Are you joining as the user? What behavior did you see vs. what you’re expecting?

Thank you!

Hi Gianni,

thanks for your help!

Unfortunately I’m just observing our live usage logs and I don’t really know what the user did.

Anyways with the goal of saving us both a lot of time, I did some digging and I have a good and a bad news, better detailed as follows:

Tracking multiple delivery attempts

This is the bad news; zm_trackingid cannot be used to track delivery attempts of the same web hook, here is what I see on my side

Webhook request 1) received at 2021-07-14 22:21:24 with X-Zm-Trackingid = Webhook_0f9fdb3d5c2d4ea0b5b422d3d9f3c32e

  payload: &1
    account_id: [CENSORED]
    object:
      uuid: UFl5dhENRIaQEw6yvhNihw==
      participant: 
        leave_time: '2021-07-14T22:21:22Z'
        user_id: '33555456'
        user_name: [CENSORED]
        registrant_id: lyF4h08rRuq8APkkzh5n1Q
        id: lyF4h08rRuq8APkkzh5n1Q
        email: [CENSORED]
      id: '97711976001'
      type: 2
      topic: Week 1 - Day 3 - 6pm to 9pm
      host_id: [CENSORED]
      duration: 180
      start_time: '2021-07-14T21:53:42Z'
      timezone: America/Los_Angeles
  event_ts: 1626301284427
  event: meeting.participant_left

Webhook request 2) received at 2021-07-14T22:26:31 (5 minutes after the previous one) with X-Zm-Trackingid = Webhook_4f937d099f2c4b278b12d127ccfa2341

  payload: &1
    account_id: [CENSORED]
    object: 
      uuid: UFl5dhENRIaQEw6yvhNihw==
      participant: 
        leave_time: '2021-07-14T22:21:22Z'
        user_id: '33555456'
        user_name: [CENSORED]
        registrant_id: lyF4h08rRuq8APkkzh5n1Q
        id: lyF4h08rRuq8APkkzh5n1Q
        email: [CENSORED]
      id: '97711976001'
      type: 2
      topic: Week 1 - Day 3 - 6pm to 9pm
      host_id: [CENSORED]
      duration: 180
      start_time: '2021-07-14T21:53:42Z'
      timezone: America/Los_Angeles
  event_ts: 1626301284427
  event: meeting.participant_left

As you can see the same payload was delivered twice, probably because our server was under load and was not able to reply within 3 seconds.
The second delivery attempt has a different X-zm-trackingid though, which prevents us from identifying it as a re-delivery.

One option is to dedupe based on the event_ts, and live with the fact that it’s very unlikely that there will be two different events with the same millisecond-precise timestamp. I would be happy though if there was a more clean way. Thoughts?

Joining from multiple devices
I figured the reason of the double join event in my second message, the user had two devices in the meeting and the web hook meeting API is generating a new event for each.

The user_id field in the payload can be used to identify that, which is great

The only issue that is open now is how to safely identify multiple delivery attempts.

Thanks for your help

Hey @startupkid,

I’m happy to help out here! First, when it comes to deduplicating these events I think we’ll want to start with ensuring that the endpoint is able to respond to these requests in a timely manner in most cases. This might involve isolating resources, increasing the resources available to the server or adjusting app logic.

Realistically, responding with a 200 OK status should take the server milliseconds which is why our 3 second timeout is ample in many cases. If you’re performing operations on the data that take more than three seconds, I recommend validating the data and then sending a 200 OK status immediately -before starting your operation.

I bring this up because I think the deduplicating the events is ultimately a bandaid on the wound created by slow server response times. If you’re able to provide more information on why load is so high on your server I might be able to offer some tips.

However, if that doesn’t help and you must manually deduplicate events, then you’ll want to use a combination of event_ts along with the participant.id and meeting UUID.

While under load you may see events come in with the same event_ts, you won’t see an event where one user joined a meeting twice within milliseconds so it’s safe to say that if an event has a matching event_ts, UUID and participant.id as a previous event it’s a duplication.

I hope that helps! Let me know if you have any questions.

Thanks,
Max

1 Like

Thanks @MaxM!

I agree with you that the main focus should be in making the endpoint response time as fast as possible and we are working on returning the 200 OK right after data validation and queue the callback for asynchronous processing.

Our app is a Rails app hosted on Heroku for now, and unfortunately sometimes we have seen stalling requests, not necessarily tied to extreme load. We are working on having a better understanding of the cause of this stalls that occurs application-wide, not only in the zoom callback code.

Thanks for the comments on deduping, hopefully we will manage to make > 3 secs response times edge cases, but deduping will still be useful as a safety net.

Our ultimate goal is to make a pseudo-realtime attendance tracking system that keeps track of participants joining and notifies them in case of delay. Also we need to make sure that participants are attending for at least a 90% of the meeting duration. Tracking attendance via the joined / left event seemed the easiest thing to do, but we are not having challenges when users join from multiple devices, etc…

Any other API calls that you think would suit us better? I’m looking into the dashboard API and past participants API, but any recommendation would be really appreciated!

Thanks for your help!

Hey @startupkid,

Glad Max could help out!

In regards to API alternatives, I should note that our Dashboard/Report APIs should offer much of the same info for participants. If you’d like to take this approach, I’d recommend checking out these endpoints:

Thanks!
Will