How to get video data from custom video layout frame by frame

[PB_UV (Peter)]

Hello,

I need to get the video from a custom participant composition into my Direct3D world. I know how to display an ICustomizedVideoContainer into a GDI+ Window but how can I access the video data directly. Maybe as a memory stream or some similar to do some processings on it.

I don’t need the raw data from each single participant - the composition from a video container is more than enough.

Thanks!

[amanda-recallai]

@PB_UV, to get the video data from a Zoom meeting you can either use the Zoom RTMP live streaming or use the Zoom windows raw data SDK to build a bot.

If you’re using the RTMP output, you will get a mixed video containing multiple participants. However if you’re using the Zoom Windows SDK raw data feature you will receive separate raw video frames for each participant, and you will need to merge them yourself into a combined stream. I don’t think there is a way to get the mixed video streams directly from an ICustomizedVideoContainer.

If you’re using the Zoom Raw Data SDK, these are the steps:

1. Spin up a server. We recommend AWS, GCP, or Digital Ocean.
2. Use either the Windows or Mac Zoom SDK to launch an instance of the Zoom client.
3. Once you have the Zoom SDK launched, and use the Raw Data functionality to extract the video and audio streams.
4. This will return the video in I420 raw frames and audio in PCM 16LE raw format, so you’ll need to encode the audio and video yourself afterwards.
5. Once you have one instance of this working, you’ll need to scale this across several servers if you want to run multiple bots simultaneously, which is required to have bots for multiple meetings.

Finally, another option is Recall.ai. It’s a simple 3rd party API that lets you use meeting bots to get raw audio/video from meetings without you needing to spend months to build, scale and maintain these bots.

Let me know if you have any questions!