I want to integrate the Zoom app into my React website, but I need to customize it to display the real-time heartbeat of all participants in an ongoing Zoom meeting. Specifically, I want to detect each participant’s face and display their heartbeat on their face during the meeting. However, I’ve read that I can’t access the raw video feed from Zoom through the Web SDK for this type of task.
Can anyone help me figure out how to handle this issue?
This sounds like a really interesting concept.
I have previously used face-api.js to determine if a guest is too close/far from their camera and if they need to move left or right, or move higher or lower, to be perfectly centered for a call; I had to make some custom CSS stuff to tie into that and output to canvas over the webrtc embed. If you knew in advance who the participants were and had good photos for face detection to work, it’d probably be pretty easy to correlate the heartbeat data to the right person and size/position the overlay properly.
Following up: If you’d need that overlay to sent back to all the participants, it’d take some looping back of your app’s render back into your camera input. Not impossible but not doable purely within react (I’m thinking your app would be open and being screen captured by something like OBS, then your Zoom would use OBS as the camera; participants would have to pin you for it to work right, probably).
We have an official sample app built with React on GitHub
Regarding the raw data requirement, we currently provide APIs to manipulate the captured video stream. Support for raw data in the receiving stream will be available in a future release.
Thanks for the response, Vic! I’m currently using the same video SDK you suggested. Could you please clarify which API you’re referring to for manipulating the captured video stream? Also, I would like to know if it’s possible to send the processed video (with the heartbeat display on participants’ faces) back to Zoom for playback during the meeting.
If this is feasible, could you guide me on how to achieve this and which specific APIs I would need to use?