How to get image Data from recipient video canvas?

I am using zoom web sdk for video conferencing feature. i am able do video call with sample-app-web.
Now i want to integration a third-party AI tool to analyse user emotions during call. For this, i need access to recipient(peer2) video frames(image data of canvas) during video conferencing. which is required as input to AI tool.

I am able query Peer1(canvas[id=“suspension-my-canvas”]) to get the image Data. but not able to get image data from peer2 in the same way. it seems canvas context type is different for peer1 and peer2. Please see the attach screenshot for reference.

AI tool : https://knowledge.affectiva.com/v3.2/docs/analyze-a-video-frame-stream-4

To Reproduce(If applicable)
Please refer to screenshot

Screenshots

Anyone could help me to get the image data of Peer2 either from video tag or canvas.

Hey @sandeepv, thanks for posting and using Zoom!

I was able to get the canvas for the attendee with canvas[id=sv-active-video]

Let me know if that helps!

Thanks,
Tommy

Thanks @tommy, I can also get the canvas by querying the DOM but i am not able to get the canvas[‘sv-active-video’] context. it returns null as below

const myCanvas = document.getElementById('sv-active-video');
const context = myCanvas.getContext('2d')  //context value is null now. 

but with peer1 canvas[id=“suspension-my-canvas”], i can get the context and able to get the image data out of it.

const myCanvas = document.getElementById('suspension-my-canvas');
const context = myCanvas.getContext('2d')  

//console.log context output

context.getImageData(0, 0, context.canvas.width, context.canvas.height);

Hey @sandeepv,

Seems this is not Zoom specific. I am guessing you are requesting document.getElementById('sv-active-video').getContext('2d') before the page has loaded. Have you tried googling this issue?

Thanks,
Tommy

1 Like

@tommy, NO i am not requesting before DOM load. i am doing after call started. in console.
document.getElementById('sv-active-video').getContext('2d')

you can try sample web app, create a meeting and open application in two tabs to join the meeting and try to access document.getElementById('sv-active-video').getContext('2d'). you will get null

Hey @sandeepv,

Yes I can reproduce the null.

This is not a bug on Zooms end, I would suggest trying an alternative way to get the canvas or trying in a different browser.

Thanks,
Tommy

@tommy, i didn’t say. it’s zoom sdk bug. i asked how to get imagedata from canvas[id=sv-active-video] since it’s returning null with getContext(‘2d’). there can be other context type like webgl, 3d… Do you know which type of context canvas[id=sv-active-video] has?

At least my i know how zoom sdk draw image on canvas[id=sv-active-video]?

Hey @sandeepv,

I tried with webgl and it worked! :slight_smile:

document.getElementById('sv-active-video').getContext('webgl')

29%20AM

Thanks,
Tommy

Hi @tommy, Thanks for the response but now context instance giving getImageData is not a function

document.getElementById(‘sv-active-video’).getContext(‘webgl’)
context.getImageData(0, 0, context.canvas.width, context.canvas.height);
context.getImageData is not a function

So my original question still open "how to get image data from ‘sv-active-video’ canvas?

Hey @sandeepv,

Unfortunately getting the image data is not a feature we support or have documented.

My best suggestion is to reach out to https://knowledge.affectiva.com to have them help.

Thanks,
Tommy

I need a snapshot.
Are there any plans for implementation?

Hey @ke-kudo,

We are working on something like this, but do not have a timeline.

You can stay updated here: https://marketplace.zoom.us/docs/changelog

Thanks,
Tommy