I’m maintaining an app where teachers can host live lessons to students. In the past, the Zoom client was triggered from inside our Angular app (using the Zoom Meeting SDK). This was troublesome for different reasons, e.g. every user needed to install the Zoom client, users leaving the app when the meeting starts (we want to keep them inside our app), etc, etc. So we decided to make it possible to host meetings inside our app and settled down on using Zoom Video SDK for this purpose. This is the UX setup I ended up implementing:
The user has the option to switch back and forth between the views. When switching to another view, HtmlElements are remembered and used in the next view, if possible. Anything that has to do with rendering the above views (think resizing, change in participants, etc) goes via RXJS with proper debouncing to make the setup more stable. There’s also two HtmlCanvasElements placed on top of the view to render things like participant names and active speaker. The videos are rendered at different qualities, depending on the size of the participant on screen, ranging from 160p to 720p. The teacher is rendered in higher quality, but not more than 720p.
We’ve started using this implementation in our live environment for a limited number of users (the most we had is 7 I think) and I’ve noticed a few things that concerned me:
high CPU usage. E.g. this morning there was a session with 3 users. The teacher was running a powerful laptop, Windows and browser MS Edge (running in a PWA app). The CPU was at 50% throughout the session.
high mem usage. I’ve noticed up to 10GB of memory being used in one case, most of it GPU memory
there’s video lag / freezing for most of the participants drawn on a single HtmlCanvasElement. Every few minutes, one of the participant’s video will freeze for 2-3 seconds and then continue to be rendered normally.
I’m curious about the setup of other implementations and what their experience is with respect to performance or how they solved it. Please reply if you have advice, it’s greatly appreciated!
Is the “video freezing” a well-known issue? How can it be mitigated? Thank you!
To enhance performance, we recommend enabling SharedArrayBuffer on the page and rendering videos on the same canvas. You can find more information about this approach in our documentation:
Additionally, it’s great to see that you’ve begun experimenting with our new rendering approach, attachVideo and video-player.
Regarding the differentiation between self-view and remote video in your figure above, these can all be achieved using stream.attachVideo. Our Video SDK handles these logics internally."
I’m already using the SharedArrayBuffer to display students. However, there’s also one videostream I need to render on a VideoPlayer: the teacher. So this means one SharedArrayBuffer and one VideoPlayer on screen at most. I need the VideoPlayer because I need be able to zoom in/out on the teacher’s video. Could the fact that I use two elements for rendering cause performance issues? Is there a way to zoom in/out on the teacher when the teacher is rendered together with the students in a single SharedArrayBuffer?
I didn’t know btw that stream.attachVideo als handles self-view automatically, thanks for the tip.
When you mention SharedArrayBuffer, are you referring to the canvas or the video-player-container?
Regarding the video-player under a video-player-container , we don’t restrict having only one. On PC, the maximum limit is 25.
For zooming in/out on the teacher’s view, you can simply use CSS styles to enlarge or shrink the video-player bound to the teacher, similar to manipulating regular DOM elements.
I tried using multiple VideoPlayers initially, but I could only use 4 or 5 (I don’t remember exactly) of those on some devices. So I decided to stick with a single SharedArrayBuffer rendered on HtmlCanvasElement instead, and use only one VideoPlayer for the teacher. Zooming in & out on the teacher using a VideoPlayer is already working for me, indeed with some javascript/css tricks. I was wondering whether I can use a single HtmlCanvasElement for both teacher & students, and still being able to zoom in on the teacher?
Does it affect performance a lot when I’m using a VideoPlayer and a HtmlCanvasElement next to each other? Why do users experience occasional freezing up of the image on the HtmlCanvasElement?
Actually, you haven’t enabled SharedArrayBuffer on the page. In this situation, if you want to render multiple videos on the page, there are two approaches you can try:
Follow this doc and enable SharedArrayBuffer on your page, this will greatly improve the performance and your layout implementation can look like this:
For up to 4 users, their videos can be rendered using the video-player. For any additional users, each video can be rendered using one video per canvas. However, this approach may lead to poorer performance:
maybe I wasn’t clear enough I have already enabled SharedArrayBuffer (with correct headers etc) and I am using a single HtmlCanvasElement to render all the students. Besides that I’m only using a single VideoPlayer to render the teacher (for the zoom in/out functionality). I’m not using other constructs. Is the fact that I’m using that VideoPlayer for the teacher causing the freeze issues on the HtmlCanvasElement, or what else could cause those issues? Maybe I have to downsize the quality of the streams depending on network bandwidt hof the client? or downsize relative to CPU power (if there’s a way to determine that) ?
That’s not a problem. One thing to note is to try to reuse canvas elements and video-player-container elements as much as possible, as frequent destruction and creation may also lead to video freezes.
If you encounter similar issues, could you share the relevant sessionId with us? We can help troubleshoot the issue.
I have already enabled SharedArrayBuffer (with correct headers etc) and I am using a single HtmlCanvasElement to render all the students. Besides that I’m only using a single VideoPlayer to render the teacher (for the zoom in/out functionality). I’m not using other constructs. Is the fact that I’m using that VideoPlayer for the teacher causing the freeze issues on the HtmlCanvasElement, or what else could cause those issues? Maybe I have to downsize the quality of the streams depending on network bandwidt hof the client? or downsize relative to CPU power (if there’s a way to determine that) ?