Meeting SDK Type and Version
v5.15.11.21082
Description
I am developing a Unity app that displays video and audio from zoom using zoom’s raw data callbacks. I am able to play “mixed” and “one way” sound, but if I mute and unmute myself, then the audio will start dropping out.
I am aware that dropouts can happen if you take too much processing time in the raw data callback, but I’m skeptical that is happening since it works great until I toggle mute.
Troubleshooting Routes
I’ve tried an assortment of techniques to speed up my code, and I don’t know if there’s anything else I can do.
Hey @Wolf_Isaac I’m happy to help! We use the Windows Meeting SDK extensively to power our meeting bots and have definitely run into edge-cases like this one before. To clarify what you mean specifically by the audio dropping out when you mute and unmute, are you saying that:
- You are in a Zoom meeting with yourself (joining via Zoom desktop app), and the Meeting SDK application
- You’re currently receiving audio through the Zoom raw data callback through the Meeting SDK
- When you mute yourself (on the Zoom desktop app) and unmute, you get dropped audio on the Meeting SDK side?
If so, a few follow-up questions:
- When you say dropped audio, what do you mean specifically? If you could send over an audio file that would be super helpful.
- How are you processing the audio in the callback? Our recommendation would be to push the raw audio into an in-memory queue and do processing from a different thread to ensure no frames are dropped.
Hello @amanda-recallai Thank you for reaching out. I happened to find a workaround that I’ll share here! But first I’ll elaborate more on my previous implementation.
The last bullet point you mentioned, pushing raw audio into a queue, happens to be one the techniques I tried! I happen to have two queues: the “data” buffer and the “sample” buffer.
My raw data callback pushes byte arrays to the data buffer. I am confident that I am not losing samples because I wrote some code to save the samples to disk, after completing the callback, and then I convert the data to wav and do not hear any drop outs (even after muting and unmuting).
A separate thread processes the audio for Unity. I increment over every two bytes and convert them to shorts. Then I convert the short to a float. Since the program is written in Unity, I’m required to convert all zoom audio to 48khz, so I take my float and linearly interpolate a certain “factor” of samples between the previous sample I queued.
To output audio in Unity, I implement a callback called “AudioFilterRead,” where I simply take samples from my sample queue and put them in a float array provided by the callback.
Unfortunately this is the implementation that was dropping out, so I tried a new approach. The meeting sdk allows you to select an output speaker. I found that if I set the speaker to a virtual speaker, join voip, and then capture the virtual speaker output in OBS (OBS is already one of our program’s dependencies), then we can stream zoom audio, and adjust the volume level of this audio separately from the Unity app.
Although I’m using this workaround, I would love to find a patch to allow our app to output zoom audio in the Unity engine, since there are lots of neat things we can do with audio if it works!