[BUG] Setting virtualAudioSpeaker in ZMVideoSDKSessionContext freezes Zoom Video SDK

mark_coniglio · June 27, 2021, 9:47am

Description
Before joining a session, I am attempting to set virtualAudioSpeaker in ZMVideoSDKSessionContext freezes Zoom Video SDK to an object that conforms to the ZMVideoSDKVirtualAudioSpeaker protocol.

if I set this property and the call [[ZMVideoSDK sharedVideoSDK] joinSession:sessionContext], the Zoom Video SDK hangs preventing further execution.

If I do not set this property, everything works as expected.

In case it’s important: I am executing [[ZMVideoSDK sharedVideoSDK] joinSession:sessionContext] from a background thread

The object conforming to the ZMVideoSDKVirtualAudioSpeaker protocol is declared like this:

@interface SoundOutputAdapter : NSObject <ZMVideoSDKVirtualAudioSpeaker>
{
}
-(void)onVirtualSpeakerMixedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata;
-(void)onVirtualSpeakerOneWayAudioReceived:(ZMVideoSDKAudioRawData*)rawdata user:(ZMVideoSDKUser*)user;
-(void)onVirtualSpeakerSharedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata;
@end

Which Desktop Video SDK version?
macOS 1.1.0

To Reproduce(If applicable)
Steps to reproduce the behavior:

After creating the ZMVideoSDKSessionContext, set the virtualAudioSpeaker property to a valid object that implements the ZMVideoSDKVirtualAudioSpeaker
call `[[ZMVideoSDK sharedVideoSDK]

Device (please complete the following information):

Mac Book Pro 15" (Intel)
OS: macOS Mojave 10.13

Additional context
Here’s the call stack at the moment everything stops working

frame #0: 0x00007fff6516bf02 libsystem_kernel.dylib`__psynch_mutexwait + 10
frame #1: 0x000000010291583b libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 114
frame #2: 0x0000000102912e46 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 231
frame #3: 0x000000010ef71bd1 util`ssb::thread_mgr_t::attach(ssb::thread_wrapper_t*, ssb::thread_wrapper_t*, unsigned int) + 93
frame #4: 0x000000010ef71af0 util`ssb::thread_mgr_t::spawn(unsigned int, unsigned int, unsigned int, bool, signed char*, unsigned int) + 788
frame #5: 0x000000012023560b ssb_sdk`___lldb_unnamed_symbol734$$ssb_sdk + 423
frame #6: 0x000000011f1a8278 zVideoApp`___lldb_unnamed_symbol2965$$zVideoApp + 270
frame #7: 0x000000011f17fb59 zVideoApp`___lldb_unnamed_symbol2753$$zVideoApp + 7463
frame #8: 0x000000011f1bfadd zVideoApp`___lldb_unnamed_symbol3219$$zVideoApp + 357
frame #9: 0x000000011f1a7b66 zVideoApp`___lldb_unnamed_symbol2963$$zVideoApp + 6136
frame #10: 0x000000011f19cf99 zVideoApp`___lldb_unnamed_symbol2917$$zVideoApp + 3861
frame #11: 0x000000011f19bc53 zVideoApp`___lldb_unnamed_symbol2911$$zVideoApp + 783
frame #12: 0x000000011f4fc1f7 zWebService`___lldb_unnamed_symbol1973$$zWebService + 7259
frame #13: 0x000000011f4b7132 zWebService`___lldb_unnamed_symbol1601$$zWebService + 64400
frame #14: 0x000000011f4e1020 zWebService`___lldb_unnamed_symbol1809$$zWebService + 574
frame #15: 0x000000011f46afb2 zWebService`___lldb_unnamed_symbol1393$$zWebService + 2136
frame #16: 0x0000000112fdc0e6 ZMVideoSDK`___lldb_unnamed_symbol696$$ZMVideoSDK + 30
frame #17: 0x0000000112fe5ea2 ZMVideoSDK`___lldb_unnamed_symbol954$$ZMVideoSDK + 18
frame #18: 0x0000000112fe5e85 ZMVideoSDK`___lldb_unnamed_symbol953$$ZMVideoSDK + 27
frame #19: 0x00007fff3906dd78 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 20
frame #20: 0x00007fff3906d924 CoreFoundation`__CFRunLoopDoTimer + 851
frame #21: 0x00007fff3906d46a CoreFoundation`__CFRunLoopDoTimers + 330
frame #22: 0x00007fff3904e582 CoreFoundation`__CFRunLoopRun + 2130
frame #23: 0x00007fff3904dade CoreFoundation`CFRunLoopRunSpecific + 455
frame #24: 0x00007fff382ac1ab HIToolbox`RunCurrentEventLoopInMode + 292
frame #25: 0x00007fff382abee5 HIToolbox`ReceiveNextEventCommon + 603
frame #26: 0x00007fff382abc76 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
frame #27: 0x00007fff3664377d AppKit`_DPSNextEvent + 1135
frame #28: 0x00007fff3664246b AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1361
frame #29: 0x00007fff3663c588 AppKit`-[NSApplication run] + 699
frame #30: 0x0000000100e0b086 XXXXXXX`IZ_AppRun() at IZApp.mm:1007:3
frame #31: 0x00000001007af942 XXXXXXX`LCarbonApp::Run(this=0x00007ffeefbe4760) at LCarbonApp.cpp:178:2
frame #32: 0x00000001009385e6 XXXXXXX`::xmain() at CPPTestApp.cpp:872:11
frame #33: 0x0000000100e0adba XXXXXXX`main(argc=3, argv=0x00007ffeefbff5d0) at IZApp.mm:791:2
frame #34: 0x00007fff650343d5 libdyld.dylib`start + 1

And here’s the call stack from the thread that called [[ZMVideoSDK sharedVideoSDK] joinSession:sessionContext]

frame #0: 0x00007fff6517136a libsystem_kernel.dylib`poll + 10
frame #1: 0x000000010ef6cea6 util`ssb::notifier_pipe_t::wait(int, bool) + 58
frame #2: 0x000000010ef69ce8 util`ssb::msg_queue_t::send_msg(ssb::msg_it*, ssb::msg_queue_sink_it*, int) + 226
frame #3: 0x000000010ef74ce7 util`ssb::thread_wrapper_t::create_channel(ssb::thread_wrapper_t*, unsigned int, unsigned int) + 255
frame #4: 0x000000010ef71c89 util`ssb::thread_mgr_t::attach(ssb::thread_wrapper_t*, ssb::thread_wrapper_t*, unsigned int) + 277
frame #5: 0x000000010ef71af0 util`ssb::thread_mgr_t::spawn(unsigned int, unsigned int, unsigned int, bool, signed char*, unsigned int) + 788
frame #6: 0x0000000116ec48a0 mcm`___lldb_unnamed_symbol1385$$mcm + 1732
frame #7: 0x0000000116eab0b7 mcm`___lldb_unnamed_symbol946$$mcm + 233
frame #8: 0x0000000116eac571 mcm`___lldb_unnamed_symbol986$$mcm + 13
frame #9: 0x000000011f115325 zVideoApp`___lldb_unnamed_symbol1107$$zVideoApp + 335
frame #10: 0x0000000112ff12e2 ZMVideoSDK`___lldb_unnamed_symbol1315$$ZMVideoSDK + 70
frame #11: 0x0000000112ff1786 ZMVideoSDK`___lldb_unnamed_symbol1320$$ZMVideoSDK + 46
frame #12: 0x0000000112fd2e13 ZMVideoSDK`___lldb_unnamed_symbol342$$ZMVideoSDK + 55
frame #13: 0x0000000112fe3c83 ZMVideoSDK`___lldb_unnamed_symbol884$$ZMVideoSDK + 1295
frame #14: 0x0000000112fce9c8 ZMVideoSDK`___lldb_unnamed_symbol142$$ZMVideoSDK + 991
frame #15: 0x000000010eefe16f Zoom Instant Receiver`::-[NSZoomInstantMac joinSession:sessionPassword:userName:](self=0x0000600003328a00, _cmd="joinSession:sessionPassword:userName:", inSessionName="unique-session-name", inSessionPassword="1234", inUserName="johnny") at CZoomInstantMac.mm:489:3
frame #16: 0x000000010eef66db Zoom Instant Receiver`CZoomInstant::JoinSession(this=0x00006000019192c0, inSessionName="unique-session-name", inSessionPassword="1234", inUserName="johnny") at CZoomInstant.mm:109:3
frame #17: 0x000000010eefa6d4 Zoom Instant Receiver`ReceiveMessage(ip=0x0000000101610788, inMessageMask=4096, (null)=0, inData=0x0000000000000000, (null)=0, inRefCon=0x000000010495df20) at XXXXXXXPlugin.cpp:1454:25

While this lockup is happening, this is being sent to the console repeatedly:

2021-06-27 11:46:00.325434+0200 XXXXXXX[16782:1065796] Dropping audio sample buffer 0x10494c2a0 for output <AVCaptureAudioDataOutput: 0x6000002e3180> connection <AVCaptureConnection: 0x600000e26fa0 [type:soun][enabled:1][active:1]>
2021-06-27 11:46:00.325993+0200 XXXXXXX[16782:1065796] Dropping audio sample buffer 0x115bb6f30 for output <AVCaptureAudioDataOutput: 0x6000002e3180> connection <AVCaptureConnection: 0x600000e26fa0 [type:soun][enabled:1][active:1]>
2021-06-27 11:46:00.326774+0200 XXXXXXX[16782:1065796] Dropping audio sample buffer 0x115bbc9c0 for output <AVCaptureAudioDataOutput: 0x6000002e3180> connection <AVCaptureConnection: 0x600000e26fa0 [type:soun][enabled:1][active:1]>
2021-06-27 11:46:00.327441+0200 XXXXXXX[16782:1065796] Dropping audio sample buffer 0x115bb6f30 for output <AVCaptureAudioDataOutput: 0x6000002e3180> connection <AVCaptureConnection: 0x600000e26fa0 [type:soun][enabled:1][active:1]>
2021-06-27 11:46:00.425330+0200 XXXXXXX[16782:1065796] Dropping audio sample buffer 0x102afa840 for output <AVCaptureAudioDataOutput: 0x6000002e3180> connection <AVCaptureConnection: 0x600000e26fa0 [type:soun][enabled:1][active:1]>
2021-06-27 11:46:00.426160+0200 XXXXXXX[16782:1065796] Dropping audio sample buffer 0x102afaba0 for output <AVCaptureAudioDataOutput: 0x6000002e3180> connection <AVCaptureConnection: 0x600000e26fa0 [type:soun][enabled:1][active:1]>
2021-06-27 11:46:00.427049+0200 XXXXXXX[16782:1065796] Dropping audio sample buffer 0x102e612b0 for output <AVCaptureAudioDataOutput: 0x6000002e3180> connection <AVCaptureConnection: 0x600000e26fa0 [type:soun][enabled:1][active:1]>

jon.zoom · June 28, 2021, 9:21pm

Hi @mark_coniglio, thanks for the post.

Sorry to hear you’re running into issues with implementing the ZMVideoSDKVirtualAudioSpeaker. In your implementation, are you doing anything within the audio callbacks themselves?

Thanks!

mark_coniglio · June 30, 2021, 10:53am

@jon.zoom

Yes, I’m using an external video source. My setup code is below, along with the definition of the Here is SoundOutAdapter class in case that is helpful.

Best Wishes,
Mark

ZMVideoSDKAudioOption* audioOption = [[[ZMVideoSDKAudioOption alloc] init] autorelease];
audioOption.connect = YES;
audioOption.mute = NO;

ZMVideoSDKVideoOption* videoOption = [[[ZMVideoSDKVideoOption alloc] init] autorelease];
videoOption.localVideoOn = YES;

ZMVideoSDKSessionContext *sessionContext = [[[ZMVideoSDKSessionContext alloc] init] autorelease];;

// alloxcate our video sender
mSendYUVAdapter = [[SendYUVAdapter alloc] init];

mSoundOutputAdapter = [[SoundOutputAdapter alloc] init];

sessionContext.token = [NSString stringWithUTF8String:not_a_real_jwt_token];
sessionContext.sessionName = [NSString stringWithUTF8String:inSessionName.c_str()];
sessionContext.sessionPassword = [NSString stringWithUTF8String:inSessionPassword.c_str()];
sessionContext.userName = [NSString stringWithUTF8String:inUserName.c_str()];
sessionContext.audioOption = audioOption;
sessionContext.videoOption = videoOption;
sessionContext.externalVideoSource = mSendYUVAdapter;
sessionContext.virtualAudioSpeaker = mSoundOutputAdapter;

Here is The definition of the my SoundOutAdapter class, which is used above:

@interface SoundOutputAdapter : NSObject <ZMVideoSDKVirtualAudioSpeaker>
{

}
-(void)onVirtualSpeakerMixedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata;
-(void)onVirtualSpeakerOneWayAudioReceived:(ZMVideoSDKAudioRawData*)rawdata user:(ZMVideoSDKUser*)user;
-(void)onVirtualSpeakerSharedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata;
@end

jon.zoom · June 30, 2021, 5:14pm

Hi @mark_coniglio,

Apologies if it wasn’t clear, but I was asking about the implementation of the virtual speaker. Can you provide some context around the implementation?

Thanks!

mark_coniglio · June 30, 2021, 8:10pm

@jon.zoom

It’s simply placeholders at the moment but the methods are never called. (I have breakpoints in each one.)

As I explained in the first post, simply trying to call [[ZMVideoSDK sharedVideoSDK] joinSession:sessionContext] causes a total hang.

@interface SoundOutputAdapter : NSObject <ZMVideoSDKVirtualAudioSpeaker>
{

}
-(void)onVirtualSpeakerMixedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata;
-(void)onVirtualSpeakerOneWayAudioReceived:(ZMVideoSDKAudioRawData*)rawdata user:(ZMVideoSDKUser*)user;
-(void)onVirtualSpeakerSharedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata;
@end


@implementation SoundOutputAdapter
-(void)onVirtualSpeakerMixedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata
{
	NSDebugLog(@"**ZOOM** onVirtualSpeakerMixedAudioReceived: buffer len = %d, samp rate = %d, chan = %d", (int)[rawdata bufferLen], (int)[rawdata sampleRate], (int)[rawdata channelNum]);
}

-(void)onVirtualSpeakerOneWayAudioReceived:(ZMVideoSDKAudioRawData*)rawdata user:(ZMVideoSDKUser*)user
{
	NSDebugLog(@"**ZOOM** onVirtualSpeakerOneWayAudioReceived: buffer len = %d, samp rate = %d, chan = %d", (int)[rawdata bufferLen], (int)[rawdata sampleRate], (int)[rawdata channelNum]);
}

-(void)onVirtualSpeakerSharedAudioReceived:(ZMVideoSDKAudioRawData*)rawdata
{
	NSDebugLog(@"**ZOOM** onVirtualSpeakerSharedAudioReceived: buffer len = %d, samp rate = %d, chan = %d", (int)[rawdata bufferLen], (int)[rawdata sampleRate], (int)[rawdata channelNum]);
}

jon.zoom · June 30, 2021, 11:43pm

Hi @mark_coniglio,

Thanks for clarifying. Looking into this further, there does appear to be a crash happening when setting the virtual speaker. We will investigate and let you know as soon as we have any updates.

Thanks!

jon.zoom · July 1, 2021, 7:56pm

Hi @mark_coniglio,

Sorry for the miscommunication above, it turns out the crash was actually a mistake on my part.

I have not been able to reproduce this after resolving the issue I was seeing. Are you able to provide a simple application in which this issue is reproducible so that we can investigate?

Thanks!

mark_coniglio · July 2, 2021, 8:45am

@jon.zoom

Thanks for looking into this Jon; I’ve found the root of the problem.

If you want a system where you can see this in action, please contact me off list (I assume you can see my email since you’re on staff) and I can sent you my app and my plugin so that you can follow steps to recreate the issue.

BACKGROUND Among other things, my app can capture audio and video using AVFoundation capture session. This capture functionality is optional: the user must enable it either manually or programatically.

VIRTUAL SPEAKER WORKS: If I do things in this order, the virtual speaker functionality works as expected:

Start up my app, and start video and audio capture.(Note that audio capture input is set to the internal Mac Book Pro Microphone.)
Load the patch that contains the Zoom SDK Plugin, which will cause ZMVideoSDKErrors zoomErr = [[ZMVideoSDK sharedVideoSDK] initialize:initParams]; to be called. Note that the default audio device chosen by the SDK at this point is also the Mac Book Pro Microphone.
Initialize the session using joinSession

VIRTUAL SPEAKER DEADLOCKS If I however do it like this, the virtual speaker functionality causes a deadlock between yor functions util:ssb::thread_mgr_t::attach and util:ssb::notifier_pipe_t::wait (See the stack traces included in my original post for more detail.)

Start up my app but do not start capturing video or audio yet.
Load the patch that contains the Zoom SDK Plugin, which will cause ZMVideoSDKErrors zoomErr = [[ZMVideoSDK sharedVideoSDK] initialize:initParams]; to be called. Again, the default audio input chosen by the Zoom
Start capturing audio (again using the the internal Mac Book Pro Microphone).
Attempt to start a session

WORKAROUND: If after step #2 in the list immediately above I do the following to choose a different audio input device before I start capturing video and audio in my app, then

ZMVideoSDKAudioHelper* audioHelper = [[ZMVideoSDK sharedVideoSDK] getAudioHelper];
NSArray* mics = [audioHelper getMicList];
ZMVideoSDKMicDevice* d = [mics objectAtIndex:1];
[[[ZMVideoSDK sharedVideoSDK] getAudioHelper] selectMic:[d deviceId] deviceName:[d deviceName]];

NOTES Note that when I was not attempting to enable the virtual speaker code, everything worked fine: my app and Zoom SDK captured audio simultaneously from the same audio input device. So this problem is specifically related to the virtual speaker and the conflict that is created when both my app and your SDK want to capture from the same audio input.

Best Wishes,
Mark

jon.zoom · July 2, 2021, 8:28pm

Hi @mark_coniglio,

Thank you for the detailed post. We should have enough information to begin investigating this. If we need any additional information regarding the AVFoundation implementation in your application, I will be sure to email you directly using the email on your dev forum account.

Thanks!

system · August 2, 2021, 6:29am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Missing Documentation: enableEchoCancellation, enableHighFidelityMusicMode, or enableStereoAudio Fail Unless Mic Original Input is Enabled macOS	2	569	December 15, 2022
Video freezing when setting virtual background macOS	14	2397	June 19, 2022
Participant can't hear anyone after joining a session Web	4	401	October 13, 2022
Web Video SDK Version 1.2.5 New Releases	1	2169	May 3, 2022
Unable to get instance of IZoomSDKAudioRawDataHelper Windows	2	461	April 1, 2023

[BUG] Setting virtualAudioSpeaker in ZMVideoSDKSessionContext freezes Zoom Video SDK

Related Topics