Sending raw video data sendVideoFrame() method

Description
Is sendVideoFrame() supposed to be a function that we build ourselves, and where would I find documentation to help with that?

Which Desktop Video SDK version?
Windows 1.0.2

Device (please complete the following information):

  • Device: Lenovo Ideapad 320
  • OS: Windows 10

Additional context
I am trying to create a Zoom app that will implement the IZoomInstantSDKVideoSource class and the documentation for how to do this seems incomplete. The subheading for this step of the Render Video portion has the exact same text as the Receive video frames section above it, so there is no context provided for how to use the code snippet they include on how to send raw video data. After trying to incorporate the code example into my app, it was requiring me to create a IZoomInstantSDKVideoSender to pass into the onInitialize() method but when I tried to declare an instance of IZoomInstantSDKVideoSender or create a subclass for it, it would not allow either to compile because the sendVideoFrame() method is a pure virtual function so the classes are forced to be abstract.

1 Like

Hey @rhrdy,

Thanks for using the dev forum!

No this function is simply called when a video frame is meant to be sent. The SDK handles this function, you just need to call it on your Sender object that you received from the onInitialize function.

Ah yep, you are right. Looks like this was a copy-paste mistake in the documentation. We will get that fixed as soon as possible.

Hmm, you should only have to implement the functions in the IZoomInstantSDKVideoSource for your video source class, then when the SDK calls onInitialize you should just have to grab the sender from the parameter. Let me share my code with some things stripped away:

class MyRawDataClass
	, public IZoomInstantSDKDelegate
	, public IZoomInstantSDKRawDataPipeDelegate
	, public IZoomInstantSDKVideoSourcePreProcessor
	, public IZoomInstantSDKVideoSource
{
// Construction
public:
	MyRawDataClass(CWnd* pParent = NULL);	
protected:
///

public: //IZoomInstantSDKDelegate
	virtual void onSessionJoin();
	virtual void onSessionLeave();
	virtual void onError(ZoomInstantSDKErrors errorCode, int detailErrorCode);

	virtual void onUserJoin(IZoomInstantSDKUserHelper* pUserHelper, IInstantSDKVector<IZoomInstantSDKUser*>* userList);
	virtual void onUserLeave(IZoomInstantSDKUserHelper* pUserHelper, IInstantSDKVector<IZoomInstantSDKUser*>* userList);

	virtual void onUserVideoStatusChanged(IZoomInstantSDKVideoHelper* pVideoHelper, IInstantSDKVector<IZoomInstantSDKUser*>* userList);
	virtual void onUserAudioStatusChanged(IZoomInstantSDKAudioHelper* pAudioHelper, IInstantSDKVector<IZoomInstantSDKUser*>* userList);
	virtual void onUserShareStatusChanged(IZoomInstantSDKShareHelper* pShareHelper, IZoomInstantSDKUser* pUser, ZoomInstantSDKShareStatus status);
	virtual void onLiveStreamStatusChanged(IZoomInstantSDKLiveStreamHelper* pLiveStreamHelper, ZoomInstantSDKLiveStreamStatus status);
	virtual void onChatNewMessageNotify(IZoomInstantSDKChatHelper* pChatHelper, IZoomInstantSDKChatMessage* messageItem);
	virtual void onUserHostChanged(IZoomInstantSDKUserHelper* pUserHelper, IZoomInstantSDKUser* pUser);
	virtual void onUserActiveAudioChanged(IZoomInstantSDKAudioHelper* pAudioHelper, IInstantSDKVector<IZoomInstantSDKUser*>* list);

	virtual void onSessionNeedPassword(IZoomInstantSDKPasswordHandler* handler);
	virtual void onSessionPasswordWrong(IZoomInstantSDKPasswordHandler* handler);

	virtual void onMixedAudioRawDataReceived(AudioRawData* data_);
	virtual void onOneWayAudioRawDataReceived(AudioRawData* data_, IZoomInstantSDKUser* pUser);
	virtual void onUserManagerChanged(IZoomInstantSDKUser* pUser);
	virtual void onUserNameChanged(IZoomInstantSDKUser* pUser);

public: //IZoomInstantSDKRawDataPipeDelegate
	virtual void onRawDataFrameReceived(YUVRawDataI420* data_);
	virtual void onRawDataStatusChanged(RawDataStatus status);

public: //IZoomInstantSDKVideoSourcePreProcessor
	virtual void onPreProcessRawData(YUVProcessDataI420* rawData);

public: //IZoomInstantSDKVideoSource
	virtual	void onInitialize(IZoomInstantSDKVideoSender* sender, IInstantSDKVector<VideoSourceCapability >* support_cap_list, VideoSourceCapability& suggest_cap);
	virtual void onPropertyChange(IInstantSDKVector<VideoSourceCapability >* support_cap_list, VideoSourceCapability suggest_cap);
	virtual void onStartSend();
	virtual void onStopSend();
	virtual void onUninitialized();

public:
// ... 
protected:
// ...

private:
	IZoomInstantSDK* m_pInstantSDK;
	IInstantSDKVector<IZoomInstantSDKCameraDevice*>* m_pCameraList;
// ...
	IZoomInstantSDKVideoSender* m_pSender;
// ...
private:
// ...

protected:  
	virtual void OnOK();  
	virtual void OnCancel(); 

public:
// A bunch of button methods here
};

.CPP file:

void MyRawDataClass::onInitialize(IZoomInstantSDKVideoSender* sender, IInstantSDKVector<VideoSourceCapability >* support_cap_list, VideoSourceCapability& suggest_cap)
{
	//when set virtual video source, go here!!!
	if (!sender) return;
	m_pSender = sender;

	//capability
	if (!support_cap_list) return;
	TCHAR szDbg[512] = {0};
	unsigned int nCount = support_cap_list->GetCount();	

	for(int i=0; i<nCount; i++)
	{
		VideoSourceCapability& cap_ = support_cap_list->GetItem(i);
		wsprintf(szDbg, _T("onInitialize cap index:%d: frame=%d, Height=%d, width=%d \r\n"), i, cap_.frame, cap_.height, cap_.width);
		OutputDebugString(szDbg);
	}
}

When I call sendVideoFrame:

UINT send_raw_data(LPVOID pParam) 
{ 
MyRawDataClass* pDemoDlg = (MyRawDataClass*)pParam;		
	if (!pDemoDlg) return 0;
	IZoomInstantSDKVideoSender* sender = pDemoDlg->m_pSender;
	if (!sender) return; int w, h;
	int nFrameLength;
	char* pFrameData;
	FILE* fp = NULL;
	char file_path[MAX_PATH] = { 0 };
	MyRawDataClass::SEND_SIZE_TYPE send_size_type;


	while(1)
	{	

		GetModuleFileNameA(NULL, file_path, MAX_PATH);
		strcat(file_path, "\\..\\");

		if (pDemoDlg->m_send_size_type == MyRawDataClass::SEND_SIZE_TYPE_1080P)
		{
			w = 1920;
			h = 1080;
			nFrameLength = (w*h) + (w*h)/4 + (w*h)/4;
			pFrameData = new char[nFrameLength];
			strcat(file_path, "videofile_1920_1080.yuv");
		}
		else if (pDemoDlg->m_send_size_type == MyRawDataClass::SEND_SIZE_TYPE_720P)
		{
			w = 1280;
			h = 720;
			nFrameLength = (w*h) + (w*h)/4 + (w*h)/4;
			pFrameData = new char[nFrameLength];
			strcat(file_path, "videofile_1280_720.yuv");
		}
		else 
		{
			w = 640;
			h = 480;
			nFrameLength = (w*h) + (w*h)/4 + (w*h)/4;
			pFrameData = new char[nFrameLength];
			strcat(file_path, "videofile_640_480.yuv");
		}

		send_size_type = pDemoDlg->m_send_size_type;
		fp = fopen(file_path, "rb");
		if (!fp) break;

		while (ReadFrameData(fp, nFrameLength, pFrameData))
		{
			if (pDemoDlg->m_send_size_type != send_size_type)
			{
				break;
			}
			sender->sendVideoFrame(pFrameData, w, h, nFrameLength, 0);
			Sleep(35);
		}

		delete[] pFrameData;
		fclose(fp);
	}

	return 0;
}

@Michael_Condon ,

Thank you for giving such a detailed response and providing some example code as guidance.

I looked at your examples and tried to incorporate it into my project but I continued getting null-pointer exceptions whenever I tried to send a video frame. Eventually, I determined that the callbacks for the VideoSource class were never being triggered.

My OnInitialize() method is identical to yours and I do assign the VideoSender object from the parameter to my own class member but after I have declared an instance of my video class, I check the value of the pointer to the VideoSender and it is null. I also have a vector to contain supported capabilities that are filled during the for-loop of the OnInitialize method. I tested to see if the problem was in the initial if-statement of OnInitialize checking if the parameter sender was valid but it never reaches the code.

When is the OnInitialize callback supposed to be triggered?

I tried extending IZoomInstantSDKVideoSource in its own class as described in the documentation here and then I tried mimicking your example code by extending the IZoomInstantSDKDelegate and the IZoomInstantSDKVideoSource in one class but neither approach triggered the callbacks upon being declared.

One error I did see pop up several times while testing this was ‘The stream number provided was invalid.’ but I can’t find details on what that means anywhere.

Thank you for your help.

Last night I was able to figure out the issues with the callbacks by adding it to the sessionContext object and turning off other sources of video but I am still unable to receive the videoFrames on a different computer.

This error persists ‘The stream number provided was invalid.’ and I am wondering if it has something to do with the way we are supposed to prepare the data before sending it through the sendVideoFrame() method.

while (ReadFrameData(fp, nFrameLength, pFrameData))
{
if (pDemoDlg->m_send_size_type != send_size_type)
{
break;
}
sender->sendVideoFrame(pFrameData, w, h, nFrameLength, 0);
Sleep(35);
}

Here you call a ReadFrameData() method that I assume is an abstraction for our personal handling of data but I have trouble intuiting what needs to be done to prepare that buffer for sendVideoFrame().

Do we have to convert all videos we send to one of the supported video capabilities?

If we do convert a video to one of the supported capabilities, is there any additional data that needs to go along with it? I see OnRawDataFrameReceived() takes in a YUVRawDataI420 object, but I haven’t found any information on what makes that different from a standard yuv buffer object. Is that conversion internally handled by the SDK?

I am using OpenCV to convert videos to the YUV420 type.

1 Like

Hey @rhrdy,

My ReadFrameData method is just a testing method that takes in a file and writes it to a buffer:

bool ReadFrameData(FILE* fp, int nFrameLength, char* pBuffer)
{
if (!fp || !pBuffer)
{
return false;
}
size_t nTotalReaded=0;
size_t nReaded=0;
size_t nWantRead = nFrameLength;

while(1)
{
	nReaded = fread(pBuffer, 1, nWantRead , fp);
	if (nReaded != nWantRead)
	{	
		if (feof(fp) || ferror(fp))
		{
			return false;
		}
		else 
		{
			nTotalReaded += nReaded;
			pBuffer += nReaded;
			nWantRead = nFrameLength - nTotalReaded;		
		}
	}
	else 
	{
		return true;	
	}
	}
}

The file it is reading from is a file that is already in yuv format.

Yes.

If you’d like, you can send over your project in a zip to me at DeveloperSupport@zoom.us and I can take a deeper look?

Thanks!
Michael

What is ‘LPVOID pParam’? Where can I call ‘send_raw_data(LPVOID pParam)’?

Hey @tlol91,

pParam is a void pointer to keep track of this class while threading. You should be able to start sending raw data once you have joined the session and onInitialize has been triggered.

Thanks!
Michael