# sendVideoFrame Does Not Work As Expected **URL:** \ **Category:** iOS\ **Tags:** video-sdk\ **Created:** [June 14, 2023, 2:25pm UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163 "2023-06-14T14:25:03Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [June 14, 2023, 2:25pm UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/1 "2023-06-14T14:25:03Z")
**Description** In my app I have a custom camera implementation, from which I can intercept frames. my plan is to get frames from my camera and feed them into the Zoom Video SDK using `ZoomVideoSDKVideoSender.sendVideoFrame`. However, the frames I send using this function are not received anywhere. Here are the relevant steps in the setup: 1. I create a session as follow: ```swift let sessionContext = ZoomVideoSDKSessionContext() sessionContext.token = token sessionContext.sessionName = topic sessionContext.userName = name let vidOption = ZoomVideoSDKVideoOptions() vidOption.localVideoOn = false let audOptions = ZoomVideoSDKAudioOptions() audOptions.connect = true audOptions.mute = false sessionContext.videoOption = vidOption sessionContext.audioOption = audOptions sessionContext.externalVideoSourceDelegate = self ZoomVideoSDK.shareInstance()?.joinSession(sessionContext) ``` 1. As I set the external video source delegate to `self`, I extend `ZoomVideoSDKVideoSource` ```swift extension ZoomViewController: ZoomVideoSDKVideoSource { func onInitialize(_ rawDataSender: ZoomVideoSDKVideoSender, supportCapabilityArray: [Any], suggest suggestCapability: ZoomVideoSDKVideoCapability) { self.frameSender = rawDataSender } } ``` 1. When a user joins the session, I find that user and subscribe to their video pipe ```swift other.getVideoPipe()?.subscribe(with: self, resolution: ._360) ``` 1. and I extend with `ZoomVideoSDKRawDataPipeDelegate` as follow: ```swift extension ZoomViewController: ZoomVideoSDKRawDataPipeDelegate { func onPixelBuffer(_ pixelBuffer: CVPixelBuffer?, rotation: ZoomVideoSDKVideoRawDataRotation) { print("Frame received") } func onRawDataFrameReceived(_ rawData: ZoomVideoSDKVideoRawData?) { print("Received raw frame ") } } ``` 1. When I receive a frame from my camera implementation, I send it using the frame sender initialized earlier Now the issue is that neither of `onPixelBuffer` and `onRawDataFrameReceived` (in step 4) is ever triggered when I send frames (step 5) Is my use of `sendVideoFrame` correct? **Which iOS Video SDK version?** iOS 1.7.10 ---
**Author:** ![chunsiong.zoom](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/chunsiong.zoom/32/35488_2.png) [@chunsiong.zoom](https://devforum.zoom.us/u/chunsiong.zoom)\ **Post date:** [June 15, 2023, 3:07am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/2 "2023-06-15T03:07:15Z")
@wbaroudi , could you share the code on how you are using sendVideoFrame? The understanding of your flow above is correct. There is a possibility the issue on the sendVideoFrame side. ---
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [June 15, 2023, 2:49pm UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/3 "2023-06-15T14:49:37Z")
Hello @chunsiong.zoom , My bad, I missed that part. `sendVideoFrame` is used in the following process: 1. I have a struct `ZoomFrame` which looks like this ```swift struct ZoomFrame { let buffer: UnsafeMutablePointer let width: UInt let height: UInt let dataLength: UInt let rotation: ZoomVideoSDKVideoRawDataRotation } ``` 1. My camera implementation gives me frames as `CMSampleBuffer`s, and I convert them to `ZoomFrame`s as follows: ```swift extension CMSampleBuffer { func toZoomFrame() -> ZoomFrame? { // Ensure buffer data is ready guard CMSampleBufferDataIsReady(self) else { return nil } // Try to get an image buffer guard let imageBuffer = CMSampleBufferGetImageBuffer(self) else { return nil } // Get the width and height of the image buffer let width = UInt(CVPixelBufferGetWidth(imageBuffer)) let height = UInt(CVPixelBufferGetHeight(imageBuffer)) // Lock the base address CVPixelBufferLockBaseAddress(imageBuffer, .readOnly) // Get the data pointer guard let dataPointer = CVPixelBufferGetBaseAddress(imageBuffer) else { return nil } // Get the data length let bytesPerRow = UInt(CVPixelBufferGetBytesPerRow(imageBuffer)) let dataLength = bytesPerRow * height // Cast to UnsafeMutablePointer let charPointer = dataPointer.assumingMemoryBound(to: CChar.self) // Unlock the base address CVPixelBufferUnlockBaseAddress(imageBuffer, .readOnly) return ZoomFrame(buffer: charPointer, width: width, height: height, dataLength: dataLength, rotation: .rotationNone) } } ``` 1. I feed the converted frames into the following function: ```swift func sendFrame(frame: ZoomFrame) { if let sender = frameSender { sender.sendVideoFrame(frame.buffer, width: frame.width, height: frame.height, dataLength: frame.dataLength, rotation: frame.rotation, format: .I420) // TODO: maybe change format? print("Frame sent") } } ``` And the print “Frame sent” is reached. Hope this makes things clearer. Thanks! ---
**Author:** ![chunsiong.zoom](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/chunsiong.zoom/32/35488_2.png) [@chunsiong.zoom](https://devforum.zoom.us/u/chunsiong.zoom)\ **Post date:** [June 16, 2023, 2:03am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/4 "2023-06-16T02:03:31Z")
@wbaroudi After converting CMSampleBuffer to `ZoomFrame`, is it possible to do either of these test? I’m assuming this is in YUV420 format, but there is a possibility the format might not be valid. **Testing if conversion has a valid YUV420 format** 1. Instead of sending the converted YUV420 frames to `sendFrame`, save it locally to a `recording.yuv` file. 2. Convert the `recording.yuv` file into a mp4 file using `ffmpeg` command line (might not be 100% accurate, you might need some adjustments for the parameters) `ffmpeg -f rawvideo -pix_fmt yuv420p -framerate 25 -i recording.yuv -f mp4 recording.mp4` 3. Verify that the mp4 plays, and is in correct format without any distortion or color errors **Testing with a video file instead of camera buffer stream.** 1. Download a sample video. I’ve previously used big buck bunny 720p 10s 1MB [MP4 ( H.264 ) | Test Videos](https://test-videos.co.uk/bigbuckbunny/mp4-h264) 2. Convert the video into YUV420 buffer 3. Send the buffer into sendVideoFrame ---
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [June 16, 2023, 3:15pm UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/5 "2023-06-16T15:15:51Z")
@chunsiong.zoom , Apparently the `CVPixelBuffers` in my frames are in `32BGRA` format, which may be the cause of the issue. However, I have tried the video test, and, unfortunately, I still didn’t manage to receive a frame on the other side. Here are the steps in my test: 1. Get the video and convert it to a YUV420 buffer ```swift func getSampleVideoBuffer(url: URL) -> CVPixelBuffer? { // Create AVAsset let asset = AVAsset(url: url) // Get video track guard let videoTrack = asset.tracks(withMediaType: .video).first else { print("Failed to get video track") return nil } // Get video settings let videoSettings: [String: Any] = [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, kCVPixelBufferWidthKey as String: videoTrack.naturalSize.width, kCVPixelBufferHeightKey as String: videoTrack.naturalSize.height ] // Create AVAssetReader do { let reader = try AVAssetReader(asset: asset) // Create AVAssetReaderTrackOutput and assign to reader let readerOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: videoSettings) reader.add(readerOutput) // Start reading reader.startReading() // Create pixel buffer from sample buffer if let sampleBuffer = readerOutput.copyNextSampleBuffer(), let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) { // Release sample buffer CMSampleBufferInvalidate(sampleBuffer) // Return pixel buffer in YUV420 format return pixelBuffer } else { print("Failed to create sample buffer") return nil } } catch { print("Failed to create AVAssetReader: \(error)") return nil } } ``` 1. Send the buffer: ```swift if let zoomFrame = sample?.toZoomFrame() { sendFrame(frame: zoomFrame) } ``` Note that I could view the resulting `zoomFrame` in the debugger and it looked fine. ---
**Author:** ![chunsiong.zoom](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/chunsiong.zoom/32/35488_2.png) [@chunsiong.zoom](https://devforum.zoom.us/u/chunsiong.zoom)\ **Post date:** [June 17, 2023, 7:45am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/6 "2023-06-17T07:45:07Z")
@wbaroudi , if you are able to see the frame just fine, there is a high chance the format is not correct. YUV420 raw format looks something like this ![image](https://us1.discourse-cdn.com/flex016/uploads/zoomdeveloper/original/3X/3/f/3f1311e4d2bbe748ce7c5ed0da15d10af21b26fc.png) Reference [How to extract frames from a .yuv video file (YUV420) using python and openCV? - Stack Overflow](https://stackoverflow.com/questions/60704826/how-to-extract-frames-from-a-yuv-video-file-yuv420-using-python-and-opencv) ---
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [June 18, 2023, 10:57am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/7 "2023-06-18T10:57:37Z")
@chunsiong.zoom , I’m not sure I understand the issue. If the format is correct, the frame should look in grayscale (similar to the sample you showed), instead of its normal colors? The format of the frame is `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` is there a specific format that `sendVideoFrame` expects? ---
**Author:** ![chunsiong.zoom](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/chunsiong.zoom/32/35488_2.png) [@chunsiong.zoom](https://devforum.zoom.us/u/chunsiong.zoom)\ **Post date:** [June 19, 2023, 2:07am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/8 "2023-06-19T02:07:07Z")
@wbaroudi , I don’t have swift code, but here’s how I’ve managed to do it on c++ & windows. The process should be similar. This code reads from a mp4 file, converts it frame by frame to yuv420 format and sends it ```auto int video_is_playing = 1; char* frameBuffer; // Calculate the size of the YUV420 buffer based on frame dimensions int frameLen = height / 2 * 3 * width; // Height/2 for subsampled U and V planes frameBuffer = (char*)malloc(frameLen); // Execute in a thread. while (video_is_playing > 0 && video_sender) { Mat frame; // Represents a single video frame VideoCapture cap; // Represents a video capture device; cap.open(video_source); //opens up a mp4 file from local storage path if (!cap.isOpened()) { cerr << "ERROR! Unable to open camera stream or video file\n"; video_is_playing = 0; return; } // GRAB AND WRITE LOOP cout << "Start grabbing" << endl; while (cap.read(frame)) { // Wait for a new frame from the camera and store it into 'frame' // Check if we succeeded in reading a frame if (frame.empty()) { cerr << "ERROR! Blank frame grabbed\n"; break; } // Convert Mat to YUV buffer Mat yuv; cv::cvtColor(frame, yuv, COLOR_BGRA2YUV_I420); // Get the YUV buffer data char* frameBuffer = (char*)yuv.data; int width = yuv.cols; int height = yuv.rows; int frameLen = yuv.total() * yuv.elemSize(); // Calculate YUV420 buffer size // Send the YUV buffer to the video sender SDKError err = ((IZoomSDKVideoSender*)video_sender)->sendVideoFrame(frameBuffer, width, height, frameLen, 0); if (err != SDKERR_SUCCESS) { cout << "sendVideoFrame failed: Error " << err << endl; } } cap.release(); } video_is_playing = -1; ``` ---
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [June 19, 2023, 8:44am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/9 "2023-06-19T08:44:02Z")
@chunsiong.zoom I see. As far as I understand, all else being the same, if I use OpenCV to convert my camera frames from 32BGRA to YUV\_I420, `sendVideoFrame` should work fine and I should receive frames on the other side which is subscribed to my `videoPipe`. Anything else I should change? Thanks! ---
**Author:** ![chunsiong.zoom](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/chunsiong.zoom/32/35488_2.png) [@chunsiong.zoom](https://devforum.zoom.us/u/chunsiong.zoom)\ **Post date:** [June 19, 2023, 9:09am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/10 "2023-06-19T09:09:42Z")
@wbaroudi the frame length is also slightly different, do reference the code i’ve shared ---
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [July 3, 2023, 1:52pm UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/12 "2023-07-03T13:52:19Z")
Hello, @chunsiong.zoom I was trying again with the video test that you mentioned, attempting to replicate your c++ conversion code in swift. However, in the process, I noticed that the frames coming from the video I downloaded are already in YUV420 bi-planar (full range) format. Is that the case or am I missing something? and if so, then there’s no need for the `cv::cvtColor(frame, yuv, COLOR_BGRA2YUV_I420);` step in the first place, correct? ---
**Author:** ![chunsiong.zoom](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/chunsiong.zoom/32/35488_2.png) [@chunsiong.zoom](https://devforum.zoom.us/u/chunsiong.zoom)\ **Post date:** [July 4, 2023, 7:18am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/13 "2023-07-04T07:18:08Z")
@wbaroudi , You are trying to send video right? The video which you downloaded, is this referring to your camera stream which has been converted to YUV420 format? `cv::cvtColor(frame, yuv, COLOR_BGRA2YUV_I420)` is not necessary if it is already converted to YUV ---
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [July 4, 2023, 9:36am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/14 "2023-07-04T09:36:41Z")
@chunsiong.zoom yes, I’m trying to send send the `big buck bunny 720p 10s 1MB` video frame by frame. Here’s the full code for it: ```swift import AVFoundation import ZoomVideoSDK class VideoProcessor { var videoIsPlaying: Int = 1 var assetReader: AVAssetReader? var asset: AVAsset? var videoTrackOutput: AVAssetReaderTrackOutput? private var frameSender: ZoomVideoSDKVideoSender? init?(videoURL: URL, sender: ZoomVideoSDKVideoSender) { frameSender = sender self.asset = AVAsset(url: videoURL) do { try self.assetReader = AVAssetReader(asset: self.asset!) } catch { print("ERROR! Unable to open video file") return nil } guard let videoTrack = self.asset?.tracks(withMediaType: .video).first else { print("ERROR! Could not get video track") return nil } let outputSettings: [String: Any] = [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange ] self.videoTrackOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: outputSettings) self.assetReader?.add(videoTrackOutput!) } func startProcessing() { self.assetReader?.startReading() while self.videoIsPlaying > 0 { if let sampleBuffer = self.videoTrackOutput?.copyNextSampleBuffer() { // Get the YUV buffer data if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) { let width = CVPixelBufferGetWidth(imageBuffer) let height = CVPixelBufferGetHeight(imageBuffer) let frameLen = CVPixelBufferGetDataSize(imageBuffer) // Access the YUV data (ensure the memory block is locked for reading) CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly) let rawPointer = CVPixelBufferGetBaseAddress(imageBuffer) // Convert UnsafeMutableRawPointer to UnsafeMutablePointer let frameBuffer = rawPointer?.assumingMemoryBound(to: CChar.self) // Send the YUV buffer to the video sender sendVideoFrame(frameBuffer, width, height, frameLen, 0) CVPixelBufferUnlockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly) } CMSampleBufferInvalidate(sampleBuffer) } else { // End of file or an error occurred self.videoIsPlaying = -1 } } } func sendVideoFrame(_ frameBuffer: UnsafeMutablePointer?, _ width: Int, _ height: Int, _ frameLen: Int, _ timestamp: Int64) { if let sender = frameSender, let buffer = frameBuffer { sender.sendVideoFrame(buffer, width: UInt(width), height: UInt(height), dataLength: UInt(frameLen), rotation: .rotationNone, format: .I420) // TODO: maybe change format? print("Frame sent") } else { print("Sender is not initialized") } } } ``` Note that since the frames from this video are already in bi-planar yuv420, I skipped the conversion step. The issue persists, and I don’t seem to receive any frames on the other side. ---
**Author:** ![wbaroudi](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/wbaroudi/32/30319_2.png) [@wbaroudi](https://devforum.zoom.us/u/wbaroudi)\ **Post date:** [July 18, 2023, 11:41am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/15 "2023-07-18T11:41:51Z")
Hi @chunsiong.zoom Would you happen to have a working sample where you make use of the `sendVideoFrame` function in iOS? ---
**Author:** ![chunsiong.zoom](https://sea2.discourse-cdn.com/flex016/user_avatar/devforum.zoom.us/chunsiong.zoom/32/35488_2.png) [@chunsiong.zoom](https://devforum.zoom.us/u/chunsiong.zoom)\ **Post date:** [July 18, 2023, 11:54am UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/16 "2023-07-18T11:54:45Z")
@wbaroudi sorry no I don’t. Here’s a sample on windows nonetheless. I hope it helps in some way > **[GitHub - tanchunsiong/MSDK\_RawDataDemos: Consolidation of 1 Solution with...](https://github.com/tanchunsiong/MSDK_RawDataDemos)** > > Consolidation of 1 Solution with multiple raw data projects - GitHub - tanchunsiong/MSDK\_RawDataDemos: Consolidation of 1 Solution with multiple raw data projects ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/zoomdeveloper/original/3X/6/1/614bdd549b610bbaa46ff934617683a02bdaa03c.png) [@system](https://devforum.zoom.us/u/system)\ **Post date:** [August 17, 2023, 9:55pm UTC](https://devforum.zoom.us/t/sendvideoframe-does-not-work-as-expected/90163/17 "2023-08-17T21:55:16Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.