Description
In my app I have a custom camera implementation, from which I can intercept frames. my plan is to get frames from my camera and feed them into the Zoom Video SDK using ZoomVideoSDKVideoSender.sendVideoFrame. However, the frames I send using this function are not received anywhere.
Here are the relevant steps in the setup:
I create a session as follow:
let sessionContext = ZoomVideoSDKSessionContext()
sessionContext.token = token
sessionContext.sessionName = topic
sessionContext.userName = name
let vidOption = ZoomVideoSDKVideoOptions()
vidOption.localVideoOn = false
let audOptions = ZoomVideoSDKAudioOptions()
audOptions.connect = true
audOptions.mute = false
sessionContext.videoOption = vidOption
sessionContext.audioOption = audOptions
sessionContext.externalVideoSourceDelegate = self
ZoomVideoSDK.shareInstance()?.joinSession(sessionContext)
As I set the external video source delegate to self, I extend ZoomVideoSDKVideoSource
Hello @chunsiong.zoom ,
My bad, I missed that part. sendVideoFrame is used in the following process:
I have a struct ZoomFrame which looks like this
struct ZoomFrame {
let buffer: UnsafeMutablePointer<CChar>
let width: UInt
let height: UInt
let dataLength: UInt
let rotation: ZoomVideoSDKVideoRawDataRotation
}
My camera implementation gives me frames as CMSampleBuffers, and I convert them to ZoomFrames as follows:
extension CMSampleBuffer {
func toZoomFrame() -> ZoomFrame? {
// Ensure buffer data is ready
guard CMSampleBufferDataIsReady(self) else { return nil }
// Try to get an image buffer
guard let imageBuffer = CMSampleBufferGetImageBuffer(self) else { return nil }
// Get the width and height of the image buffer
let width = UInt(CVPixelBufferGetWidth(imageBuffer))
let height = UInt(CVPixelBufferGetHeight(imageBuffer))
// Lock the base address
CVPixelBufferLockBaseAddress(imageBuffer, .readOnly)
// Get the data pointer
guard let dataPointer = CVPixelBufferGetBaseAddress(imageBuffer) else { return nil }
// Get the data length
let bytesPerRow = UInt(CVPixelBufferGetBytesPerRow(imageBuffer))
let dataLength = bytesPerRow * height
// Cast to UnsafeMutablePointer<CChar>
let charPointer = dataPointer.assumingMemoryBound(to: CChar.self)
// Unlock the base address
CVPixelBufferUnlockBaseAddress(imageBuffer, .readOnly)
return ZoomFrame(buffer: charPointer, width: width, height: height, dataLength: dataLength, rotation: .rotationNone)
}
}
I feed the converted frames into the following function:
After converting CMSampleBuffer to ZoomFrame, is it possible to do either of these test?
I’m assuming this is in YUV420 format, but there is a possibility the format might not be valid.
Testing if conversion has a valid YUV420 format
Instead of sending the converted YUV420 frames to sendFrame, save it locally to a recording.yuv file.
Convert the recording.yuv file into a mp4 file using ffmpeg command line (might not be 100% accurate, you might need some adjustments for the parameters) ffmpeg -f rawvideo -pix_fmt yuv420p -framerate 25 -i recording.yuv -f mp4 recording.mp4
Verify that the mp4 plays, and is in correct format without any distortion or color errors
Testing with a video file instead of camera buffer stream.
Apparently the CVPixelBuffers in my frames are in 32BGRA format, which may be the cause of the issue.
However, I have tried the video test, and, unfortunately, I still didn’t manage to receive a frame on the other side. Here are the steps in my test:
Get the video and convert it to a YUV420 buffer
func getSampleVideoBuffer(url: URL) -> CVPixelBuffer? {
// Create AVAsset
let asset = AVAsset(url: url)
// Get video track
guard let videoTrack = asset.tracks(withMediaType: .video).first else {
print("Failed to get video track")
return nil
}
// Get video settings
let videoSettings: [String: Any] = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
kCVPixelBufferWidthKey as String: videoTrack.naturalSize.width,
kCVPixelBufferHeightKey as String: videoTrack.naturalSize.height
]
// Create AVAssetReader
do {
let reader = try AVAssetReader(asset: asset)
// Create AVAssetReaderTrackOutput and assign to reader
let readerOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: videoSettings)
reader.add(readerOutput)
// Start reading
reader.startReading()
// Create pixel buffer from sample buffer
if let sampleBuffer = readerOutput.copyNextSampleBuffer(),
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
// Release sample buffer
CMSampleBufferInvalidate(sampleBuffer)
// Return pixel buffer in YUV420 format
return pixelBuffer
} else {
print("Failed to create sample buffer")
return nil
}
} catch {
print("Failed to create AVAssetReader: \(error)")
return nil
}
}
Send the buffer:
if let zoomFrame = sample?.toZoomFrame() {
sendFrame(frame: zoomFrame)
}
Note that I could view the resulting zoomFrame in the debugger and it looked fine.
@chunsiong.zoom ,
I’m not sure I understand the issue. If the format is correct, the frame should look in grayscale (similar to the sample you showed), instead of its normal colors?
The format of the frame is kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
is there a specific format that sendVideoFrame expects?
I don’t have swift code, but here’s how I’ve managed to do it on c++ & windows.
The process should be similar. This code reads from a mp4 file, converts it frame by frame to yuv420 format and sends it
int video_is_playing = 1;
char* frameBuffer;
// Calculate the size of the YUV420 buffer based on frame dimensions
int frameLen = height / 2 * 3 * width; // Height/2 for subsampled U and V planes
frameBuffer = (char*)malloc(frameLen);
// Execute in a thread.
while (video_is_playing > 0 && video_sender) {
Mat frame; // Represents a single video frame
VideoCapture cap; // Represents a video capture device;
cap.open(video_source); //opens up a mp4 file from local storage path
if (!cap.isOpened()) {
cerr << "ERROR! Unable to open camera stream or video file\n";
video_is_playing = 0;
return;
}
// GRAB AND WRITE LOOP
cout << "Start grabbing" << endl;
while (cap.read(frame))
{
// Wait for a new frame from the camera and store it into 'frame'
// Check if we succeeded in reading a frame
if (frame.empty()) {
cerr << "ERROR! Blank frame grabbed\n";
break;
}
// Convert Mat to YUV buffer
Mat yuv;
cv::cvtColor(frame, yuv, COLOR_BGRA2YUV_I420);
// Get the YUV buffer data
char* frameBuffer = (char*)yuv.data;
int width = yuv.cols;
int height = yuv.rows;
int frameLen = yuv.total() * yuv.elemSize(); // Calculate YUV420 buffer size
// Send the YUV buffer to the video sender
SDKError err = ((IZoomSDKVideoSender*)video_sender)->sendVideoFrame(frameBuffer, width, height, frameLen, 0);
if (err != SDKERR_SUCCESS) {
cout << "sendVideoFrame failed: Error " << err << endl;
}
}
cap.release();
}
video_is_playing = -1;
@chunsiong.zoom
I see.
As far as I understand, all else being the same, if I use OpenCV to convert my camera frames from 32BGRA to YUV_I420, sendVideoFrame should work fine and I should receive frames on the other side which is subscribed to my videoPipe.
Anything else I should change?
Hello, @chunsiong.zoom
I was trying again with the video test that you mentioned, attempting to replicate your c++ conversion code in swift. However, in the process, I noticed that the frames coming from the video I downloaded are already in YUV420 bi-planar (full range) format. Is that the case or am I missing something? and if so, then there’s no need for the cv::cvtColor(frame, yuv, COLOR_BGRA2YUV_I420); step in the first place, correct?
@chunsiong.zoom
yes, I’m trying to send send the big buck bunny 720p 10s 1MB video frame by frame.
Here’s the full code for it:
import AVFoundation
import ZoomVideoSDK
class VideoProcessor {
var videoIsPlaying: Int = 1
var assetReader: AVAssetReader?
var asset: AVAsset?
var videoTrackOutput: AVAssetReaderTrackOutput?
private var frameSender: ZoomVideoSDKVideoSender?
init?(videoURL: URL, sender: ZoomVideoSDKVideoSender) {
frameSender = sender
self.asset = AVAsset(url: videoURL)
do {
try self.assetReader = AVAssetReader(asset: self.asset!)
} catch {
print("ERROR! Unable to open video file")
return nil
}
guard let videoTrack = self.asset?.tracks(withMediaType: .video).first else {
print("ERROR! Could not get video track")
return nil
}
let outputSettings: [String: Any] = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
]
self.videoTrackOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: outputSettings)
self.assetReader?.add(videoTrackOutput!)
}
func startProcessing() {
self.assetReader?.startReading()
while self.videoIsPlaying > 0 {
if let sampleBuffer = self.videoTrackOutput?.copyNextSampleBuffer() {
// Get the YUV buffer data
if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
let frameLen = CVPixelBufferGetDataSize(imageBuffer)
// Access the YUV data (ensure the memory block is locked for reading)
CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)
let rawPointer = CVPixelBufferGetBaseAddress(imageBuffer)
// Convert UnsafeMutableRawPointer to UnsafeMutablePointer<CChar>
let frameBuffer = rawPointer?.assumingMemoryBound(to: CChar.self)
// Send the YUV buffer to the video sender
sendVideoFrame(frameBuffer, width, height, frameLen, 0)
CVPixelBufferUnlockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)
}
CMSampleBufferInvalidate(sampleBuffer)
} else {
// End of file or an error occurred
self.videoIsPlaying = -1
}
}
}
func sendVideoFrame(_ frameBuffer: UnsafeMutablePointer<CChar>?, _ width: Int, _ height: Int, _ frameLen: Int, _ timestamp: Int64) {
if let sender = frameSender, let buffer = frameBuffer {
sender.sendVideoFrame(buffer, width: UInt(width), height: UInt(height), dataLength: UInt(frameLen), rotation: .rotationNone, format: .I420) // TODO: maybe change format?
print("Frame sent")
} else {
print("Sender is not initialized")
}
}
}
Note that since the frames from this video are already in bi-planar yuv420, I skipped the conversion step.
The issue persists, and I don’t seem to receive any frames on the other side.
