Converting Zoom Video Raw data into video frames

Video SDK
1

I’m working on a macOS application using the Zoom Video SDK, and as per the SDK documentation, the user video is received in the ZMVideoSDKYUVRawDataI420 format from onRawDataFrameReceived which is mention on here(Video SDK - macOS - Video) However, the documentation doesn’t explain how to convert this I420 raw data into video frames for display in a view. How to achieve this and show the user video in a view?

I am using following code to convert it into frames

  func renderVideoFrame(_ frame: ZMVideoSDKYUVRawDataI420) {
        let width = Int(frame.streamWidth)
        let height = Int(frame.streamHeight)
        let yPlane = frame.yBuffer
        let uPlane = frame.uBuffer
        let vPlane = frame.vBuffer
        guard let rgbData = convertYUVtoRGB(yPlane: yPlane, uPlane: uPlane, vPlane: vPlane, width: width, height: height) else {
            return
        }
        
        let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.noneSkipFirst.rawValue)
        let colorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: rgbData, width: width, height: height, bitsPerComponent: 8, bytesPerRow: width * 4, space: colorSpace, bitmapInfo: bitmapInfo.rawValue)


        if let cgImage = context?.makeImage() {
            let nsImage = NSImage(cgImage: cgImage, size: NSSize(width: width, height: height))
            DispatchQueue.main.async {
                self.viewController.displayImage(nsImage: nsImage)
            }
        }
    }

    func convertYUVtoRGB(yPlane: UnsafeMutablePointer<CChar>?, uPlane: UnsafeMutablePointer<CChar>?, vPlane: UnsafeMutablePointer<CChar>?, width: Int, height: Int) -> UnsafeMutablePointer<UInt8>? {
        guard let yPlane = yPlane, let uPlane = uPlane, let vPlane = vPlane else {
            return nil
        }

        let rgbSize = width * height * 4
        let rgbData = UnsafeMutablePointer<UInt8>.allocate(capacity: rgbSize)

        DispatchQueue.concurrentPerform(iterations: height) { y in
            for x in stride(from: 0, to: width, by: 2) {
                let yIndex = y * width + x

                // Calculate chroma indices for the subsampled U and V planes
                let chromaX = x / 2
                let chromaY = y / 2
                let uIndex = chromaY * (width / 2) + chromaX
                let vIndex = uIndex

                // Fetch U, V values
                let U = Int(UInt8(bitPattern: uPlane[uIndex])) - 128
                let V = Int(UInt8(bitPattern: vPlane[vIndex])) - 128

                // Precompute UV components
                let v1 = 5727 * V
                let uv1 = -(1617 * U) - (2378 * V)
                let u1 = 8324 * U

                // Process even pixel
                var Y = Int(UInt8(bitPattern: yPlane[yIndex])) << 12
                var R = (Y + v1) >> 12
                var G = (Y + uv1) >> 12
                var B = (Y + u1) >> 12

                // Clamp RGB values to valid range [0, 255]
                R = min(max(R, 0), 255)
                G = min(max(G, 0), 255)
                B = min(max(B, 0), 255)

                // Store the RGB values in the output buffer
                var rgbIndex = yIndex * 4
                rgbData[rgbIndex] = UInt8(R)
                rgbData[rgbIndex + 1] = UInt8(G)
                rgbData[rgbIndex + 2] = UInt8(B)
                rgbData[rgbIndex + 3] = 0  // Alpha value

                // Process odd pixel
                Y = Int(UInt8(bitPattern: yPlane[yIndex + 1])) << 12
                R = (Y + v1) >> 12
                G = (Y + uv1) >> 12
                B = (Y + u1) >> 12

                // Clamp RGB values to valid range [0, 255]
                R = min(max(R, 0), 255)
                G = min(max(G, 0), 255)
                B = min(max(B, 0), 255)

                // Store the RGB values in the output buffer
                rgbIndex = (yIndex + 1) * 4
                rgbData[rgbIndex] = UInt8(R)
                rgbData[rgbIndex + 1] = UInt8(G)
                rgbData[rgbIndex + 2] = UInt8(B)
                rgbData[rgbIndex + 3] = 0  // Alpha value
            }
        }

        return rgbData
    }
2

I think what you are looking for is here - Video SDK - macOS - Raw data

// Used to receive video's YUV420 data.
- (void)onRawDataFrameReceived:(ZMVideoSDKYUVRawDataI420*)data {
    // Access the raw data for each of the 3 components.
    char *yBuffer = [data yBuffer];
    char *uBuffer = [data uBuffer];
    char *vBuffer = [data vBuffer];

    // Get format of raw data.
    bool isVideoFormatLimitedI420 = [data isLimitedI420];

    // Get rotation of raw data video stream.
    unsigned int rotation = [data rotation];
}

// Called when the raw data has been toggled.
- (void)onRawDataStatusChanged:(ZMVideoSDKRawDataStatus)status {
    switch (status) {
        case ZMVideoSDKRawData_On:
            // User's raw data is now on.
            break;
        case ZMVideoSDKRawData_Off:
            // User's raw data is now off.
            break;
    }
}

// Pass the delegate into the video pipe of a specific user (e.g. myself)
ZMVideoSDKUser *user = [session getMySelf];
[user.getVideoPipe subscribe:ZMVideoSDKResolution_720P listener
:delegate];
3

Hey @kellyjandrews
What I was asking is how I can convert ZMVideoSDKYUVRawDataI420 raw data into video frames or RGB values to show to the user. Basically, I am creating a video call application where I need to display the user’s video on the screen.

1 Like
4

Ok - I may not be an expert here - but I will see if I can find out before someone who is answers first :slight_smile:

1 Like