Initial rotation of my local video

[blindgaenger (Bernd)]

Hi there,

I’m using an iPhone in .portrait mode. When subscribing to my video and not handling any orientation then its initially show as rotated to .landscapeLeft. I assume this is the internal default orientation.

Is this the expected behaviour or am I doing something wrong?

Of course I’m observing orientation change, and call rotateMyVideo accordingly, which works fine. But I’m not sure when to initially set the orientation before any orientation change happens.

Before joining and onSessionJoin callback won’t update the video. And onUserVideoStatusChanged does work until I stop/start my video, then it’s in landscape mode again.

Should rotateMyVideo be called every time I subscribe to my video? Or is there a way to keep the orientation I’ve set?

[blindgaenger (Bernd)]

Unfortunately I’ve encountered a related issue of the late rotation of the local video.

A user subscribes to another participant’s video onUserJoin. Then for a short time the landscape video shows up, then the correctly rotated portrait one.

Beside that the aspect ratio is not recomputed, so the video on the receiver’ side shows up as letterbox (black bars). The video was subscribed with .panAndScan.

After toggling the video on the sender’s side the video shows up on the receiver correctly in portrait mode with the correct aspect.

I am grateful for any advice. Thanks!

[Michael_Condon (Michael)]

Hey @blindgaenger.

Thanks for using the dev forum!

At what point in your ViewControllers lifecycle are you handling the SDK views?

Thanks!
Michael

[blindgaenger (Bernd)]

Hi Michael,

That’s the right but a tough question as I’m using the SDK in a React Native module. So there is no separate ViewController and the UIView encapsulates all the logic.

The UIView is subscribed to the sharedInstance’s delegate events. These are the callbacks I’m using.

func onSessionJoin() {
  // subscribe in onUserVideoStatusChanged as video could be off or unavailable
}

func onSessionLeave() {
  if let user = self.user {
    unsubscribeVideo(user: user)
  }
}

func onUserVideoStatusChanged() {
  if let myself = observable.getMySelf(),
    let isVideoEnabled = myself.videoStatus()?.on {
    if isVideoEnabled {
      updateVideoOrientation()
      subscribeVideo(user: myself)
    } else {
      unsubscribeVideo(user: myself)
    }
  }
}

Of course the view is instantiated before joining a session. I’m planing to show the local video preview in the very same view. But I need to know when it’s safe to switch from preview to live video.

[Michael_Condon (Michael)]

Hey @blindgaenger,

I see. In a native implementation of the video SDK, the rotation can be incorrect if the view is set up when the ViewController is loaded instead of when the ViewController appears. I am not to sure how reactnative manages this pattern but if it is possible to set up the view later that may help.

@blindgaenger

But I need to know when it’s safe to switch from preview to live video.

It should be safe after onSessionJoin has been called.

Thanks!
Michael

[blindgaenger (Bernd)]

Hi @Michael_Condon ,

Thanks for your reply!

I had another look at the ZoomInstantSample application on how the video subscription was implemented. There is a deadline of 200 ms on the main thread used between creating+subscribing the view and adding it as subview. I’m not 100% convinced that’s the proper way, but at least the video shows up consistently on my local device.

Using the very same code the remote device is not always picking up the video, though. And the rotation is in landscape mode again if I toggle the video, although the initial rotation works.

I think I’ve missed an important aspect of the whole flow. Is there a stripped down example on how and when the views should be subscribed, rotated, released?

[Michael_Condon (Michael)]

Hey @blindgaenger,

Yes, I would put this code in viewDidAppear:

if let usersVideoCanvas = user.getVideoCanvas() {
    // Set video aspect.
    let videoAspect = ZoomInstantSDKVideoAspect.panAndScan

    // Subscribe User's videoCanvas to render their video stream.
    usersVideoCanvas.subscribe(with: view, andAspectMode: videoAspect)
}

Thanks!
Michael

[blindgaenger (Bernd)]

Hi @Michael_Condon ,

Thanks for your answer!

After some research seems there is no way to hook into viewDidAppear without a view controller. React Native’s approach is that the views are solely controlled from JavaScript, so I refactored my current approach and gave the control back to JS. Downside are a couple of milliseconds delay as the events have to pass the React Native bridge.

So the first problem with the local view got solved. But the remote view still uses the aspect ratio from before the rotation, although it shows up correctly. After toggling the video the aspect ratio gets fixed (recalculated?).

Hope you’ve got any ideas on that. Further is there a way to inspect the “internal” dimensions of a view for debugging?

Cheers,
Bernd

[Michael_Condon (Michael)]

Hey @blindgaenger,

I would try and call setVideoAspect again for the remote view. As to where, I am not entirely sure since I do not know how reactnative handles the views lifecycle.

Thanks!
Michael

[blindgaenger (Bernd)]

Hi @Michael_Condon ,

I really appreciate you sticking with this topic.

The local video works correctly now with the following code in the UIView. Even without calling rotateMyVideo upfront.

override func layoutSubviews() {
  self.frame = self.frame
  self.bounds = self.bounds
}

Unfortunately, I have not been able to solve the video aspect for remote users yet. It’s clearly a problem of when React Native sets the view’s frame. I will try to figure this out.

Thanks again!

Cheers,
Bernd

[Michael_Condon (Michael)]

Hey @blindgaenger,

You are very welcome
Please let me know what you find.

Thanks!
Michael

[blindgaenger (Bernd)]

Hi @Michael_Condon ,

Unfortunately, I still haven’t managed to solve the issue properly. I’ve hard-coded the frame, but still no luck. The rotateMyVideo call does not seem to have any effect. I’ve put it everywhere, but keep it in onSessionJoin.

I discovered that it sometimes occurs when starting/stopping the video using the video helper, too. Both views are subscribed the whole time and the frame and bounds don’t change.

I’ve created a test screen for easier debugging. Here the remote video (on the right) fills the view.

This happens when the user joins or after toggling the remote video on/off a couple of times. Also the video rotates very quickly from landscape to portrait.

I need to understand the lifecycle. What actually determinates the dimensions of the received video? Is it my local view or the remote one? Is there a way to invalidate the received video and request to a re-layout?

I really hope you can help me, but this is a huge blocker to me.

Cheers,
Bernd

[Michael_Condon (Michael)]

Hey @blindgaenger,

The local view will determine the dimensions of frame that the received video is contained in. If possible the received video will be stretched/shrunk inside of the local frame depending on what aspect ratio the SDK is told to use. Would you be able to run your app once more and obtain the SDK logs from that instance, then email them to me at DeveloperSupport@zoom.us? Normally we do not provide much support for reactnative but the logs might be able to shed some light on what is going on here.

Thanks!
Michael

[blindgaenger (Bernd)]

Hi @Michael_Condon ,

I’ve send you the logs and a video to give more context. Thanks a lot!

Cheers,
Bernd

[Michael_Condon (Michael)]

Hey @blindgaenger,

You are welcome I will follow up here with what we find.

Thanks!
Michael

[blindgaenger (Bernd)]

Hi @Michael_Condon ,

Is there any news? Have you been able to find out anything?

I’ve reimplemented the debug screen in a fresh Swift project and with a ViewController, but still the same result. Therefore, I can assure that it is not a problem of React Native.

It’s reproducible that the local video always shows up currently, even after “muting” it. The rotation can can be controlled with rotateMyVideo after joining the session.

The remote video on the other hand always shows up as letterbox, and after calling stopVideo/startVideo the video fills the view.

// must be called before subscribe to rotate the view correctly
// but will not fix the aspect ratio issue, when start/stop the video
DispatchQueue.main.async {
  ZoomInstantSDK.shareInstance()?.getVideoHelper().rotateMyVideo(.portrait)
}
    
let videoView = UIView()
// hard-coded for demonstration, but would use `view.bounds`
videoView.frame = CGRect(x: 0, y: 0, width: 100, height: 100)
view.addSubview(videoView)

DispatchQueue.main.asyncAfter(deadline: .now() + 2.0) {
  user.getVideoCanvas().subscribe(with: videoView, andAspectMode: .original)
}

I’ve tested all aspect modes and they always change the local view, but never the remote view. I use the same implementation of course.

Maybe you have an idea how to get the behavior at least consistent.

Cheers,
Bernd

[Michael_Condon (Michael)]

Hey @blindgaenger,

Here is my ViewController code, maybe this will shed some light on the issue here:

import UIKit
import ZoomInstantSDK

class ViewController: UIViewController {

    // MARK: UIViewController

    override func loadView() {
        super.loadView()

        ZoomInstantSDK.shareInstance()?.delegate = self
    }

    override func viewDidLoad() {
        super.viewDidLoad()

        joinSession()
    }

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)

        setupViews()
    }

    // MARK: Views

    lazy var selfVideoCanvasView: UIView = {
        let view = UIView(frame: self.view.bounds)
        view.backgroundColor = .black
        return view
    }()

    lazy var selfVideoCanvas: ZoomInstantSDKVideoCanvas? = {
        let selfVideoCanvas = ZoomInstantSDK.shareInstance()?.getSession()?.getMySelf()?.getVideoCanvas()
        return selfVideoCanvas
    }()

    lazy var otherUserVideoCanvasView: UIView = {
        let view = UIView(frame: CGRect(x: 0, y: 0, width: self.view.frame.width / 4, height: self.view.frame.height / 4))
        view.backgroundColor = .black
        return view
    }()

    lazy var otherUserVideoCanvas: ZoomInstantSDKVideoCanvas? = {
        let otherUser = ZoomInstantSDK.shareInstance()?.getSession()?.getAllUsers()?.first { $0 != ZoomInstantSDK.shareInstance()?.getSession()?.getMySelf() }
        let otherUserVideoCanvas = otherUser?.getVideoCanvas()
        return otherUserVideoCanvas
    }()

    lazy var debugStackView: UIStackView = {
        let stackView = UIStackView()
        stackView.axis = .vertical
        stackView.distribution = .equalSpacing
        stackView.alignment = .center
        stackView.spacing = 5.0

        stackView.addArrangedSubview(leaveSessionButton)
        stackView.addArrangedSubview(toggleAudioButton)
        stackView.addArrangedSubview(sendChatMessageToAllButton)
        stackView.addArrangedSubview(sendChatMessageToOtherUserButton)
        stackView.addArrangedSubview(amIHostLabel)
        stackView.addArrangedSubview(makeOtherUserHostButton)
        stackView.addArrangedSubview(shareAViewButton)
        stackView.addArrangedSubview(switchCameraButton)

        stackView.translatesAutoresizingMaskIntoConstraints = false

        return stackView
    }()

    lazy var leaveSessionButton: UIButton = {
        let button: UIButton = UIButton()
        button.backgroundColor = .black
        button.setTitle("Leave Session", for: .normal)
        button.addTarget(self, action:#selector(self.leaveSessionbuttonClicked), for: .touchUpInside)
        return button
    }()

    lazy var toggleAudioButton: UIButton = {
        let button: UIButton = UIButton()
        button.backgroundColor = .black
        button.setTitle("Toggle Audio", for: .normal)
        button.addTarget(self, action:#selector(self.toggleAudioButtonClicked), for: .touchUpInside)
        return button
    }()

    lazy var sendChatMessageToAllButton: UIButton = {
        let button: UIButton = UIButton()
        button.backgroundColor = .black
        button.setTitle("Send Chat Message To Everyone", for: .normal)
        button.addTarget(self, action:#selector(self.sendChatMessageToEveryoneButtonClicked), for: .touchUpInside)
        return button
    }()

    lazy var sendChatMessageToOtherUserButton: UIButton = {
        let button: UIButton = UIButton()
        button.backgroundColor = .black
        button.setTitle("Send Chat Message To Other User", for: .normal)
        button.addTarget(self, action:#selector(self.sendChatMessageToOtherUserButtonClicked), for: .touchUpInside)
        return button
    }()

    lazy var amIHostLabel: UILabel = {
        let label: UILabel = UILabel()
        label.backgroundColor = .black
        label.text = "Am I the host? Unknown"
        return label
    }()

    lazy var makeOtherUserHostButton: UIButton = {
        let button: UIButton = UIButton()
        button.backgroundColor = .black
        button.setTitle("Make Other User Host", for: .normal)
        button.addTarget(self, action: #selector(self.makeOtherUserHostButtonClicked), for: .touchUpInside)
        return button
    }()

    lazy var shareAViewButton: UIButton = {
        let button: UIButton = UIButton()
        button.backgroundColor = .black
        button.setTitle("Share a view", for: .normal)
        button.addTarget(self, action: #selector(self.shareAViewButtonClicked), for: .touchUpInside)
        return button
    }()
    
    lazy var switchCameraButton: UIButton = {
        let button: UIButton = UIButton()
        button.backgroundColor = .black
        button.setTitle("Share a view", for: .normal)
        button.addTarget(self, action: #selector(self.switchCameraButtonPressed), for: .touchUpInside)
        return button
    }()

    lazy var dummyShareView: UIView = {
        let view = UIView(frame: CGRect(x: 100, y: 0, width: self.view.frame.width / 4, height: self.view.frame.height / 4))
        view.backgroundColor = .red
        return view
    }()

    // MARK: Actions
    
    @objc func switchCameraButtonPressed() {
        let result = ZoomInstantSDK.shareInstance()?.getVideoHelper().switchCamera()
        print(result)
    }

    @objc func makeOtherUserHostButtonClicked() {
        print(ZoomInstantSDK.shareInstance()?.getSession()?.getAllUsers())
        print(ZoomInstantSDK.shareInstance()?.getSession()?.getMySelf())
        print(ZoomInstantSDK.shareInstance()?.getSession()?.getHost())
        if let me = ZoomInstantSDK.shareInstance()?.getSession()?.getMySelf() {
            print(ZoomInstantSDK.shareInstance()?.getUserHelper().changeName("newName", with: me))
        }
    }

    @objc func shareAViewButtonClicked() {

    }

    // MARK: Internal

    var rawDataSender: ZoomInstantSDKVideoSender?

    func setupViews() {
        self.view.addSubview(selfVideoCanvasView)
        self.view.addSubview(otherUserVideoCanvasView)
        self.view.addSubview(debugStackView)
        self.view.addSubview(dummyShareView)

        debugStackView.centerXAnchor.constraint(equalTo: self.view.centerXAnchor).isActive = true
        debugStackView.centerYAnchor.constraint(equalTo: self.view.centerYAnchor).isActive = true

        selfVideoCanvas?.subscribe(with: selfVideoCanvasView, andAspectMode: .panAndScan)
        otherUserVideoCanvas?.subscribe(with: otherUserVideoCanvasView, andAspectMode: .panAndScan)
    }
}

Thanks!
Michael

[blindgaenger (Bernd)]

Hi @Michael_Condon ,

Thanks a lot for the example!

Right off the bat, I can’t see where I’m doing things differently. Except that I rely on the callbacks.

Unfortunately I have little time to test the code in the project. But I will come back to it.

Cheers,
Bernd

[Michael_Condon (Michael)]

Hey @blindgaenger,

Awesome! Let me know when you have a chance to test things out.

Thanks!
Michael