Cannot get Camera Mode to work in Layers API

Context:
Working with NodeJS and VueJS

Description
So the issue I’m experiencing at the moment that can be forwarded to the zoom developers’ team are the following:

  • Camera mode never gets initiated on my mac after calling runRenderingContext with options { view: “camera” }. If I change the view value to immersive { view: “immersive” }, I can see that the runningContext changes to “inImmersive” mode whereas I do not get “inCamera” mode whenever I set view to camera. Also, I’d be glad if there is a way to use developer tools in camera mode as it isn’t very helpful that one cannot determine when runRenderingContext has switched to “inCamera”.

  • When i’m using the drawWebView and drawImage APIs, none of them seems to work, but while testing drawParticipant API, it works in Immersive mode so it is not clear why both drawWebView and drawImage wouldn’t work in camera mode, and these two are important in rendering the webview (the feature). This issue might also be because the camera mode is not being initiated as discussed in the first point.

How To Reproduce
Call runRenderingContext with camera mode. Currently both drawWebView and drawImage are not working after calling runRenderingContext

await zoomSdk.runRenderingContext({ view: ‘camera’ })
.then(async(ctx) => {
console.log(“runRenderingContext returned”, ctx);
})
.catch(async(e) => {
console.log(e);
});

Then call drawWebView

await zoomSdk.drawWebView({
webviewId: “speaking-time-overlay”,
x: 0,
y: 0,
width: 300,
height: 300,
zIndex:2
})
.then(async(ctx) => {
console.log(“drawWebView returned”, ctx);
})
.catch((e) => {
console.log(e);
});

OR call

await zoomSdk.drawImage({
imageData: imageData,
x: 0, y: 0, zIndex:3
})
.then((ctx) => {
console.log(“drawImage returned imageID”, ctx);
console.log(“drawImage returned imageID”, ctx.imageId);
})
.catch((e) => {
console.log(e);
});

This is the getImageData function
const getImageData = (width, height) => {
const canvas = document.createElement(“canvas”);
canvas.width = width;
canvas.height = height;

const img = new Image();
img.src = "HowTo2.png"; // our image url - change baseurl

const ctx = canvas.getContext("2d");
ctx.drawImage(img, 0, 0, width, height);
return ctx.getImageData(0, 0, width, height);

};

Did you call zoomSdk.config with the capabilities first? I am also coding in Vue.js and intialize the application like

      zoomSdk.config({
        popoutSize: { width: 480, height: 360 },
        capabilities: [
          'getMeetingUUID',
          'getRunningContext',
          'getMeetingContext',
          'runRenderingContext',
          'closeRenderingContext',
          'drawParticipant',
          'clearParticipant',
          'drawImage',
          'clearImage',
          'drawWebView',
          'clearWebview',
          'postMessage',
          'sendAppInvitationToAllParticipants',
          'getVideoState',
          // Events
          'onRenderedAppOpened',
          'onMessage',
          'onMyMediaChange',
          "onSendAppInvitation",
          "onAppPopout"
        ],
      }).then(
        (configResponse) => {
          this.inZoom = true
          this.checkContext().then(() => {
            if (this.controllerMode) {
              zoomSdk.getVideoState().then((state) => {
                if (state.video) {
                  this.startCamera()
                }
              })
              zoomSdk.onMyMediaChange((event) => {
                if (event.media.video.state) {
                  this.startCamera()
                } else {
                  this.stopCamera()
                }
              })
            }
            if (this.cameraMode) {
              zoomSdk.getVideoState().then((state) => {
                if (state.video) {
                  this.renderWebView()
                }
              })
              zoomSdk.onMyMediaChange((event) => {
                if (event.media.video.state) {
                  this.renderWebView()
                }
              })
            }
          })
        }
      ).catch(e => {
        this.controllerMode = false
        this.inZoom = false
      })
    }

Render web view is like

    renderWebView() {
      zoomSdk.drawWebView({
        webviewId: 'MyCamera',
        x: 0, y: 0, width: 1280, height: 720, zIndex: 5
      })
      .then((ctx) => {
        this.logger("Web View Rendered")
      })
      .catch((e) => {
        // console.log(e);
      });
    },

The web view is another browser window running the same app but it’s console log you can not see. You can use zoomSdk.onMessage to send message between the two. So my component has a template like:

  <div class="container">
    <zoom-camera-app v-if="cameraMode"></zoom-camera-app>
    <zoom-controller-app v-if="controllerMode"></zoom-controller-app>
  </div>

drawWebView doesn’t work in immersive mode, it only works in camera mode.

drawImage and drawParticipant should work in both immersive and camera mode.

If you call the drawImage, or drawParticipant functions from the sidebar (inMeeting) or the immersive window (inImmersive) they only affect the immersive view.

We are working on docs in this area!

1 Like

Thank you so much @henry2. Super helpful.

How do i load dynamic content into the webview considering there’s no place to retrieve webviewId or would you suggest adding the code in the then block (where you added this.logger in your code)? Switching the view in my template as explained in your example affects the side bar. The side bar currently displays what i want to show in the camera mode on my webview.

Please help. Thank you

As Jon states you can not do both camera and immersive at the same time.

Your code can check to see if you are in the camera or panel using getVideoState

zoomSdk.getVideoState().then((state) => {
  if (state.video) {
       this.renderWebView()
  }
})

That is also what I effectively use to determine which component to display in my template (panel and camera are in separate components)

Then if I have a button or something that initiates a display in the camera I use zoomSdk.postMessage and zoomSdk.onMessage to send a message between the panel and the camera components. Post message can send a JSON structure so your on message and determine what to do based on that.

btw - I found that for some reason with Vue3 that the message received was a string instead of JSON as per the Zoom doc. So I parse it back into a JSON structure if it was not received as such:

processMessage({payload}) {
      let pl = (typeof payload == "string") ? JSON.parse(payload) : payload
      // For some reason the payload is a string form of the json
      // ......
 }

Thanks @henry2

I understand what you said and to be clear, I’m not using immersive mode with camera mode. All i want to do is use camera mode, I only switch to immersive mode to confirm if the code is running in another mode as it should. The issue i’m having is based on the example code that you provided above, after callingRunRenderingContext, it looks like camera view is not being rendered, even by specifying this {view: “camera”}. I’m testing with the drawImage function, so I do this:

zoomSdk.getVideoState().then((state) => {
  if (state.video) { // remember this is calling view: "camera" and by now should be in inCamera mode
       drawImage()
  }
})

The above code does not work in camera view. But if i change runRenderingContext to {view: “immersive”}, i see the image. Note: I only switch to immersive mode to confirm if the code is working and i’m testing with drawImage function since it works in both immersive and camera view.

Please help. Also, i see the template i want to show in camera view on the canvas when i switch to immersive, just for testing BUT drawWebView and drawImage are not working in cameraMode with the same code.

I use a method to check the running context that initialises the state. I contains code such as

zoomSdk.getRunningContext().then(
  (ctx) => {
    this.controllerMode = ctx.context == 'inMeeting';
    this.cameraMode = ctx.context == 'inCamera';
  }
).catch(
  (e) => {
    console.debug("*** ", e)
  }
)

Then I use the zoomSdk.drawWebView if the running context is inCamera. It takes a few seconds for the display to in the layer over the camera, but it does work. The camera has to be running, so I also have code that tests the video state and when the state is video I start the running context and when the state changes (camera start or stop) I start or top the running context.

I think the key is that the start the running context has to happen after the camera is turned on.

2 Likes

Thank you @henry2 for all your help. I’d try this again.

1 Like

hi i need help with somethin

Please create a new topic and include details about your issue and we’ll work to assist you in that thread

Hi Henry, the code snippets you paste in that topic, may i know what you are trying to display on zoom video, I also have a similar thing i need to print timer on Zoom video. so can you help me please thank you?

Hi,

I am displaying votes in the video window. When the user clicks one of various buttons in the side panel the vote will appear in their video.

The drawWebView function will load the same web page that was loaded in the panel. Hence I use zoomSdk.getRunningContext() to determine the context and display what is needed depending on whether the code is inCamera, inMeeting or inMainClient. When the meeting is started the context will be inMeeting for the side panel and when the drawWebView is used and video is on then the context is inCamera.

I also setup the event listeners using onMessage so that the app in the panel can sent messages to the app in the camera hence control what is displayed. You could do the same to start and stop a timer that is displayed on top of the video.

You can get the video webview’s width and height if needed from the zoom config (returned from the zoomSdk.config() call. Also set the zIndex of the webview so it is above the video.
My code for that looks like (I stored the zoom config as a variable of my component)

zoomSdk.drawWebView({
        webviewId: 'MyCamera',
        x: 0, y: 0, 
        width:  this.config.media.renderTarget.width,
        height: this.config.media.renderTarget.height, 
        zIndex: 2
})

btw - I only call drawWebView when the camera for the context inCamera. And that is initiated by the runRenderingContext({view: 'camera'}) from the inMeeting app.
For a guest application the call to get the video state does not work, so I call the runRenderingContext and if the video is not on ignore the exception. I also use onMyMediaChange (which is allowed for guests) to detect when the video is turned on and then initiate the runRenderingContext.

@henry2 Wow henery you are a rockstar, just because of you I understand how I can communicate between the side panel app and the camera mode app, I am using this zoom sample app for starting point GitHub - zoom/zoomapps-cameramode-vuejs: A frontend only sample that shows how to use Camera Mode and Immersive Mode in a Zoom App

and I am able to draw the timer i have some doubts like onMyMediaChange Where should i put this event according to the sample app and what code will be there, i want it will work smoothly like if user videos off just alert the user and if the user will on the video just show the timer without breaking the whole application. do you have any suggestions regarding this Henery?

And also is there a way to optimize a little currently in my side panel app there is a button show timer when i click on this it will call drawWebView which takes some time to show a timer so why it takes time to load? thanks for the help :slightly_smiling_face:!

Glad you found the information useful. I have code like the following for onMyMediaChange:

zoomSdk.config({
// list of capabilities including 'onMyMediaChange'
}).then{
  (cfgResponse) => {
     // use zoomSdk.getAppContext to see if we are in the side panel (inMeeting)
     // if we are in the meeting (not the video then use onMyMediaChange
                zoomSdk.onMyMediaChange((event) => {
                  if (event.media.video.state) {
                     // logic for zoomSdk.runRenderingContext({view: 'camera'})                    
                  } else {
                     // logic for zoomSdk.closeRenderingContext() 
                  }
                })

  }
)

The drawWebView takes time because it is reloading the webpage and JS effectively in a new window. I suspect that if you have cache headers in your directive the zoom app can make use of the local “browser” cache when loading the page which may speed things up a bit, and then it will depend on your JS code and the rendering time.

Hi @henry2 How are you ?
I am confused a bit like i play around with Zoom sample app and add some additional features like timer or so. now i want to actually integrate into my main project which built on react.

so I am wondering how you setup zoom configuration like i want to setup in one file all the functions and if the view in camera i have a component i will call this component,

can you share your configuration like a dummy code please hide your app code i just want to see how you setup zoom functions and how you add conditions like when in camera mode or in meetings how you display content etc.

it would be really helpful for me if you do this actually I am a beginner in coding so i would appreciate your effort and help :slightly_smiling_face:

I am not using react, but am using Vue.js. The logic should be similar. I put the zoom code in a mixin which I include in my Vue components. The state is stored in a Vuex store (which is effectively the local storage of the browser). Below is (I think) the parts of the zoom mixin I wrote which should help you.

I call zoomInit for the mount method of the main app. The map state and mutations is for access to the Vuex store - in react you probably would use zustand or other libraries to provide a similar functionality.

export const zoomMixin = {
  methods: {
    computed: {
      ...mapState([
        'inZoom',
        'inMainClient',
        'controllerMode',
        'cameraMode',
        'cameraInitialized',
        'zoomAppContext',
      ])
    },
    ...mapMutations({
      setInZoom: SET_IN_ZOOM,
      setInMainClient: SET_IN_MAIN_CLIENT,
      setControllerMode: SET_CONTROLLER_MODE,
      setCameraMode: SET_CAMERA_MODE,
      setCameraInitialized: SET_CAMERA_INITIALIZED,
      setZoomAppContext: SET_ZOOM_APP_CONTEXT,
    }),
    // Start the web view within the Camera
    renderWebView() {
      return zoomSdk.drawWebView({
        webviewId: 'MyCamera',
        x: 0, y: 0,
        width:  this.config.media.renderTarget.width,
        height: this.config.media.renderTarget.height,
        zIndex: 2
      }).catch((e) => {
        // this.logger("Web View Errored")
        console.log(e);
      });
    },
    // Start the camera view - if the camera is not already initialized
    startCamera() {
      return new Promise((res, rej) => {
        if(this.controllerMode && !this.cameraInitialized) {
          // NOTE: this starts a new app instance in the camera requesting the SAME endpoing...
          zoomSdk.runRenderingContext({view: 'camera'}).then((ctx) => {
            this.setCameraInitialized(true)
            res(ctx)
          }).catch(rej)
        } else {
          res({});
        }
      })
    },
    // Stop the camera view - if the camera is already initialized
    stopCamera() {
      return new Promise((res, rej) => {
        if(this.controllerMode && this.cameraInitialized) {
          zoomSdk.closeRenderingContext().then((ctx) => {
            this.setCameraInitialized(false)
            res(ctx)
          }).catch(rej)
        } else {
          res({});
        }
      })
    },
    // Check which context the app is running in
    checkContext() {
      return zoomSdk.getRunningContext().then(
        (ctx) => {
          this.setInMainClient(ctx.context == 'inMainClient')
          this.setControllerMode(ctx.context == 'inMeeting')
          this.setCameraMode(ctx.context == 'inCamera')
        }
      ).catch(
        (e) => {
          this.setControllerMode(true)
        }
      )
    },
    getAppContext() {
      return zoomSdk.getAppContext().then(
        (ctx) => { this.setZoomAppContext(ctx); }
      ).catch(
        (e) => { console.debug("*** ", e) }
      );
    },
    // Pass in methods to call after initialization or we can do via then()
    zoomInit() {
      return zoomSdk.config({
        popoutSize: { width: 480, height: 720 },
        capabilities: [
          'getMeetingUUID',
          'getRunningContext',
          'getMeetingContext',
          'getUserContext',
          'runRenderingContext',
          'closeRenderingContext',
          'showNotification',
          'drawWebView',
          'clearWebview',
          'postMessage',
          'sendAppInvitationToAllParticipants',
          'getVideoState',
          'onBreakoutRoomChange',
          'onRenderedAppOpened',
          'onMessage',
          'onMyMediaChange',
          "onSendAppInvitation",
          "onAppPopout",
          "onMyUserContextChange"
        ],
      }).then(
        (configResponse) => {
          this.setInZoom(true)
          this.config = configResponse

          // getAppContext - to pass back to the server on cable connection
          return this.getAppContext().then( () => {
            this.checkContext().then(() => {
              if (this.controllerMode) {
                // Get Video State does not work for Guest mode
                // to get round this just start the camera anyway
                this.startCamera().catch(() => { })
                // Listen for media change events - start/stop of video
                zoomSdk.onMyMediaChange((event) => {
                  if (event.media.video.state) {
                    this.startCamera()
                  } else {
                    this.stopCamera()
                  }
                })
              }
              if (this.cameraMode) {
                this.renderWebView()
              }
            })
          })
        }
      ).catch(e => {
        console.debug("**** Failed to initialize Zoom!")
        this.setControllerMode(true)
        this.setInZoom(false)
      })
    }
  }
}

FYI - my “controller mode” is just my main app (just in case you are wondering)

Thank you so much @henry2 you are a really helpful person you should have been on the Zoom team, I play around with this code and I can display a timer, but it takes a lot of time. is it because my app size is large? also, I am unable to see the dev console on the Zoom app how can we open it? and henry is there a way to have a notification on zoom app and how can we send notification to others participants?

There was another thread on enabling the dev console. Once done you will be able to see the dev console just like a browser for the app in the panel on the side. (see How to Access Console While Developing Zoom App)

You are not able to see the dev console for the camera view. But you can send messages using post message between the app running in the camera and the one in the side panel. (see How to inspect element on the camera layer - #5 by henry2)

As for speed of the app - the camera view is loading the JS again and I assume that if the files have cache headers it probably does not need to make a network request which I think will help. But I also find the display in the take multiple seconds for the initial load when I am developing.

As for messages between participants - that is not a zoom api issue per se. I am using web-sockets with a ruby backend for that at the moment. You can do similar with node or something. You will need to do some server side coding - regular web stack coding, or integrate with another service for messages.

oK, again thanks @henry2 for your Help.
I tried to find the dev console but I couldn’t succeed please just provide some more info regarding this.
one question @henry2 have you used setVirtualForeground method of zoom ? i believe it could be really fast than drawWebView ? and if you know about timer app which is build by zoom you notice how fast the timer works on video feed when we start a time / off a timer whatever…
i read their developer documentation they are mentions what they are using please see the below image.

now I am wondering what if i used this method to show a timer on video feed because its fast, so do you know how can i implement this thanks for always help :slightly_smiling_face: