[Request] add more resources to Tokyo data center / gateway

Client Web SDK
#1

I know this has nothing to do with web SDK but there are no sub section which fit for this kind of topic so please forgive me to post that here and feel free to move it to a better section.

Description

We are a Japanese company with 100% of our activities here in Japan. So our zoom account is setup to enable JP as available region for gateway / data center ping when connecting to Zoom. US is the default and non modifiable fallback.

From mid August to beginning of September, most if not all of our meeting were hosted in Japan region which was perfect. Then from beginning of September we noticed that most of the meeting started to be hosted in US instead.

From beginning of September, our users started to report audio quality degradation on Zoom services, and we suspect that being redirected to US data center is part of the cause. As you can see in the following screenshot, users can have more than half a second of latency to reach these regions.

image
image865Ă—142 21.1 KB

Since few days, we changed our settings to enable Hong Kong region as this is closer so should be better. But this is not going to fix the root cause.

Here are our concerns regarding this situation:

  • Why a 404 is returned on the ping ? I guess your did not remove that data center as we sometime can join it. So this is to say “i’m not available”, right ? Can you instead return a 503 which is the code to say that you are not available due to overload ?
  • Is it possible to increase resources allocated to Tokyo data center / gateway to accept more traffic for this region ?

Please understand that this has an impact in term of service quality delivered to our customers.

#2

Another information:

It looks like it’s only during business peak hours that we cannot use Tokyo. Right now at 8:24 am for example, all meetings are hosted on TK data center.
But starting from 10am to probably around 10pm it will be 95% out of Japan.

So the resources required to absorb the load in JP appear to be underestimated compared to the actual demand.
Is it possible to review it ?

#3

Hey @nvivot,

Happy to get this into the right hands!

@yosuke.sawamura, do you have any idea of what is going on here?

Thanks,
Tommy

#4

Hi nvivot,

There are three account and two are free trial accounts.
Free trial accounts have lower prioritized using our infrastructure due to the fact its free.

Can you provide one of the the meeting id as an example from what you are stating?
If those informations are sensitive then please use developersupport@zoom.us to share us.

Regards,
Yosuke

#5

I’m only talking about our production account, not the other ones.

I sent you an email to provide your a meeting ID.

Regards,
Nicolas Vivot.

#6

Hi,

Sorry for the delay in response, I’ve received your info and will get back to you with some clarification. (ZOOM-199643)

Regards,
Yosuke

#8

Hi,

Why a 404 is returned on the ping ?

Basically this is by design and we will have to keep this as it is for now.
As in a high level explanation, handling WebClient/WebSdk calls will first seek for all available gateways and and tries to choose the least loaded.
If the gateway in that location is full we will reply with 404.

Is it possible to increase resources allocated to Tokyo data center / gateway to accept more traffic for this region ?

We are currently in the faze of enhancement in the performance and capacity perspective and have added your request as well.
This may take a while and for now our recommendation will be, from your admin portal, deselect HK and instead choose Singapore. (you can keep other location as it is)

WebPortal: ADMIN -> Account Settings -> In Meeting (Advanced) -> “Select data center regions for meetings/webinars hosted by your account”: uncheck HK and check Singapore

#9

HI @yosuke.sawamura

Thank you for your update.
We will switch to Singapore and hope more capacity will be provided soon.

1 Like
#10

Thanks @yosuke.sawamura! :slight_smile:

@nvivot, We will keep you updated!

-Tommy

#11

@yosuke.sawamura @tommy

We updated our production account by removing HK and adding Singapore.
image

Unfortunately since that change, only Tokyo and US data centers remains

image
image1311Ă—182 61.1 KB

As a result, since Tokyo reply as not available most of time then most of our users are routed to US.

Where is Singapore ?

This is less and less acceptable so please fix it urgently

#12

@tommy no news about the fact that’s impossible to use Singapore ??

#13

Hi nvivot,

Thank you for your reply and so sorry for keep you waiting.
Seems that still our APAC resource was not enough to reroute using Singapore as well.
We are discussing this issue internally on how we can resolve.
We will try to get back to you asap.

Regards,
Yosuke

1 Like
#14

13 days without any news and Singapore data center is still not available… what’s the hell is going on seriously ?

Since yesterday, almost all our users being hosted on Hong Kong are reporting extremely bad audio and video performances (choppy, lags, freeze), and we can easily figure quite a huge latency from zoom meeting dashboard for those users.

What’s going on with HK data center and Zoom network ?

Please give us feedback as soon as possible.

#15

Another point is, now that you have Osaka (OS) and Tokyo (TY) data centers, we can figure that sometime users go on Tokyo and sometime on Osaka (very low percentage)

But this is only when TY Gateway reply with 200 instead of 404…

image
image863Ă—182 28.6 KB

US regions have multiple Gateways, so why don’t you increase Gateways for Japan region as well as it looks like the bottleneck here.

#16

Hey @nvivot,

We appreciate your patience as we work on resolving this issue. We are going to fix this. :slight_smile:

We will get back to you with an update as soon as possible.

Thanks,
Tommy

#17

Hi,

We have reviewed the situation with our engineering and also is planned to increase the capacity for Japan location.
Once the deployment is done we will let you know once more.

Regards,
Yosuke

#18

Hi @yosuke.sawamura,

I guess you’re talking about reducing the schedule to November to increase JP gateway capacity as explained in the email exchange.
If not, could you re-explain what are you planning ?

Thanks.

#19

We are trying to increase the capacity.

1 Like
#20

Looks like we are now mainly hosted on tokyo even during rush hours. Thank you for the capacity increase.

This thread can be closed, unless this is still happening for someone else.

1 Like
#22