AYON via vpn

Administration
,
1

Hello team,

We have remote users connecting to AYON via VPN, and they’ve been reporting intermittent connection issues—particularly that publishing occasionally fails without any clear reason.

We investigated further and added more detailed logging to the launcher. Below is a sample log captured from one of the affected users during a failed attempt:

Unable to connect the server. Connection ERROR ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
TRACEBACK BEGIN
Traceback (most recent call last):
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\connectionpool.py", line 466, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\connectionpool.py", line 461, in _make_request
    httplib_response = conn.getresponse()
  File "http\client.py", line 1377, in getresponse
  File "http\client.py", line 320, in begin
  File "http\client.py", line 289, in _read_status
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\util\retry.py", line 550, in increment
    raise six.reraise(type(ERROR), ERROR, _stacktrace)
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\packages\six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\connectionpool.py", line 466, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\urllib3\connectionpool.py", line 461, in _make_request
    httplib_response = conn.getresponse()
  File "http\client.py", line 1377, in getresponse
  File "http\client.py", line 320, in begin
  File "http\client.py", line 289, in _read_status
urllib3.exceptions.ProtocolERROR: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\ayon_api\server_api.py", line 1270, in _do_rest_request
    response = function(url, **kwargs)
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\requests\sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\user\Desktop\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\AYON 1.1.1 - PZ\dependencies\requests\adapters.py", line 501, in send
    raise ConnectionERROR(err, request=request)
requests.exceptions.ConnectionERROR: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

TRACEBACK END

Notably, there are no errors on the server side when this occurs, which makes it harder to pinpoint the cause.

Please advise on how we should proceed or if there are known issues related to VPN usage or remote disconnections.

Regards,
Tu

2

Was that user trying to connect to a self hosted AYON instance or an instance on Ynput Cloud?

3

Hello @mustafa_jafar. The user was connecting to a self-hosted instance of AYON.

4

I’m not sure if this is related to your setup or if the ayon backend or ayon python api need some additional implementations to work with the VPN.

Tagging @martin.wacker and @iLLiCiT I believe they can provide further info.

5

Thank you @mustafa_jafar.

@martin.wacker @iLLiCiT Just a bit more information.

The remote artist is located in Canada, while our studio is based in Australia.

In our initial testing, we did not encounter the issue when using my personal workstation here in Australia.

A reliable way to replicate the issue on the artist’s machine in Canada was to use the launcher and repeatedly click different objects in the context menu, effectively spamming the connection with AYON.

I attempted the same test on my personal workstation but was unable to reproduce the issue.

I also updated the _do_rest_request function to increase the sleep interval between retries, as 0.1 seconds seemed too short. However, this adjustment did not resolve the problem.

What step should I take next to further debug this issue?

6

Do you guys know if AYON server has a timeout period? If so where can I find it?

7

maybe

8

I believe you’ll need these for client side retries and timeouts:

With the default timeout being 10 in Python API?

And the retries defaulting to 3:

You can try playing with those env vars and see if any combination improves it from the “client” side.

Server side the timeout is passed through here: ayon-backend/start.sh at f255be0ce93adba12769dd2022360ac551dd9655 · ynput/ayon-backend · GitHub

1 Like