AYON / Openpype Deadline Setup

Here’s my personal experience exploring how to make this setup.
Installation take place on both sides Ayon/OpenPype and deadline.

Prerequisites

  1. you should have a running deadline server.

    🛠️ Install Deadline Server and Client

    The best way to start learning about deadline is from Deadline official tutorials.
    The following steps are the minimal installation required for this guide:

  2. you should have AYON API Key

    🔑 Get AYON_API_KEY

    It’s brought from Ayon server.
    Go to the Users tab and create a new service user, generate key and copy that key.

    make sure to save as you will see it once!

    Create Service User Generate Key Copy Key

Deadline Server Side:

  • Configure Deadline for AYON/Openpype

    1. copy openpype/modules/deadline/repository/custom to deadline_repository/custom

      please please, when updating theses custom plugins later on, don’t forget to to copy the whole plugin not just GlobalJobPreLoad.py

    2. Configure deadline plugins: using deadline monitor go to Tools > Configure PLugins

      Ayon:

      • Ayon executable : path of ayon_console.exe, windows default installation path (C:\Program Files\Ynput\AYON 1.0.0-beta.6\ayon_console.exe)
      • Ayon Server URL : it’s something like http://Server_IP:5000/
      • Ayon API Key : Service API Key

      If you are running Ayon Launcher live from source, you’d like to use the ayon_console.bat it’s in tools folder inside Ayon Launcher Repo.
      Note, this assumes you are running the ayon launcher in production mode. ayon dev mode is going to have its separate guide.

      🛠️ Configure OpenPype plugin

      In OpenPype:

      • OpenPype Installation Directories : base folder where all openpype version are installed, (you can keep it as it is if you didn’t change installation path while openpype installation)
      • OpenPype executable : path of openpype_console.exe, (you don’t have to use it unless you are running OP from source)

      If you are running OP from source, you’d like to use the openpype_console.bat it’s in tools folder inside OpenPype Repo

      🛠️ Configure OpenPype TileAssembler plugin
      • For OpenPype TileAssembler, I used the one came with openpype which was in my case

        C:\OpenPype\3.16.0\vendor\bin\oiio\windows\oiiotool.exe
        
    3. Run Webservice from CMD

      "C:\Program Files\Thinkbox\Deadline10\bin\deadlinewebservice.exe"
      

      It’s possible to change the listening port with the flag -port

      "C:\Program Files\Thinkbox\Deadline10\bin\deadlinewebservice.exe" -port 8082
      

      Instead of running from CMD, you can modify deadline.ini as krishna suggests below

    4. testing the webservice in browser, in my case it’s https not http

      https://deadline_machine_ip:port_number
      

    5. Disable Require Authentication if needed.
      Recent deadline addon version supports authentication.

AYON / Openpype Side:

The only setting you need to configure is Deadline webservice URL, and yeah without a slash at the end!

AYON OpenPype

you can also set Deadline webservice URL on project level

AYON OpenPype

you can also modify default deadline arguments values on project level

AYON OpenPype

Testing

Testing Houdini mantra rendering

And here it’s on deadline

1 Like

We installed the deadline as a service client during installation and added a couple of lines in deadline.ini (C:\Users\XXXX\AppData\Local\Thinkbox\Deadline10\deadline.ini), so we don’t have to worry about opening the .exe web service

Added these lines in deadline.ini

LaunchWebServiceAtStartup=True
KeepWebServiceRunning=True
2 Likes

Brilliant!
It works like magic!
Thanks man :pray:

This is a cool option too for running deadline webservices.
By @tokestuartjepsen on discord

"Another approach to keep the webservice running, is to have it as a job on the farm. You can then setup a redundancy machine to pick up the webservice job if the main machine fails. "

I faced a problem "Failed to run OpenPype process to extract environments." when moving from 3.15.x to 3.16.x

If you push a new installation of OP, you’ll need to update OpenPype Executable in the Deadline plugin path.
I didn’t need to update OpenPype Installation Directories

It’s possible to leave OpenPype Executable empty because the GlobalJobPreLoad should retrieve the OpenPype executable path from OpenPype Installation Directories, read more.
However, it didn’t work for me so that I had to set the executable explicitly!


I also updated the oiio path

Was this needed?
Pretty sure that plugin is deprecated and we encourage to use Deadline Draft instead.

I just did it because it was pointing to a path that didn’t exist

I don’t know much about openpype and deadline workflow.

Hi,

I’m encountering an error in Deadline for Nuke render. It’s indicating a “bad value for display & view process.” during the review render baking. Is there any known fix for this problem?

@mustafa_jafar the guide is missing the part when you need to create a new service user with API key to be entered into the Ayon plugin in the Deadline.

I’ve updated the guide :smiley:

I scratched my head a bit with recent DL version where connection to webservice didn’t work. (It failed in my tests in browser and in it was failing with 401 in ValidateDeadlineConnection).

The issue was that newer versions of DL set Require Authentication in Web Service Settings (in Tools > Configure Repository Options...), I had to disable it.

Maybe this would help someone else.

image

2 Likes

Has anyone tried Deadline with ayon-core 3.0.0 or greater? I get a KeyError when extracting environments, this didn’t happen in ayon-core 0.2.1

KeyError
2024-05-22 15:24:41:  0: Loading Job's Plugin timeout is Disabled
2024-05-22 15:24:41:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'administrator'
2024-05-22 15:24:43:  0: Executing plugin command of type 'Initialize Plugin'
2024-05-22 15:24:43:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\license02\plugins\664d815f742ba62c7d342863\Nuke.py'
2024-05-22 15:24:43:  0: INFO: Plugin execution sandbox using Python version 3
2024-05-22 15:24:43:  0: INFO: About: Nuke Plugin for Deadline
2024-05-22 15:24:43:  0: INFO: The job's environment will be merged with the current environment before rendering
2024-05-22 15:24:43:  0: Done executing plugin command of type 'Initialize Plugin'
2024-05-22 15:24:43:  0: Start Job timeout is disabled.
2024-05-22 15:24:43:  0: Task timeout is disabled.
2024-05-22 15:24:43:  0: Loaded job: renderCompMain.####.exr (664d815f742ba62c7d342863)
2024-05-22 15:24:43:  0: Executing plugin command of type 'Start Job'
2024-05-22 15:24:43:  0: DEBUG: S3BackedCache Client is not installed.
2024-05-22 15:24:43:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\license02\plugins\664d815f742ba62c7d342863\GlobalAssetTransferPreLoad.py'
2024-05-22 15:24:43:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2024-05-22 15:24:43:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2024-05-22 15:24:43:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2024-05-22 15:24:43:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2024-05-22 15:24:43:  0: INFO: Executing global job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\license02\plugins\664d815f742ba62c7d342863\GlobalJobPreLoad.py'
2024-05-22 15:24:43:  0: PYTHON: *** GlobalJobPreload 1.1.0 start ...
2024-05-22 15:24:43:  0: PYTHON: >>> Getting job ...
2024-05-22 15:24:43:  0: PYTHON: >>> Injecting AYON environments ...
2024-05-22 15:24:43:  0: PYTHON: --- Ayon executable: Y:\Pipeline\Apps\Ynput\AYON 1.0.0\ayon.exe
2024-05-22 15:24:43:  0: PYTHON: >>> Temporary path: C:\Users\ADMINI~1\AppData\Local\Temp\20240522052443789190_98b5ad76-17fb-11ef-bca8-04d3b038551e.json
2024-05-22 15:24:43:  0: INFO: Setting Process Environment Variable AYON_SERVER_URL to http://10.0.4.7:4000
2024-05-22 15:24:43:  0: INFO: Setting Process Environment Variable AYON_API_KEY to 2f13e32ce1f64055b5d49928ffccc412
2024-05-22 15:24:43:  0: INFO: Setting Process Environment Variable AYON_BUNDLE_NAME to 2024.01.0-full_shotgrid-2024-05-22-03
2024-05-22 15:24:43:  0: PYTHON: >>> Executing: Y:\Pipeline\Apps\Ynput\AYON 1.0.0\ayon.exe --headless addon applications extractenvironments C:\Users\ADMINI~1\AppData\Local\Temp\20240522052443789190_98b5ad76-17fb-11ef-bca8-04d3b038551e.json --envgroup farm --project AYON_dev --folder /101/sq001/sh010 --task Comp --app nuke/15-0
2024-05-22 15:24:51:  0: STDOUT: *** WRN: >>> { ModulesLoader }: [  Addon core 0.3.1 has no content to import  ] 
2024-05-22 15:24:52:  0: STDOUT: Usage: ayon addon [OPTIONS] COMMAND [ARGS]...
2024-05-22 15:24:52:  0: STDOUT: Try 'ayon addon --help' for help.
2024-05-22 15:24:52:  0: STDOUT: Error: No such command 'applications'.
2024-05-22 15:24:52:  0: STDOUT: >>> loading environments ...
2024-05-22 15:24:52:  0: STDOUT:   - global AYON ...
2024-05-22 15:24:52:  0: STDOUT:   - for addons ...
2024-05-22 15:24:52:  0: STDOUT: *** AYON [1.0.0] ---
2024-05-22 15:24:52:  0: STDOUT: >>> Using AYON from [ Y:\Pipeline\Apps\Ynput\AYON 1.0.0 ]
2024-05-22 15:24:52:  0: STDOUT: ... AYON variant: [ production ]
2024-05-22 15:24:52:  0: STDOUT: ... AYON bundle:  [ 2024.01.0-full_shotgrid-2024-05-22-03 ]
2024-05-22 15:24:52:  0: PYTHON: Failed to run AYON process to extract environments. Trying to use legacy arguments.
2024-05-22 15:24:59:  0: STDOUT: *** WRN: >>> { ModulesLoader }: [  Addon core 0.3.1 has no content to import  ] 
2024-05-22 15:25:00:  0: STDOUT: Traceback (most recent call last):
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\lib\applications.py", line 1777, in prepare_context_environments
2024-05-22 15:25:00:  0: STDOUT:     workdir = get_workdir_with_workdir_data(
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\pipeline\workfile\path_resolving.py", line 135, in get_workdir_with_workdir_data
2024-05-22 15:25:00:  0: STDOUT:     template_obj = anatomy.templates_obj[template_key]["folder"]
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\lib\path_templates.py", line 267, in __getitem__
2024-05-22 15:25:00:  0: STDOUT:     return self.objected_templates[key]
2024-05-22 15:25:00:  0: STDOUT: KeyError: 'default'
2024-05-22 15:25:00:  0: STDOUT: During handling of the above exception, another exception occurred:
2024-05-22 15:25:00:  0: STDOUT: Traceback (most recent call last):
2024-05-22 15:25:00:  0: STDOUT:   File "start.py", line 740, in main_cli
2024-05-22 15:25:00:  0: STDOUT:   File "Y:\Pipeline\Apps\Ynput\AYON 1.0.0\dependencies\click\core.py", line 1130, in __call__
2024-05-22 15:25:00:  0: STDOUT:     return self.main(*args, **kwargs)
2024-05-22 15:25:00:  0: STDOUT:   File "Y:\Pipeline\Apps\Ynput\AYON 1.0.0\dependencies\click\core.py", line 1055, in main
2024-05-22 15:25:00:  0: STDOUT:     rv = self.invoke(ctx)
2024-05-22 15:25:00:  0: STDOUT:   File "Y:\Pipeline\Apps\Ynput\AYON 1.0.0\dependencies\click\core.py", line 1657, in invoke
2024-05-22 15:25:00:  0: STDOUT:     return _process_result(sub_ctx.command.invoke(sub_ctx))
2024-05-22 15:25:00:  0: STDOUT:   File "Y:\Pipeline\Apps\Ynput\AYON 1.0.0\dependencies\click\core.py", line 1404, in invoke
2024-05-22 15:25:00:  0: STDOUT:     return ctx.invoke(self.callback, **ctx.params)
2024-05-22 15:25:00:  0: STDOUT:   File "Y:\Pipeline\Apps\Ynput\AYON 1.0.0\dependencies\click\core.py", line 760, in invoke
2024-05-22 15:25:00:  0: STDOUT:     return __callback(*args, **kwargs)
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\cli.py", line 178, in extractenvironments
2024-05-22 15:25:00:  0: STDOUT:     PypeCommands.extractenvironments(
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\pype_commands.py", line 182, in extractenvironments
2024-05-22 15:25:00:  0: STDOUT:     env = get_app_environments_for_context(
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\lib\applications.py", line 1482, in get_app_environments_for_context
2024-05-22 15:25:00:  0: STDOUT:     context.run_prelaunch_hooks()
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\lib\applications.py", line 1312, in run_prelaunch_hooks
2024-05-22 15:25:00:  0: STDOUT:     prelaunch_hook.execute()
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\hooks\pre_global_host_data.py", line 44, in execute
2024-05-22 15:25:00:  0: STDOUT:     prepare_context_environments(temp_data)
2024-05-22 15:25:00:  0: STDOUT:   File "C:\Users\administrator\AppData\Local\Ynput\AYON\addons\openpype_3.18.6\openpype\lib\applications.py", line 1785, in prepare_context_environments
2024-05-22 15:25:00:  0: STDOUT:     raise ApplicationLaunchFailed(
2024-05-22 15:25:00:  0: STDOUT: openpype.lib.applications.ApplicationLaunchFailed: Error in anatomy.format: 'default'
2024-05-22 15:25:00:  0: STDOUT: >>> loading environments ...
2024-05-22 15:25:00:  0: STDOUT:   - global AYON ...
2024-05-22 15:25:00:  0: STDOUT:   - for addons ...
2024-05-22 15:25:00:  0: STDOUT: *** AYON [1.0.0] ---
2024-05-22 15:25:00:  0: STDOUT: >>> Using AYON from [ Y:\Pipeline\Apps\Ynput\AYON 1.0.0 ]
2024-05-22 15:25:00:  0: STDOUT: ... AYON variant: [ production ]
2024-05-22 15:25:00:  0: STDOUT: ... AYON bundle:  [ 2024.01.0-full_shotgrid-2024-05-22-03 ]
2024-05-22 15:25:00:  0: STDOUT: !!! AYON crashed:
2024-05-22 15:25:00:  0: PYTHON: Traceback (most recent call last):
2024-05-22 15:25:00:  0: PYTHON:   File "C:\ProgramData\Thinkbox\Deadline10\workers\license02\plugins\664d815f742ba62c7d342863\GlobalJobPreLoad.py", line 542, in inject_ayon_environment
2024-05-22 15:25:00:  0: PYTHON:     "Failed to run AYON process to extract environments."
2024-05-22 15:25:00:  0: PYTHON: RuntimeError: Failed to run AYON process to extract environments.
2024-05-22 15:25:00:  0: PYTHON: !!! Injection failed.
2024-05-22 15:25:00:  0: 'C:\Users\administrator\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2023-12-16T003239.0000000Z' already exists. Skipping extraction of PythonSync.
2024-05-22 15:25:02:  0: Done executing plugin command of type 'Start Job'

I’m pretty sure when using ayon core 0.3.0+ this shouldn’t touch the openpype addon. Make sure to set openpype in the bundle to None.

Instead of openpype_3.18.6 it should touch ayon_core_0.3.0 or alike.

Also, are you sure you updated Deadline’s GlobalJobPreLoad.py file? It does seem like a recent version, but just checking whether you might have missed also updating that.

Also, you’d need to update the Deadline addon.
You might not hit that issue yet. but, I thought about shedding some light on it.

There’s an issue that will probably vanish when splitting the Addons into separate repos.
The issue: client code of deadline is inside ayon-core but its associated settings is in deadline addon.
and when updating ayon-core the new client code of deadline may expect new settings e.g. I got some missing key error not_verify_ssl and that key only exists in the recent deadline addon.

Hey Guys! Need some information about your deadline setups.

When we installing deadline do you copy the ceritificate (Deadline10Client.pfx) locally in each machine in C:/ drive or do you use network UNC paths and also do you connect with map drive or UNC path in repository settings.

So far we are using the UNC path and I frequently face issues with deadline webservice especially, getting authetication issues “HTTP ERROR 401”. I believe I’m loosing something.

I’m pretty sure the webservice itself should not touch UNC paths at all because it connects to just a server’s port, not a path. So UNC versus mapped drive should be irrelevant there. This sounds like it may be in the realm of Deadline support itself. :slight_smile:

When do you get the error? And do you have a stack trace describing where AYON may be causing it?

Hey @BigRoy Thanks for responding!

I think this is deadline issue, I’m testing without the AYON only, I can able to connect to deadline web service using localhost but not the IP.
My port was open in the machine and IP I’m trying to access is also correct.
I believe this is more of “HTTP ERROR 401” an authentication problem, not sure how to tackle it. I get this once or twice everymonth, I basically overwrite the installation everytime, this is frustrating!

These seem like firewall issues - not sure if @Petr_Kalis or @mustafa_jafar have seen this before with Deadline but if not I think you may be better off hearing from Deadline support or their community. :slight_smile:

I don’t know much about Deadline installation.
Basically, I only followed these three videos.

For reference, I’ve my deadline server installed on a windows virtual machine
and used main PC as deadline worker. I don’t think I touched the firewall.
Also, I didn’t need to touch the certificates.