Deadline publish job fails - invalid AYON_API_KEY

dan · 11 March 2025 17:33

Hi all,

I’ve asked this on discord but thought I would move it here as this seems to be where all the main questions are being asked.

I can render via ayon’s publisher on deadline from houdini (solaris) - but I’m having to add the AYON_API_KEY and AYON_SERVER_URL directly to the job env vars in deadline for it to not error with:

!!! Please use 'AYON_SERVER_URL' and 'AYON_API_KEY' environment variables to specify valid server url and api key for headless mode.

I’ve followed the docs from what I can tell, here is the houdini publish dialog and the deadline plugins config for ayon:

I have made the service user on AYON and thats the API key being used in deadline.

Anything I could be missing here?

Petr_Kalis · 11 March 2025 21:12

I would guess that API key might be wrong.
You could double check if what you set in Deadline plugin is actually propagated correctly. Just check render job log, it should show up there

Pay extra attention if there are no asterisks on the line with AYON_API_KEY. There is some Deadline configuration that masks passwords and pushes through just “*” instead of real value.

dan · 12 March 2025 10:19

Yea the API key is correct, I’ve tried re-entering and even making a new service user for a fresh api key, restarting the deadline server ect - still the same issue.

Full log is here:

=======================================================
Error
=======================================================
Error: Global job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\escobar\plugins\67d15e858e50ff41a9586582\GlobalJobPreLoad.py': Failed to run AYON process to extract environments. (FranticX.Scripting.PythonNetException)
  File "C:\ProgramData\Thinkbox\Deadline10\workers\escobar\plugins\67d15e858e50ff41a9586582\GlobalJobPreLoad.py", line 611, in inject_ayon_environment
    raise RuntimeError(
  File "C:\ProgramData\Thinkbox\Deadline10\workers\escobar\plugins\67d15e858e50ff41a9586582\GlobalJobPreLoad.py", line 728, in __main__
    inject_ayon_environment(deadlinePlugin)
   at Python.Runtime.PythonException.ThrowLastAsClrException() in C:\thinkbox-conda\conda-bld\dotnet_pythonnet_1726863100822\work\src\runtime\PythonException.cs:line 53
   at Python.Runtime.PyObject.Invoke(PyObject[] args) in C:\thinkbox-conda\conda-bld\dotnet_pythonnet_1726863100822\work\src\runtime\PythonTypes\PyObject.cs:line 750
   at Python.Runtime.PyObject.InvokeMethod(String name, PyObject[] args) in C:\thinkbox-conda\conda-bld\dotnet_pythonnet_1726863100822\work\src\runtime\PythonTypes\PyObject.cs:line 818
   at FranticX.Scripting.PythonNetScriptEngine.CallFunction(String moduleName, String functionName, Object[] args)
   at Deadline.Plugins.PluginWrapper.d()
   at Deadline.Plugins.PluginWrapper.StartJob(String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgt, CancellationToken bgu)
   at Deadline.Plugins.SandboxedPlugin.StartJob(Job job, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajx, CancellationToken ajy)

=======================================================
Log
=======================================================
2025-03-12 10:14:53:  0: Loading Job's Plugin timeout is Disabled
2025-03-12 10:14:53:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'danwn'
2025-03-12 10:14:57:  0: Executing plugin command of type 'Initialize Plugin'
2025-03-12 10:14:57:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\escobar\plugins\67d15e858e50ff41a9586582\Houdini.py'
2025-03-12 10:14:57:  0: INFO: Plugin execution sandbox using Python version 3
2025-03-12 10:14:57:  0: INFO: About: Houdini Plugin for Deadline
2025-03-12 10:14:57:  0: INFO: The job's environment will be merged with the current environment before rendering
2025-03-12 10:14:57:  0: Done executing plugin command of type 'Initialize Plugin'
2025-03-12 10:14:57:  0: Start Job timeout is disabled.
2025-03-12 10:14:57:  0: Task timeout is disabled.
2025-03-12 10:14:57:  0: Loaded job: balt_pipe_deadline_lgt_v008.hiplc - usdrenderMain [EXPORT IFD] (67d15e858e50ff41a9586582)
2025-03-12 10:14:57:  0: Executing plugin command of type 'Start Job'
2025-03-12 10:14:57:  0: DEBUG: S3BackedCache Client is not installed.
2025-03-12 10:14:57:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\escobar\plugins\67d15e858e50ff41a9586582\GlobalAssetTransferPreLoad.py'
2025-03-12 10:14:57:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2025-03-12 10:14:57:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2025-03-12 10:14:57:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2025-03-12 10:14:57:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2025-03-12 10:14:57:  0: INFO: Executing global job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\escobar\plugins\67d15e858e50ff41a9586582\GlobalJobPreLoad.py'
2025-03-12 10:14:57:  0: PYTHON: *** GlobalJobPreload 1.2.0 start ...
2025-03-12 10:14:57:  0: PYTHON: >>> Getting job ...
2025-03-12 10:14:57:  0: PYTHON: >>> Injecting AYON environments ...
2025-03-12 10:14:57:  0: PYTHON: --- Ayon executable: C:\Program Files\Ynput\AYON 1.1.1\ayon_console.exe
2025-03-12 10:14:58:  0: PYTHON: >>> Temporary path: C:\Users\danwn\AppData\Local\Temp\20250312101457938382_d9ca160d-ff2a-11ef-abe4-d97eb93a10ef.json
2025-03-12 10:14:58:  0: INFO: Setting Process Environment Variable AYON_SERVER_URL to http://d2-server:5000/
2025-03-12 10:14:58:  0: INFO: Setting Process Environment Variable AYON_API_KEY to *********
2025-03-12 10:14:58:  0: INFO: Setting Process Environment Variable AYON_BUNDLE_NAME to 2025.02.0-full-2025-03-11-01
2025-03-12 10:14:58:  0: PYTHON: >>> Executing: C:\Program Files\Ynput\AYON 1.1.1\ayon_console.exe --headless addon applications extractenvironments C:\Users\danwn\AppData\Local\Temp\20250312101457938382_d9ca160d-ff2a-11ef-abe4-d97eb93a10ef.json --envgroup farm --project balt --folder /shots/rand/pipe/deadline --task lighting --app houdini/20-5
2025-03-12 10:14:58:  0: STDOUT: !!! Got invalid credentials.
2025-03-12 10:14:58:  0: STDOUT: Invalid API key for 'http://d2-server:5000/'.
2025-03-12 10:14:58:  0: STDOUT: !!! Please use 'AYON_SERVER_URL' and 'AYON_API_KEY' environment variables to specify valid server url and api key for headless mode.
2025-03-12 10:14:58:  0: PYTHON: Failed to run AYON process to extract environments. Trying to use legacy arguments.
2025-03-12 10:14:58:  0: STDOUT: !!! Got invalid credentials.
2025-03-12 10:14:58:  0: STDOUT: Invalid API key for 'http://d2-server:5000/'.
2025-03-12 10:14:58:  0: STDOUT: !!! Please use 'AYON_SERVER_URL' and 'AYON_API_KEY' environment variables to specify valid server url and api key for headless mode.
2025-03-12 10:14:58:  0: PYTHON: Traceback (most recent call last):
2025-03-12 10:14:58:  0: PYTHON:   File "C:\ProgramData\Thinkbox\Deadline10\workers\escobar\plugins\67d15e858e50ff41a9586582\GlobalJobPreLoad.py", line 611, in inject_ayon_environment
2025-03-12 10:14:58:  0: PYTHON:     raise RuntimeError(
2025-03-12 10:14:58:  0: PYTHON: RuntimeError: Failed to run AYON process to extract environments.
2025-03-12 10:14:58:  0: PYTHON: !!! Injection failed.
2025-03-12 10:14:58:  0: Done executing plugin command of type 'Start Job'

The only other thing I’m not sure of is TLS, currently I’m disabling any security (certs, tls) because I was having issues, so it was easier to disable for testing. Could this be causing any issues here? This dialog pops up after I configure the plugins in deadline.

Thanks mate

mustafa_jafar · 12 March 2025 10:33

I’ve compared your log with one of my DL logs.

In my DL log AYON_API_KEY shows a value not asterisks. I’m not sure if you have a configuration on your side that replaces the API key with asterisks.

from your log.

2025-03-12 10:14:58:  0: INFO: Setting Process Environment Variable AYON_API_KEY to *********

Checked some internal discussions, its because of credentials manager access rights.

dan · 12 March 2025 10:45

I’m not doing anything custom, basic ayon install and basic deadline install. Do you mean the windows credentials manager? Or something deadline specific?

Thanks for the pointers so far mate

mustafa_jafar · 12 March 2025 12:29

Deadline Secrets Management. where you need to grant access to user/machine to credentials manager.

https://docs.thinkboxsoftware.com/products/deadline/10.3/1_User%20Manual/manual/secrets-management/deadline-secrets-management.html

dan · 14 March 2025 15:13

So! Sorry for the late reply, I hadnt setup secrets during install so I just had to completely reinstall everything. Anyway, secrets and TLS is all setup and working. I can now set the ayon API key without deadline complaining but I’m still getting the same error, which is very annoying!

Is there a simple way of me just passing this API env environment key through to deadline for all jobs? Adding them here:

Petr_Kalis · 14 March 2025 15:28

So you are not seeing asterisks anymore in log? And do you see expected value for AYON_API_KEY?

You could also use Deadline Admin Docs | AYON Docs option.

Follow Deadline Admin Docs | AYON Docs to see how Additional AYON servers should look like.

dan · 14 March 2025 15:46

Ah yes, sorry forgot to check that, its still: Process Environment Variable AYON_API_KEY to *********

I might just bypass this and make an event script to add the env var to all jobs. I can’t understand why this isnt working.

Petr_Kalis · 14 March 2025 16:10

That links I sent could work as they are bypassing Deadline password field.

It is some security feature of Deadline, never had it myself to provide step-by-step how to fix that.

dan · 14 March 2025 16:23

I’m not sure what you mean in the links here, do you mean to add the API key into the “additional ayon servers”? Sorry, bit confused

mustafa_jafar · 14 March 2025 17:47

Petr is referring to adding your AYON server in the Additional AYON Servers.

This configuration is done in two steps:

Update AYON configuration on DL.

image700×468 9.52 KB
Enable this settings in AYON Deadline addon

image945×119 7.03 KB

dan · 15 March 2025 12:34

Thanks guys, I did try this but I just dont see what I’m meant to do here, as the server is being passed through fine its just the API key thats not being read properly. I’ve tried adding

server:port@AYON_API_KEY
server:port@1234567
server:port@AYON_API_KEY:1234567

Or am I missing something here?

Petr_Kalis · 18 March 2025 09:57

The issue is your Deadline configuration where there is some functionality which is not propagated password fields correctly/as we expect. (Eg. Ayon API key value is physically propagated as ‘*****’ instead of real value).

Additional YON Servers should skip that. This feature allows to use AYON_API_KEY based on AYON server. (eg. I sent in url to Deadline server used for rendering, that is being set in ayon+settings://deadline/deadline_urls, it looks for matching value (before @ character) and uses string after that as AYON_API_KEY.

You could imagine it as another way of filling values for Ayon Server Url and Ayon API key, but without DL interfering with value of AYON_API_KEY.

Here is maybe better example:

Ayon API key is irrelevant, it might be a DL UI thing.

dan · 19 March 2025 11:21

Okay I’ve tried exactly as you’ve mentioned there, complete restart of the server hosting deadline, fresh project, all for good measure. Still the exact same error where its outputting as *****.

I’m using pretty much the latest deadline, could this be an issue with the secrets management? I’d assume it wouldnt show the API key if its meant to be kept as a secret?

If I do: print(os.environ.get("AYON_API_KEY")) in the ayon console it shows me a different API key but I expect thats fine and its just for the current session?

Any other ideas? Cheers for all the help so far guys

Petr_Kalis · 20 March 2025 11:40

Have you enabled this option

(you could check it as AYON_SERVER_URL should appear in render job environments)
and filled Additional AYON Servers?

(Or as a last ditch attempt you could modify custom/GlobalJobPreLoad.py in your DL repo(around line 487 and hardcode your AYON_API_KEY value there.)

(Or figure out how to disable that Deadline feature, but I looked into DL documentation and couldnt figure out how it is actually called and how to disable it.)