Hi there,
After some updates in our systems, network, deadline, and ayon, we’re having some strange issues with rendering in Deadline through Ayon.
Why it is strange, is because there are times that it works flawlessly, and then the same render could be sent again, and it is full of errors.
This leads me to believe that it may be related to issues with networks and/or firewalls.
I’m getting a bunch of “injection” errors.
Our systems use something called “Faronics Deep Freeze”, I’m wondering if the GlobalJobPreload.py is an outdated one, that isn’t getting updated before the render.
Anyone know or have experience with this? I’m just trying to find which haystack to look for the needle in.
=======================================================
Error
=======================================================
Error: Global job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\SDD1111-00-1111\plugins\67b5636f9b340b017bc5f57c\GlobalJobPreLoad.py': Failed to run AYON process to extract environments. (FranticX.Scripting.PythonNetException)
File "C:\ProgramData\Thinkbox\Deadline10\workers\SDD1111-00-1111\plugins\67b5636f9b340b017bc5f57c\GlobalJobPreLoad.py", line 611, in inject_ayon_environment
raise RuntimeError(
File "C:\ProgramData\Thinkbox\Deadline10\workers\SDD1111-00-1111\plugins\67b5636f9b340b017bc5f57c\GlobalJobPreLoad.py", line 728, in __main__
inject_ayon_environment(deadlinePlugin)
at Python.Runtime.PythonException.ThrowLastAsClrException() in C:\thinkbox-conda\conda-bld\dotnet_pythonnet_1726863100822\work\src\runtime\PythonException.cs:line 53
at Python.Runtime.PyObject.Invoke(PyObject[] args) in C:\thinkbox-conda\conda-bld\dotnet_pythonnet_1726863100822\work\src\runtime\PythonTypes\PyObject.cs:line 750
at Python.Runtime.PyObject.InvokeMethod(String name, PyObject[] args) in C:\thinkbox-conda\conda-bld\dotnet_pythonnet_1726863100822\work\src\runtime\PythonTypes\PyObject.cs:line 818
at FranticX.Scripting.PythonNetScriptEngine.CallFunction(String moduleName, String functionName, Object[] args)
at Deadline.Plugins.PluginWrapper.d()
at Deadline.Plugins.PluginWrapper.StartJob(String& outMessage, AbortLevel& abortLevel)
=======================================================
Type
=======================================================
RenderPluginException
=======================================================
Stack Trace
=======================================================
at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgt, CancellationToken bgu)
at Deadline.Plugins.SandboxedPlugin.StartJob(Job job, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajx, CancellationToken ajy)
=======================================================
Log
=======================================================
2025-02-19 00:08:56: 0: Loading Job's Plugin timeout is Disabled
2025-02-19 00:08:56: 0: SandboxedPlugin: Render Job As User disabled, running as current user 'pjUserx'
2025-02-19 00:09:02: 0: Executing plugin command of type 'Initialize Plugin'
2025-02-19 00:09:02: 0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\SDD1111-00-1111\plugins\67b5636f9b340b017bc5f57c\MayaBatch.py'
2025-02-19 00:09:03: 0: INFO: Plugin execution sandbox using Python version 3
2025-02-19 00:09:03: 0: INFO: About: Maya Batch Plugin for Deadline
2025-02-19 00:09:03: 0: INFO: The job's environment will be merged with the current environment before rendering
2025-02-19 00:09:03: 0: Done executing plugin command of type 'Initialize Plugin'
2025-02-19 00:09:03: 0: Start Job timeout is disabled.
2025-02-19 00:09:03: 0: Task timeout is disabled.
2025-02-19 00:09:03: 0: Loaded job: wtestProjecT_Steve_A2_env_look_v009.ma - renderLookMain (67b5636f9b340b017bc5f57c)
2025-02-19 00:09:03: 0: Skipping O: because it is already mapped
2025-02-19 00:09:03: 0: Successfully mapped P: to \\storageServer\Programs\nid\projects
2025-02-19 00:09:03: 0: Executing plugin command of type 'Start Job'
2025-02-19 00:09:03: 0: DEBUG: S3BackedCache Client is not installed.
2025-02-19 00:09:03: 0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\SDD1111-00-1111\plugins\67b5636f9b340b017bc5f57c\GlobalAssetTransferPreLoad.py'
2025-02-19 00:09:03: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2025-02-19 00:09:03: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2025-02-19 00:09:03: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2025-02-19 00:09:03: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2025-02-19 00:09:03: 0: INFO: Executing global job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\SDD1111-00-1111\plugins\67b5636f9b340b017bc5f57c\GlobalJobPreLoad.py'
2025-02-19 00:09:03: 0: PYTHON: *** GlobalJobPreload 1.2.0 start ...
2025-02-19 00:09:03: 0: PYTHON: >>> Getting job ...
2025-02-19 00:09:03: 0: PYTHON: >>> Injecting AYON environments ...
2025-02-19 00:09:03: 0: PYTHON: --- Ayon executable: C:\Program Files\Ynput\AYON 1.1.1\ayon_console.exe
2025-02-19 00:09:03: 0: PYTHON: AYON Server URL submitted with job 'https://our.ayon.server' is not the Deadline AYON Plug-in default server URL 'https://our.ayon.server' but has no API key defined in Additional Server URLs. Falling back to default API key configured in Deadline repository for the AYON plug-in.
2025-02-19 00:09:03: 0: PYTHON: >>> Temporary path: C:\Users\pjUserx\AppData\Local\Temp\20250219050903400115_a2f50f03-ee7f-11ef-8526-7d8b717350a7.json
2025-02-19 00:09:03: 0: INFO: Setting Process Environment Variable AYON_SERVER_URL to https://our.ayon.server
2025-02-19 00:09:03: 0: INFO: Setting Process Environment Variable AYON_API_KEY to 11XXXXXXXXXXXXXXXXXXXXXXXXXX
2025-02-19 00:09:03: 0: INFO: Setting Process Environment Variable AYON_BUNDLE_NAME to ugh01-06
2025-02-19 00:09:03: 0: PYTHON: >>> Executing: C:\Program Files\Ynput\AYON 1.1.1\ayon_console.exe --headless addon applications extractenvironments C:\Users\pjUserx\AppData\Local\Temp\20250219050903400115_a2f50f03-ee7f-11ef-8526-7d8b717350a7.json --envgroup farm --project W25_S4_ANID --folder /environment/S4LGT_Steve_A2_env --task look --app maya/2025
2025-02-19 00:11:32: 0: STDOUT: Traceback (most recent call last):
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\core_1.0.13\ayon_core\cli.py", line 314, in main
2025-02-19 00:11:32: 0: STDOUT: main_cli(
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403071252_windows.zip\dependencies\click\core.py", line 1157, in __call__
2025-02-19 00:11:32: 0: STDOUT: return self.main(*args, **kwargs)
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403071252_windows.zip\dependencies\click\core.py", line 1078, in main
2025-02-19 00:11:32: 0: STDOUT: rv = self.invoke(ctx)
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403071252_windows.zip\dependencies\click\core.py", line 1688, in invoke
2025-02-19 00:11:32: 0: STDOUT: return _process_result(sub_ctx.command.invoke(sub_ctx))
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403071252_windows.zip\dependencies\click\core.py", line 1688, in invoke
2025-02-19 00:11:32: 0: STDOUT: return _process_result(sub_ctx.command.invoke(sub_ctx))
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403071252_windows.zip\dependencies\click\core.py", line 1688, in invoke
2025-02-19 00:11:32: 0: STDOUT: return _process_result(sub_ctx.command.invoke(sub_ctx))
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403071252_windows.zip\dependencies\click\core.py", line 1434, in invoke
2025-02-19 00:11:32: 0: STDOUT: return ctx.invoke(self.callback, **ctx.params)
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403071252_windows.zip\dependencies\click\core.py", line 783, in invoke
2025-02-19 00:11:32: 0: STDOUT: return __callback(*args, **kwargs)
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\applications_1.0.2\ayon_applications\addon.py", line 362, in _cli_extract_environments
2025-02-19 00:11:32: 0: STDOUT: env = self.get_farm_publish_environment_variables(
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\applications_1.0.2\ayon_applications\addon.py", line 107, in get_farm_publish_environment_variables
2025-02-19 00:11:32: 0: STDOUT: return self.get_app_environments_for_context(
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\applications_1.0.2\ayon_applications\addon.py", line 70, in get_app_environments_for_context
2025-02-19 00:11:32: 0: STDOUT: return get_app_environments_for_context(
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\applications_1.0.2\ayon_applications\utils.py", line 192, in get_app_environments_for_context
2025-02-19 00:11:32: 0: STDOUT: context.run_prelaunch_hooks()
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\applications_1.0.2\ayon_applications\manager.py", line 581, in run_prelaunch_hooks
2025-02-19 00:11:32: 0: STDOUT: prelaunch_hook.execute()
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\core_1.0.13\ayon_core\hooks\pre_global_host_data.py", line 24, in execute
2025-02-19 00:11:32: 0: STDOUT: temp_data = EnvironmentPrepData({
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\applications_1.0.2\ayon_applications\utils.py", line 144, in __init__
2025-02-19 00:11:32: 0: STDOUT: data["project_settings"] = get_project_settings(project_name)
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\core_1.0.13\ayon_core\settings\lib.py", line 146, in get_project_settings
2025-02-19 00:11:32: 0: STDOUT: return _AyonSettingsCache.get_value_by_project(project_name)
2025-02-19 00:11:32: 0: STDOUT: File "C:\Users\pjUserx\AppData\Local\Ynput\AYON\addons\core_1.0.13\ayon_core\settings\lib.py", line 83, in get_value_by_project
2025-02-19 00:11:32: 0: STDOUT: value = ayon_api.get_addons_settings(
2025-02-19 00:11:32: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\_api.py", line 2108, in get_addons_settings
2025-02-19 00:11:32: 0: STDOUT: return con.get_addons_settings(*args, **kwargs)
2025-02-19 00:11:32: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\server_api.py", line 3785, in get_addons_settings
2025-02-19 00:11:32: 0: STDOUT: return self.get_addons_project_settings(
2025-02-19 00:11:32: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\server_api.py", line 3727, in get_addons_project_settings
2025-02-19 00:11:32: 0: STDOUT: output = self.get_bundle_settings(
2025-02-19 00:11:32: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\server_api.py", line 3626, in get_bundle_settings
2025-02-19 00:11:32: 0: STDOUT: response.raise_for_status()
2025-02-19 00:11:32: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\server_api.py", line 222, in raise_for_status
2025-02-19 00:11:32: 0: STDOUT: raise ServerError(self._data["detail"])
2025-02-19 00:11:32: 0: STDOUT: ayon_api.exceptions.ServerError: Connection timed out.
2025-02-19 00:11:32: 0: STDOUT: >>> Connected to AYON server https://our.ayon.server
2025-02-19 00:11:32: 0: STDOUT: *** AYON [1.1.1] ---
2025-02-19 00:11:32: 0: STDOUT: >>> Using AYON from [ C:\Program Files\Ynput\AYON 1.1.1 ]
2025-02-19 00:11:32: 0: STDOUT: ... AYON variant: [ production ]
2025-02-19 00:11:32: 0: STDOUT: ... AYON bundle: [ ugh01-06 ]
2025-02-19 00:11:32: 0: STDOUT: >>> loading environments ...
2025-02-19 00:11:32: 0: STDOUT: - global AYON ...
2025-02-19 00:11:32: 0: STDOUT: - for addons ...
2025-02-19 00:11:32: 0: STDOUT: !!! AYON crashed:
2025-02-19 00:11:32: 0: PYTHON: Failed to run AYON process to extract environments. Trying to use legacy arguments.
2025-02-19 00:12:24: 0: STDOUT: Traceback (most recent call last):
2025-02-19 00:12:24: 0: STDOUT: File "__startup__.py", line 120, in run
2025-02-19 00:12:24: 0: STDOUT: File "console.py", line 16, in run
2025-02-19 00:12:24: 0: STDOUT: File "start.py", line 1050, in <module>
2025-02-19 00:12:24: 0: STDOUT: File "start.py", line 1040, in main
2025-02-19 00:12:24: 0: STDOUT: File "start.py", line 687, in boot
2025-02-19 00:12:24: 0: STDOUT: File "start.py", line 597, in _start_distribution
2025-02-19 00:12:24: 0: STDOUT: File "start.py", line 503, in _run_disk_mapping
2025-02-19 00:12:24: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\_api.py", line 2031, in get_addons_studio_settings
2025-02-19 00:12:24: 0: STDOUT: return con.get_addons_studio_settings(*args, **kwargs)
2025-02-19 00:12:24: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\server_api.py", line 3661, in get_addons_studio_settings
2025-02-19 00:12:24: 0: STDOUT: output = self.get_bundle_settings(
2025-02-19 00:12:24: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\server_api.py", line 3626, in get_bundle_settings
2025-02-19 00:12:24: 0: STDOUT: response.raise_for_status()
2025-02-19 00:12:24: 0: STDOUT: File "C:\Program Files\Ynput\AYON 1.1.1\dependencies\ayon_api\server_api.py", line 222, in raise_for_status
2025-02-19 00:12:24: 0: STDOUT: raise ServerError(self._data["detail"])
2025-02-19 00:12:24: 0: STDOUT: ayon_api.exceptions.ServerError: Connection timed out.
2025-02-19 00:12:24: 0: STDOUT: >>> Connected to AYON server https://our.ayon.server
2025-02-19 00:12:24: 0: PYTHON: Traceback (most recent call last):
2025-02-19 00:12:24: 0: PYTHON: File "C:\ProgramData\Thinkbox\Deadline10\workers\SDD1111-00-1111\plugins\67b5636f9b340b017bc5f57c\GlobalJobPreLoad.py", line 611, in inject_ayon_environment
2025-02-19 00:12:24: 0: PYTHON: raise RuntimeError(
2025-02-19 00:12:24: 0: PYTHON: RuntimeError: Failed to run AYON process to extract environments.
2025-02-19 00:12:24: 0: PYTHON: !!! Injection failed.
2025-02-19 00:12:24: 0: Done executing plugin command of type 'Start Job'
=======================================================
Details
=======================================================
Date: 02/19/2025 00:12:27
Frames: 1008
Elapsed Time: 00:00:03:32
Job Submit Date: 02/18/2025 23:51:59
Job User: steve.XXXXXXXX
Average RAM Usage: 5737438720 (9%)
Peak RAM Usage: 5788966912 (9%)
Average CPU Usage: 2%
Peak CPU Usage: 7%
Used CPU Clocks (x10^6 cycles): 162427
Total CPU Clocks (x10^6 cycles): 8121338
=======================================================
Worker Information
=======================================================
Worker Name: SDD1111-00-1111
Version: v10.4.0.10 Release (aa82ae82b)
Operating System: Windows 11 Education
Running As Service: Yes
Machine User: pjUserx
IP Address: 10.40.110.107
MAC Address: 30:D0:42:F4:1D:A4
CPU Architecture: x64
CPUs: 16
CPU Usage: 0%
Memory Usage: 5.3 GB / 63.7 GB (8%)
Free Disk Space: 1.014 TB (65.292 GB on C:\, 953.614 GB on E:\, 7.012 GB on T:\, 12.869 GB on U:\)
Video Card: Intel(R) UHD Graphics 750