USD render solaris path remapping on deadline

[adrien]

Hello,

I have been having issues for the past 3 days with rendering in H21.0.512 with Karma, using the latest addons, where AYON updates a bunch of paths when submitting to deadline, causing my scene to break entirely.

I seem to kind of be able to fix this by create a fresh scene, copying my setup in the fresh scene, new publish nodes and render. And after a while (didn’t see any pattern to break it on demand), it stops working and tries to map paths that shouldn’t be mapped.

this exact scene was rendering yesterday after moving it to a fresh scene. And again after a while, without changing the publish nodes or anything pipe related, it is trying to map some paths and breaks the setup entirely.

Any hint on why it is doing that?

Thanks for your help!

2025-12-05 15:21:11:  0: STDOUT: Updated $JOB to P:/OFF_BMW/sequences/spooky/sh01/work/rendering
2025-12-05 15:21:11:  0: STDOUT: Registering pyblish target: remote
2025-12-05 15:21:15:  0: STDOUT: Detected Houdini version: (21, 0, 512)
2025-12-05 15:21:15:  0: STDOUT: ['C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\ADCDSB-01\\plugins\\6932ea3b011b58035174e34d\\hrender_dl.py', '-f', '1004', '1006', '1', '-g', '-d', '/out/usdrenderSpookyMain', '-gpu', '0', '-tempdir', 'C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\ADCDSB-01\\jobsData\\6932ea3b011b58035174e34d\\0_tempRmw2x0', '-arnoldAbortOnLicenseFail', '1', 'P:/OFF_BMW/sequences/spooky/sh01/work/rendering/OFF_BMW_spooky_sh01_rendering_v028.hiplc']
2025-12-05 15:21:15:  0: STDOUT: Start: 1004
2025-12-05 15:21:15:  0: STDOUT: End: 1006
2025-12-05 15:21:15:  0: STDOUT: Increment: 1
2025-12-05 15:21:15:  0: STDOUT: Ignore Inputs: True
2025-12-05 15:21:15:  0: STDOUT: No output specified. Output will be handled by the driver
2025-12-05 15:21:15:  0: STDOUT: GPUs: 0
2025-12-05 15:21:15:  0: STDOUT: Driver: /out/usdrenderSpookyMain
2025-12-05 15:21:15:  0: STDOUT: Input File: P:/OFF_BMW/sequences/spooky/sh01/work/rendering/OFF_BMW_spooky_sh01_rendering_v028.hiplc
2025-12-05 15:21:16:  0: STDOUT: Begin Path Mapping
2025-12-05 15:21:16:  0: STDOUT: b''
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/rop/postrender" from "pnode = hou.pwd().parent()
2025-12-05 15:21:16:  0: STDOUT: pnode.hdaModule().postExport(pnode)
2025-12-05 15:21:16:  0: STDOUT: def theExec(cmd):
2025-12-05 15:21:16:  0: STDOUT:     exec(cmd)
2025-12-05 15:21:16:  0: STDOUT: if  pnode.evalParm("tpostrender"):
2025-12-05 15:21:16:  0: STDOUT:     run_script = {'hscript':hou.hscript, 'python':theExec}
2025-12-05 15:21:16:  0: STDOUT:     lang = pnode.evalParm("lpostrender")
2025-12-05 15:21:16:  0: STDOUT:     run_script[lang](pnode.evalParm("postrender"))
2025-12-05 15:21:16:  0: STDOUT: " to "pnode = hou.pwd().parent()"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/setup_asset_info_metadata/assetthumbnail" from "`chs("../thumbnail_render/outputimage")`" to "pnode.hdaModule().postExport(pnode)"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/si_configure_geo_layer/savepath" from "`strreplace(chs("../lopoutput"),chs("../filename"),chs("../geolayer"))`" to "def theExec(cmd):"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/si_configure_mtl_layer/savepath" from "`strreplace(chs("../lopoutput"),chs("../filename"),chs("../mtllayer"))`" to "exec(cmd)"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/si_configure_payload_layer/savepath" from "`strreplace(chs("../lopoutput"),chs("../filename"),chs("../payloadlayer"))`" to "if  pnode.evalParm("tpostrender"):"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/thumbnail_render/savetodirectory_directory" from "$HOUDINI_TEMP_DIR/usd_renders/$RENDERID" to "run_script = {'hscript':hou.hscript, 'python':theExec}"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/thumbnail_render/outputimage" from "pnode.hdaModule().postExport(pnode)" to "lang = pnode.evalParm("lpostrender")"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/lantern/variantlayersdir" from "variants" to "run_script[lang](pnode.evalParm("postrender"))"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/materiallibrary1/fern_VAR1/mtlximage1/file" from "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Albedo.jpg" to ""
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/materiallibrary1/fern_VAR1/mtlximage2/file" from "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Roughness.jpg" to "`chs("../thumbnail_render/outputimage")`"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/materiallibrary1/fern_VAR1/mtlximage3/file" from "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Opacity.jpg" to "`strreplace(chs("../lopoutput"),chs("../filename"),chs("../geolayer"))`"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/materiallibrary1/fern_VAR1/mtlximage4/file" from "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Translucency.jpg" to "`strreplace(chs("../lopoutput"),chs("../filename"),chs("../mtllayer"))`"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/materiallibrary2/karmamaterial/mtlximage1/file" from "P:/assets/TREES/Licensed/Archmodels_58_evermotion/FBX-1/059.fbm/archmodels58_059_BARK.jpg" to "`strreplace(chs("../lopoutput"),chs("../filename"),chs("../payloadlayer"))`"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/materiallibrary2/karmamaterial/mtlximage2/file" from "P:/assets/TREES/Licensed/Archmodels_58_evermotion/FBX-1/059.fbm/archmodels58_059_bark_bump.jpg" to "$HOUDINI_TEMP_DIR/usd_renders/$RENDERID"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/materiallibrary3/karmamaterial/mtlximage1/file" from "P:/OFF_BMW/io/in/models/props/tripo_convert_701fee3e-141c-4901-88ba-b712b743405c.fbm/pumpkin3dmodel_basecolor.JPEG" to "`strreplace(chs("../lopoutput"),chs("../filename"),"thumbnail.png")`"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/sh01_usdCameraSH02/thumbnail_cache_dir" from "$JOB/.houdini_loader_thumbnails" to "variants"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/spotlight1/xn__inputsshapingiesfile_n8ahd" from "P:/assets/textures/ies/leomoon-dot-com_ies-lights-pack/ies-lights-pack/three-lobe-vee.ies" to "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Albedo.jpg"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/spotlight2/xn__inputsshapingiesfile_n8ahd" from "P:/assets/textures/ies/leomoon-dot-com_ies-lights-pack/ies-lights-pack/three-lobe-vee.ies" to "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Roughness.jpg"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/sublayer1/filepath1" from "P:/OFF_BMW/sequences/spooky/sh01/work/rendering/geo/usd/cam_closeup.usd" to "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Opacity.jpg"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/sublayer2/filepath1" from "P:/OFF_BMW/sequences/main/sh01/work/rendering/usd/assets/tree_33/tree_33.usd" to "P:/assets/MegaScan/Downloaded/3dplant/plants_ferns_tkytbjdia/Textures/Atlas/tkclbb2ia_4K_Translucency.jpg"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/sublayer3/filepath1" from "P:/OFF_BMW/sequences/main/sh01/work/rendering/usd/assets/tree_38/tree_38.usd" to "P:/assets/TREES/Licensed/Archmodels_58_evermotion/FBX-1/059.fbm/archmodels58_059_BARK.jpg"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/sublayer4/filepath1" from "P:/OFF_BMW/sequences/main/sh01/work/rendering/usd/assets/tree30/tree30.usd" to "P:/assets/TREES/Licensed/Archmodels_58_evermotion/FBX-1/059.fbm/archmodels58_059_bark_bump.jpg"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/sublayer5/filepath1" from "P:/OFF_BMW/sequences/main/sh01/work/rendering/usd/assets/tree_40/tree_40.usd" to "P:/OFF_BMW/io/in/models/props/tripo_convert_701fee3e-141c-4901-88ba-b712b743405c.fbm/pumpkin3dmodel_basecolor.JPEG"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/terrain/filepath1" from "P:/OFF_BMW/assets/environment/terrain/work/shading/usd/assets/terrain_3/terrain_3.usd" to "$JOB/.houdini_loader_thumbnails"
2025-12-05 15:21:17:  0: STDOUT: b''
2025-12-05 15:21:17:  0: STDOUT: End Path Mapping
2025-12-05 15:21:17:  0: STDOUT: ROP type: usdrender
2025-12-05 15:21:17:  0: STDOUT: Rendering frame 1004 to 1006
2025-12-05 15:21:17:  0: STDOUT: Error: Caught exception: The attempted operation failed.
2025-12-05 15:21:17:  0: STDOUT: Error:       Cook error in input: /stage/karmarendersettings
2025-12-05 15:21:17:  0: STDOUT: Error:       Invalid source /stage/Tree_Instancer.
2025-12-05 15:21:17:  0: STDOUT: (Error:       No prototype primitives found.)..
2025-12-05 15:21:17:  0: STDOUT: Traceback (most recent call last):
2025-12-05 15:21:17:  0: STDOUT:   File "C:\ProgramData\Thinkbox\Deadline10\workers\ADCDSB-01\plugins\6932ea3b011b58035174e34d\hrender_dl.py", line 875, in <module>
2025-12-05 15:21:17:  0: STDOUT:     rop.render( frameTuple, resolution, ignore_inputs=ignoreInputs )
2025-12-05 15:21:17:  0: STDOUT:   File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.512/houdini/python3.11libs\hou.py", line 86292, in render
2025-12-05 15:21:17:  0: STDOUT:     return _hou.RopNode_render(self, *args, **kwargs)
2025-12-05 15:21:17:  0: STDOUT:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-12-05 15:21:17:  0: STDOUT: hou.OperationFailed: The attempted operation failed.
2025-12-05 15:21:17:  0: STDOUT: Error:       Cook error in input: /stage/karmarendersettings
2025-12-05 15:21:17:  0: STDOUT: Error:       Invalid source /stage/Tree_Instancer.
2025-12-05 15:21:17:  0: STDOUT: (Error:       No prototype primitives found.)..
2025-12-05 15:21:17:  0: Done executing plugin command of type 'Render Task'
2025-12-05 15:21:17:  0: Executing plugin command of type 'End Job'
2025-12-05 15:21:17:  0: Done executing plugin command of type 'End Job'
2025-12-05 15:21:21:  Scheduler Thread - Render Thread 0 threw a major error: 
2025-12-05 15:21:21:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025-12-05 15:21:21:  Exception Details
2025-12-05 15:21:21:  RenderPluginException -- Error: Caught exception: The attempted operation failed.
2025-12-05 15:21:21:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2025-12-05 15:21:21:  RenderPluginException.Cause: JobError (2)
2025-12-05 15:21:21:  RenderPluginException.Level: Major (1)
2025-12-05 15:21:21:  RenderPluginException.HasSlaveLog: True
2025-12-05 15:21:21:  RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline10\logs\deadlineslave_renderthread_0-ADCDSB-01-0000.log
2025-12-05 15:21:21:  Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2025-12-05 15:21:21:  Exception.Data: ( )
2025-12-05 15:21:21:  Exception.Source: deadline
2025-12-05 15:21:21:  Exception.HResult: -2146233088
2025-12-05 15:21:21:    Exception.StackTrace: 
2025-12-05 15:21:21:     at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgw, CancellationToken bgx)
2025-12-05 15:21:21:     at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
2025-12-05 15:21:21:     at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aka, CancellationToken akb)

[mustafa_jafar]

I wonder if this is the same as

Mixed Farm, Deadline Path Mapping Issue

Hi there! First off, I’m very new to Ayon and pipelines in general, but I’ve been playing around with the docker image over the weekend and had a lot of fun so far. Currently im struggling to get my linux render slave to pick up jobs from deadline that are submitted by a windows machine. Rendering on the Windows machine works perfectly fine with no issues. The Linux machine always throws this error: ======================================================= Error ===============================…

Tagging @BigRoy for visibility.

[adrien]

I have read this thread already, but this is about remapping maps from windows to linux, I am not doing this at all. The remapping is happening within houdini.

[adrien]

I just took the latest working scene from yesterday night, and added all my modifications from today by copy pasting from the non working scene to the latest working scene.

and it works. all the same withini houdini. Opening v25, copying the nodes from v32 to v25 and publishing works. Same addons, same PCs, same everything.
Very frustrating…

this is what it’s supposed to do:

025-12-05 15:49:36:  0: STDOUT: Updated $JOB to P:/OFF_BMW/sequences/spooky/sh01/work/rendering
2025-12-05 15:49:36:  0: STDOUT: Registering pyblish target: remote
2025-12-05 15:49:38:  0: STDOUT: Detected Houdini version: (21, 0, 512)
2025-12-05 15:49:38:  0: STDOUT: ['C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\adcdsb-02\\plugins\\6932efeb011b58035174e38b\\hrender_dl.py', '-f', '1007', '1009', '1', '-g', '-d', '/out/usdrenderSpookyMain', '-gpu', '0', '-tempdir', 'C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\adcdsb-02\\jobsData\\6932efeb011b58035174e38b\\0_tempTZpJL0', '-arnoldAbortOnLicenseFail', '1', 'P:/OFF_BMW/sequences/spooky/sh01/work/rendering/OFF_BMW_spooky_sh01_rendering_v035.hiplc']
2025-12-05 15:49:38:  0: STDOUT: Start: 1007
2025-12-05 15:49:38:  0: STDOUT: End: 1009
2025-12-05 15:49:38:  0: STDOUT: Increment: 1
2025-12-05 15:49:38:  0: STDOUT: Ignore Inputs: True
2025-12-05 15:49:38:  0: STDOUT: No output specified. Output will be handled by the driver
2025-12-05 15:49:38:  0: STDOUT: GPUs: 0
2025-12-05 15:49:38:  0: STDOUT: Driver: /out/usdrenderSpookyMain
2025-12-05 15:49:38:  0: STDOUT: Input File: P:/OFF_BMW/sequences/spooky/sh01/work/rendering/OFF_BMW_spooky_sh01_rendering_v035.hiplc
**2025-12-05 15:49:39:  0: STDOUT: Begin Path Mapping**
**2025-12-05 15:49:40:  0: STDOUT: b''**
**2025-12-05 15:49:41:  0: STDOUT: b''**
**2025-12-05 15:49:41:  0: STDOUT: End Path Mapping**
2025-12-05 15:49:41:  0: STDOUT: ROP type: usdrender
2025-12-05 15:49:41:  0: STDOUT: Rendering frame 1007 to 1009
Success
2025-12-05 15:49:43:  0: STDOUT: [15:49:43] husk version 21.0.512

the “Begin Path Mapping” is what is causing my problems. What is it triggered from? I can’t find infos about it anywhere.

thanks :)!

[mustafa_jafar]

@adrien

the “Begin Path Mapping” is what is causing my problems. What is it triggered from? I can’t find infos about it anywhere.

That’s coming from Houdini’s Deadline Plugin.

    print("Begin Path Mapping")

    update_mode = hou.updateModeSetting()
    hou.setUpdateMode(hou.updateMode.Manual)
    
    # gather a list of all parameters that need to be pathmapped
    parms = gather_parms_to_map()
    if parms:
        pathmap_parms(tempdir, parms)

    envs = gather_envs_to_map()
    if envs:
        pathmap_envs(tempdir, envs)
    hou.setUpdateMode(update_mode)
    print("End Path Mapping")

[adrien]

that seems right, but I can’t figure out why this is being triggered or not, it has to be a usd setting/variable somewhere. If I ever find it out, I’ll let you know. I am still having the issue.

(for the time being, I commented out the function and it is working properly).

Thanks!

[adrien]

that being said, and I am not sure to 100% as I couldn’t reproduce it all the time, but I had a buggy scene render correctly if submitted to Deadline outside of the AYON context. So it could very well be a custom AYON property that triggers it.

[BigRoy]

Ah - missed this one. But this is always a good one.

I have had this in hrender_dl.py since forever:

...
def perform_pathmapping(tempdir):
    """
    Perform pathmapping on all input parameters and other select parameters
    :param tempdir: the temporary location that a text file will be written to to aid pathmapping.
    :return: None
    """
    if True:
        print("COLORBLEED EDIT: Path mapping is force disabled (hardcoded) by Colorbleed")
        return
...

Just because I faced these exact issues.

The issue came from pre/post-render script parms on ROPs if I’m not mistaken, and if those ever had multi-line values then the logic in that file went haywire.

The logic in pathmap_parms in hrender_dl.py is abysmal, and just plain wrong.

It takes a list of parms and their string values. Writes out the paths to the file like "\n".join(parm_values) thinking that then every line in the output file will be the string value of a single parm.

Then they path map that file (because that’s a Deadline subprocess call)

Then they take the file, read all the lines and do (simplified for example):

for parm, parm_value in zip(parms, file.readlines()):
      parm.set(parm_value)

However, if any of those values was actually a multiline string - a single parm may actually be holding multiple lines, not one… hence the values get offset. Hence the path mapping starts detecting there’s “different values” after remapping… and it starts applying them.

It’s a Houdini Deadline plug-in bug in that hrender_dl.py file in pathmap_parms function.

The easiest fix:

• If you do not need path mapping - kill it.
• Exclude path mapping for any string value that is itself a multi-line one.
• Or do parm_value = parm_value.replace("\n", r"\\n") before the path mapping to escape multiline. And then do parm_value = parm_value.replace("\\n", "\n")

It could be any string parameter that is multine that Houdini (and Deadline) think is a file reference. You can trace it down in the logs roughly to where the “offsetting” starts - which for you is any of the first two:

2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/setup_asset_info_metadata/assetthumbnail" from "`chs("../thumbnail_render/outputimage")`" to "pnode.hdaModule().postExport(pnode)"
2025-12-05 15:21:16:  0: STDOUT: Updating Parm "/stage/ferns/si_configure_geo_layer/savepath" from "`strreplace(chs("../lopoutput"),chs("../filename"),chs("../geolayer"))`" to "def theExec(cmd):"

I think it’s that second one because that first line could be just that single line.

[adrien]

Thanks guys, again!

That was it <3