Blender Render + Deadline reviewable on Kitsu

Hi.

I’ve run into a small issue with the review publish from blender using deadline. The render goes through and renders fine, but the publish job in deadline fails.

It asks for the “Expected files” and shows that it’s expecting the render that just finished, but then it says it’s missing.

2025-02-05 11:04:43:  0: STDOUT: ERROR:pyblish.plugin:Traceback (most recent call last):
2025-02-05 11:04:43:  0: STDOUT:   File "C:\Users\User\AppData\Local\Ynput\AYON\dependency_packages\ayon_2411151105_windows.zip\dependencies\pyblish\plugin.py", line 528, in __explicit_process
2025-02-05 11:04:43:  0: STDOUT:     runner(*args)
2025-02-05 11:04:43:  0: STDOUT:   File "C:\Users\User\AppData\Local\Ynput\AYON\addons\deadline_0.5.4\ayon_deadline\plugins\publish\global\validate_expected_and_rendered_files.py", line 77, in process
2025-02-05 11:04:43:  0: STDOUT: RuntimeError: Missing expected files: ['bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr']
2025-02-05 11:04:43:  0: STDOUT: Expected files: ['bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr']
2025-02-05 11:04:43:  0: STDOUT: Existing files: ['renderRenderingMainViewLayer_metadata.json']
2025-02-05 11:04:43:  0: STDOUT: Traceback (most recent call last):
2025-02-05 11:04:43:  0: STDOUT:   File "C:\Users\User\AppData\Local\Ynput\AYON\dependency_packages\ayon_2411151105_windows.zip\dependencies\pyblish\plugin.py", line 528, in __explicit_process
2025-02-05 11:04:43:  0: STDOUT:     runner(*args)
2025-02-05 11:04:43:  0: STDOUT:   File "<string>", line 77, in process
2025-02-05 11:04:43:  0: STDOUT: RuntimeError: Missing expected files: ['bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr']
2025-02-05 11:04:43:  0: STDOUT: Expected files: ['bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr']
2025-02-05 11:04:43:  0: STDOUT: Existing files: ['renderRenderingMainViewLayer_metadata.json']
2025-02-05 11:04:43:  0: STDOUT: !!! ERR: 2025-02-05 11:04:43,717 >>> { CLI-publish }: [ Failed ValidateExpectedFiles: Missing expected files: ['bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr']
2025-02-05 11:04:43:  0: STDOUT: Expected files: ['bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr']
2025-02-05 11:04:43:  0: STDOUT: Existing files: ['renderRenderingMainViewLayer_metadata.json'] -- ('C:\\Users\\User\\AppData\\Local\\Ynput\\AYON\\addons\\deadline_0.5.4\\ayon_deadline\\plugins\\publish\\global\\validate_expected_and_rendered_files.py', 77, 'process', '') ]

Nuke render through deadline works normally.

Maybe someone knows what the problem is and how do solve it?
Thanks!

Whenever we submit to Deadline from a host, we write the metadata .json file describing what the expected output of the render job will be, so that it can be published accordingly.

In this case it expected a file bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr to exist in the folder, but it only finds: renderRenderingMainViewLayer_metadata.json.

As such, there is likely some logic error where we are assuming where the render will end up compared to what Deadline actually rendered too - because it couldn’t find the rendered files.

Your log excerpt shared here doesn’t include the folder it’s looking in… but usually it’s next to the metadata.json or somewhere close.

So:

  1. Investigate where blender actually did render, and do those files exist?
  2. What is the difference in path that AYON thought it would render to and what it did render to?

With that knowledge we may be able to ‘adjust’ the assumed collected path, which is being done in the code here. The current implementation is rather strict on staying close to the automatic render setup creation system - and likely you may have issues due to that.

A more ‘loose’ implementation allowing much more freedom as an artist inside of Blender would be this PR: AY-7132_Refactor render workflow to manage render compositor network outputs instead of forced render structure by BigRoy · Pull Request #67 · ynput/ayon-blender · GitHub - but it’d require more extensive testing, but most of all more time on my hands to continue the implementation (which I’m rather sparse on these past few weeks).

1 Like
  • this is where the renderRenderingMainViewLayer_metadata.json is located

\assets\environment\bg_1b\work\rendering\renders\blender\KBR_bg_1b_rendering_v002\ViewLayer\KBR_bg_1b_rendering_v004\ViewLayer

  • and the KBR_bg_1b_rendering_v0031001.exr file is located two folders up, here

\assets\environment\bg_1b\work\rendering\renders\blender\KBR_bg_1b_rendering_v002\ViewLayer

It’s kind of weird that it names the .exr ‘v0031001.exr’, there’s no separator of the version and the frame number.
Can’t find anywhere in Ayon Studio (or Project) Settings any option to actually change the filename and/or the folder name.

@BigRoy
Hey, I gave it a test and dug around searching for bugs.
And, there’s indeed something weird.

  1. On creating a new render instance. It creates a compositing node with the default path. I’m not yet sure if it complies to the render template (anatomy settings) and template naming profiles(core settings).
    H:\AYON\projects\Experiments\things\random\work\Blender\renders\blender\Expr_random_Blender_v001\ViewLayer\random_renderlayerMain_ViewLayer_beauty.####
    
  2. Then when validating my scene, the publisher was annoyed and changed the path. Note: how the layer name and scene file name was added to the path without really replacing the old one.
    H:\AYON\projects\Experiments\things\random\work\Blender\renders\blender\Expr_random_Blender_v001\ViewLayer\Expr_random_Blender_v002
    
  3. The json file associated with the render mentions two interesting things. Note how the output path is different it added one more folder with the layer name.
    "files": "random_renderlayerMain_ViewLayer_beauty.1001.exr",
    "stagingDir": "{root[work]}/Experiments/things/random/work/Blender/renders/blender/Expr_random_Blender_v001/ViewLayer/Expr_random_Blender_v002/ViewLayer",
    
  4. When my deadline finished rendering, my render file was actually saved as follows. Note how the path doesn’t include the last Expr_random_Blender_v002/ViewLayer (which is two folders far), and the render file is missing a dot between the file name and the frame number.
    H:\AYON\projects\Experiments\things\random\work\Blender\renders\blender\Expr_random_Blender_v001\ViewLayer\Expr_random_Blender_v0021001.exr
    

So, I believe the deadline report is correct:

2025-02-10 10:59:22:  0: STDOUT: RuntimeError: Missing expected files: ['random_renderlayerMain_ViewLayer_beauty.1001.exr']
2025-02-10 10:59:22:  0: STDOUT: Expected files: ['random_renderlayerMain_ViewLayer_beauty.1001.exr']


Temporary solution is to disable this render validator. ayon+settings://blender/publish/ValidateDeadlinePublish


and re-create render instances (so that you keep the default path when the node is created)

Then you should get the same output as here.

Confirming the bug and the temporary solution.
It’s not just Kitsu related, in general the publish / integration part on Deadline fails, unless the validator in AYON blender settings is disabled.

I have performed some tests with latest addons (ayon-core 1.1, deadline 0.5.4 and ayon-blender 0.2.8) and I can confirm its working well…please make sure you are running the latest addons as this issue was already resolved in recent days…

clear indication that the staging folder been correct is to select the rendering job and inspect some batch render via CTRL+O to inspect the folder…you should see that metadata.json + frame sequence as been rendered. if that combo key leads to error message in DL (folder doesnt exists) than its still malfunctioning due to the addons version imho.

@marvill_1234 @vicko can you confirm you are running the latest addons for DL, core and blender? thx!

Sorry, wasn’t in yesterday.
Looks like I’m running latest Blender and Deadline versions, but Core 1.0.14. Going to update that and see if it’s still a problem. If it is, I’ll need to disable the Deadline Output Validator.

Thanks for the find and the temp solution!

Thanks! Btw we have found it malfunctions just for multilayer EXR so e.g. outputting to single layer (e.g. EXR, PNG etc.) works atm also with that validator enabled…

We have to resolve the issue when using multilayer EXR for sure! and thanks for your report!

Glad you’re fixing it, thank you for your effort!

I’ve tested it with the newest version of the core addon now and it still seems to be the same. Disabled the option and it seems to work now (it places the file in the correct folder and names it right), though it’s giving me another error. Looks like a conversion problem.

2025-02-11 09:00:22:  0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : A.
2025-02-11 09:00:22:  0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : B.
2025-02-11 09:00:22:  0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : G.
2025-02-11 09:00:22:  0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : R.
2025-02-11 09:00:22:  0: STDOUT: [exr @ 0000024d178739c0] Missing red channel.
2025-02-11 09:00:22:  0: STDOUT: [exr @ 0000024d178739c0] Missing green channel.
2025-02-11 09:00:22:  0: STDOUT: [exr @ 0000024d178739c0] Missing blue channel.
2025-02-11 09:00:22:  0: STDOUT: Error while decoding stream #0:0: Invalid data found when processing input
2025-02-11 09:00:22:  0: STDOUT: Cannot determine format of input stream 0:0 after EOF
2025-02-11 09:00:22:  0: STDOUT: Error marking filters as finished
2025-02-11 09:00:22:  0: STDOUT: Conversion failed!

Maybe blender is rendering out the AOVs in a weird order, because it looks like the beauty render isn’t first.

Thanks @vicko - that sounds very related to AY-7315_Extract Review and OIIO transcode failing to transcode media blocking publishes · Issue #989 · ynput/ayon-core · GitHub - could you upload an example EXR that failed for you, and potentially report it with the error message in that issue so it’s tracked there too as a testable use case.

Bonus points for if we can attach the exr (preferably if it’s small, say a few pixels) to the repo at some point as a ‘test’ file too for e.g. automated testing! Or provide a scene file to easily regenerate such a file.

I was thinking it was more related to the Extract Review process, specifically the ffmpeg part. This error matches the same ffmpeg error I get when trying to manually convert it.

Managed to make it go through manually by adding -layer “composite” as an input argument

But it doesn’t work when I add it through Ayon. It keeps giving me this no matter where I put the input argument.

2025-02-11 10:30:19:  0: STDOUT: RuntimeError: Executing arguments was not successful: "('C:\\Users\\User\\AppData\\Local\\Ynput\\AYON\\addons\\ayon_third_party_1.1.3\\ayon_third_party\\downloads\\windows\\ffmpeg\\bin\\ffmpeg.exe -layer CryptoObject00 -i C:\\Users\\User\\AppData\\Local\\Temp\\ay_tmp_towcsbtg\\bg_1e_renderlayerMain_ViewLayer_beauty.1001.exr -filter:v "scale=1920x1080:flags=lanczos:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2:#000000,setsar=1" -y U:\\/projectname/assets/environment/bg_1e/work/rendering/renders/blender/KBR_bg_1e_rendering_v001/ViewLayer\\bg_1e_renderlayerMain_ViewLayer_beauty.1001_png.png',)"Error:

Seems like it’s ignoring my input and wants to access the cryptomatte layer no matter what.

@BigRoy Further testing shows that it keeps looking for the next AOV when you disable the one that doesn’t work.

I want it to choose the -layer "composite". That command works in ffmpeg and it seems like it does it Ayon, too, but then it appends a new -layer command that overrides the first one and changes depending on which AOVs you have enabled in Ayon Blender Render Settings.

1 Like

Please check this AYON setting in the deadline addon:

ayon+settings://deadline/publish/ProcessSubmittedJobOnFarm/aov_filter

here is my setup (notice the blender host and its AOVs):

this way only defined AOVs should be used for reviewables…

2025-02-11 11:29:35: 0: STDOUT: DEBUG:pyblish.ExtractReview:Executing: C:\Users\User\AppData\Local\Ynput\AYON\addons\ayon_third_party_1.1.3\ayon_third_party\downloads\windows\ffmpeg\bin\ffmpeg.exe -layer “composite” -layer Denoising Albedo -i U:/projectname/assets/environment/bg_1g/work/rendering/renders/blender/KBR_bg_1g_rendering_v001/ViewLayer\bg_1g_renderlayerMain_ViewLayer_beauty.1001.exr -filter:v “scale=1920x1080:flags=lanczos:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2:#000000,setsar=1” -y U:/projectname/assets/environment/bg_1g/work/rendering/renders/blender/KBR_bg_1g_rendering_v001/ViewLayer\bg_1g_renderlayerMain_ViewLayer_beauty.1001_png.png

For some reason, Ayon adds another -layer command after my -layer command (if I specify it, if I don’t, it still adds it)

@libor.batek My setup is the same and the same error still occurs.
(it doesn’t let me post pictures, but I checked and it’s the same as yours)

Just wanted to cross-reference this issue that seems very related as well: Extract Review: Support images/videos with only named channels · Issue #776 · ynput/ayon-core · GitHub

That’s exactly it. The -layer argument works, but for me, I get another one right after the one I’m trying to set, and I have no idea where the control for that one is.

I’ve tried changing/adding this in Core>Extract Review and I’ve checked my Deadline>Reviewable products filter, but it still has the same error.

This is added by default in the codebase here which means just the get_review_layer_name is likely failing to find the layer for your use cases.

Maybe there’s a possibility to let users define that themselves in studio settings?
We can already add one by adding an input argument in the Extract Review settings, but this one just overrides whatever we can add there.

I found that Blender doesn’t have the same order of AOVs as you set it in the compositor. It does it’s own thing. It even makes an AOV called ‘rgba’ which doesn’t get selected as a default. Funnily enough, if you delete it’s name, then it defaults to that layer being first. It’s very strange behaviour.

So maybe giving us an option to override the

        if layer_name:
            ffmpeg_input_args.extend(["-layer", layer_name])

and define the layer we want manually will make it easier when dealing with Blender’s weirdness :slight_smile:

I’d say that render product in Blender needs some love. Maybe that something to discuss on

2 Likes