I’ve run into a small issue with the review publish from blender using deadline. The render goes through and renders fine, but the publish job in deadline fails.
It asks for the “Expected files” and shows that it’s expecting the render that just finished, but then it says it’s missing.
Whenever we submit to Deadline from a host, we write the metadata .json file describing what the expected output of the render job will be, so that it can be published accordingly.
In this case it expected a file bg_1b_renderlayerMain_ViewLayer_beauty.1001.exr to exist in the folder, but it only finds: renderRenderingMainViewLayer_metadata.json.
As such, there is likely some logic error where we are assuming where the render will end up compared to what Deadline actually rendered too - because it couldn’t find the rendered files.
Your log excerpt shared here doesn’t include the folder it’s looking in… but usually it’s next to the metadata.json or somewhere close.
So:
Investigate where blender actually did render, and do those files exist?
What is the difference in path that AYON thought it would render to and what it did render to?
With that knowledge we may be able to ‘adjust’ the assumed collected path, which is being done in the code here. The current implementation is rather strict on staying close to the automatic render setup creation system - and likely you may have issues due to that.
It’s kind of weird that it names the .exr ‘v0031001.exr’, there’s no separator of the version and the frame number.
Can’t find anywhere in Ayon Studio (or Project) Settings any option to actually change the filename and/or the folder name.
@BigRoy
Hey, I gave it a test and dug around searching for bugs.
And, there’s indeed something weird.
On creating a new render instance. It creates a compositing node with the default path. I’m not yet sure if it complies to the render template (anatomy settings) and template naming profiles(core settings).
Then when validating my scene, the publisher was annoyed and changed the path. Note: how the layer name and scene file name was added to the path without really replacing the old one.
The json file associated with the render mentions two interesting things. Note how the output path is different it added one more folder with the layer name.
When my deadline finished rendering, my render file was actually saved as follows. Note how the path doesn’t include the last Expr_random_Blender_v002/ViewLayer (which is two folders far), and the render file is missing a dot between the file name and the frame number.
Confirming the bug and the temporary solution.
It’s not just Kitsu related, in general the publish / integration part on Deadline fails, unless the validator in AYON blender settings is disabled.
I have performed some tests with latest addons (ayon-core 1.1, deadline 0.5.4 and ayon-blender 0.2.8) and I can confirm its working well…please make sure you are running the latest addons as this issue was already resolved in recent days…
clear indication that the staging folder been correct is to select the rendering job and inspect some batch render via CTRL+O to inspect the folder…you should see that metadata.json + frame sequence as been rendered. if that combo key leads to error message in DL (folder doesnt exists) than its still malfunctioning due to the addons version imho.
Sorry, wasn’t in yesterday.
Looks like I’m running latest Blender and Deadline versions, but Core 1.0.14. Going to update that and see if it’s still a problem. If it is, I’ll need to disable the Deadline Output Validator.
Thanks! Btw we have found it malfunctions just for multilayer EXR so e.g. outputting to single layer (e.g. EXR, PNG etc.) works atm also with that validator enabled…
We have to resolve the issue when using multilayer EXR for sure! and thanks for your report!
I’ve tested it with the newest version of the core addon now and it still seems to be the same. Disabled the option and it seems to work now (it places the file in the correct folder and names it right), though it’s giving me another error. Looks like a conversion problem.
2025-02-11 09:00:22: 0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : A.
2025-02-11 09:00:22: 0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : B.
2025-02-11 09:00:22: 0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : G.
2025-02-11 09:00:22: 0: STDOUT: [exr @ 0000024d178739c0] Channel doesn't match layer : R.
2025-02-11 09:00:22: 0: STDOUT: [exr @ 0000024d178739c0] Missing red channel.
2025-02-11 09:00:22: 0: STDOUT: [exr @ 0000024d178739c0] Missing green channel.
2025-02-11 09:00:22: 0: STDOUT: [exr @ 0000024d178739c0] Missing blue channel.
2025-02-11 09:00:22: 0: STDOUT: Error while decoding stream #0:0: Invalid data found when processing input
2025-02-11 09:00:22: 0: STDOUT: Cannot determine format of input stream 0:0 after EOF
2025-02-11 09:00:22: 0: STDOUT: Error marking filters as finished
2025-02-11 09:00:22: 0: STDOUT: Conversion failed!
Maybe blender is rendering out the AOVs in a weird order, because it looks like the beauty render isn’t first.
Bonus points for if we can attach the exr (preferably if it’s small, say a few pixels) to the repo at some point as a ‘test’ file too for e.g. automated testing! Or provide a scene file to easily regenerate such a file.
I was thinking it was more related to the Extract Review process, specifically the ffmpeg part. This error matches the same ffmpeg error I get when trying to manually convert it.
Managed to make it go through manually by adding -layer “composite” as an input argument
But it doesn’t work when I add it through Ayon. It keeps giving me this no matter where I put the input argument.
2025-02-11 10:30:19: 0: STDOUT: RuntimeError: Executing arguments was not successful: "('C:\\Users\\User\\AppData\\Local\\Ynput\\AYON\\addons\\ayon_third_party_1.1.3\\ayon_third_party\\downloads\\windows\\ffmpeg\\bin\\ffmpeg.exe -layer CryptoObject00 -i C:\\Users\\User\\AppData\\Local\\Temp\\ay_tmp_towcsbtg\\bg_1e_renderlayerMain_ViewLayer_beauty.1001.exr -filter:v "scale=1920x1080:flags=lanczos:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2:#000000,setsar=1" -y U:\\/projectname/assets/environment/bg_1e/work/rendering/renders/blender/KBR_bg_1e_rendering_v001/ViewLayer\\bg_1e_renderlayerMain_ViewLayer_beauty.1001_png.png',)"Error:
Seems like it’s ignoring my input and wants to access the cryptomatte layer no matter what.
@BigRoy Further testing shows that it keeps looking for the next AOV when you disable the one that doesn’t work.
I want it to choose the -layer "composite". That command works in ffmpeg and it seems like it does it Ayon, too, but then it appends a new -layer command that overrides the first one and changes depending on which AOVs you have enabled in Ayon Blender Render Settings.
That’s exactly it. The -layer argument works, but for me, I get another one right after the one I’m trying to set, and I have no idea where the control for that one is.
I’ve tried changing/adding this in Core>Extract Review and I’ve checked my Deadline>Reviewable products filter, but it still has the same error.
Maybe there’s a possibility to let users define that themselves in studio settings?
We can already add one by adding an input argument in the Extract Review settings, but this one just overrides whatever we can add there.
I found that Blender doesn’t have the same order of AOVs as you set it in the compositor. It does it’s own thing. It even makes an AOV called ‘rgba’ which doesn’t get selected as a default. Funnily enough, if you delete it’s name, then it defaults to that layer being first. It’s very strange behaviour.
So maybe giving us an option to override the
if layer_name:
ffmpeg_input_args.extend(["-layer", layer_name])
and define the layer we want manually will make it easier when dealing with Blender’s weirdness
