Ayon-resolve: can't publish compound clips

[lucas_pz]

Hey there!

We’re testing if AYON+Resolve could be implemented into our VFX pipeline, but we immediately hit a roadblock with its current implementation:

We can create and publish shots just fine when using regular clips with Chocolate label.
However, if we try to Publish a Compound Clip, the shot creation plugin immediately crashes and says it couldn’t find our clip, even if the clip name is the same.

Regular Clip

Compound Clip

Crashed plugin:

collect_shots.py (from Resolve addon)

OTIO doesn’t recognise Compound Clips?

While investigating this, I found that if I print all Clips from OTIO, the Compound clips are not appearing, only their actual content.
More specifically, when get_marker_from_clip_index calls all_clips = otio_timeline.find_clips(), the result only contains regular clips and nested clips, but not the compound clips themselves. You can test this by running print([clip.name for clip in all_clips]).

Interestingly, even if I label the clip inside this Compound Clip as chocolate, they are not found by AYON either. (That would not be a practical workflow either way but I find this interesting nonetheless).

So, my questions are:

1. Is this an OTIO limitation, or just something that needs to be implemented in ayon-resolve?
2. Have anyone found a workaround for this? We rely extensively on using Compound Clips in our current workflow, so this makes it way more challenging to implement ayon in our current VFX Pipeline, without a workaround in mind at least.

Steps to reproduce:

• Open a blank Resolve project using AYON;
• Import a random clip into the timeline;
• In the Edit tab, select the imported clip, right-click into it and select New Compound Clip... (first option);
• Add any name to this clip (doesn’t really matter);
• Right-click into this Compound Clip, go to Clip Color and change it to Chocolate;
• Go to AYON menu and click on Create...;
• In the Create window, you can just leave the settings as default, as they don’t affect this. Just click on Create>> and then Publish;
• You will immediately get an error saying that the otioClip could not be retrieved.

We tested this in both 0.3.0 and 0.4.1 versions of this addon, with the same issue happening on both.

If we do all of the same procedures, but using a regular clip instead of a compound clip, this issue does not happen.

We are still investigating potential solutions for this, but we would appreciate if anyone could shed some light into how you have been handling this if you have experienced this before.

Otherwise, we’ll keep you all posted if we find any solution to this!

Traceback:

Traceback (most recent call last):
  File "< launcher path >\dependencies\pyblish\plugin.py", line 528, in __explicit_process
    runner(*args)
  File "< ayon resolve client path >\plugins\publish\collect_shots.py", line 69, in process
    raise PublishError("Could not retrieve otioClip for shot %r", instance)
ayon_core.pipeline.publish.publish_plugins.PublishError: Could not retrieve otioClip for shot %r

PS: from what I understand, issues like this should be reported in this forum, not on the issues page in github for the plugin, but if you prefer me to post this over there as well just let me know!

[mustafa_jafar]

Just adding my 2 cents.

I’m not sure if this is a bug in the editorial creator or if this is just the way it works.

The way the AYON typicall works, is having a different creator for each obejct type in the scenefile.
So, out of curiosity, have you tried implementing a new creator that expects selecting a compound clip?

Also, I believe we may need to add some warnings or a popup window if the selection doesn’t include expected object types. (I don’t think we have api for it)

[lucas_pz]

Also, during the ayon-resolve addon development, has anyone else brought up this issue with Compound Clips, or any workaround?

I’m not sure if there is any way of using Compound Clips with AYON at the moment, but I would be surprised if we were the first ones to find this roadblock, so it seems I’m missing something obvious

[lucas_pz]

Hey @mustafa_jafar ! Thank you for your suggestion, I haven’t checked if it could be a Product / Object Type issue before this.

However, this doesn’t seem to be the case, since these Compound Clips are still getting picked by the Creator plugin just fine, as you can see them changing from Chocolate to Pink, and appearing in the list of products to be created:

(the clip thing is only an illustration of what happens, meaning that when I click on “Create”, the clip turns Pink either way)

This means that these Compound Clips are being picked by the creator, but somewhere down the line they are not being properly passed to the OTIO Timeline, thus resulting in the OtioClip not being found.

---

Also, I couldn’t find any place in the create_shot_clip code that specifically filters out Compound Clips from regular Clips as far as AYON Creator Plugins are concerned, so my understanding is that this is not the issue at hand.

Moreover, if this behaviour was intentional, a.k.a the Compound Clip needing a separate Plugin, I would imagine they would not be picked by default in the Creator plugin as it is now (picture above), so this also seems to indicate that it is a bug. Otherwise it just feels like a weird error to have in this case.

---

We will further investigate this to see where exactly the flow starts to break, so it would be easier to assess whether or not this is intentional. I’m not sure if we would be able to fix this on our end (since the OTIO workflow goes WAY over my head), but at least we can find the specific moment it breaks. Might be helpful for other studios as well

[mustafa_jafar]

@lucas_pz

Moreover, if this behaviour was intentional, a.k.a the Compound Clip needing a separate Plugin, I would imagine they would not be picked by default in the Creator plugin as it is now (picture above), so this also seems to indicate that it is a bug. Otherwise it just feels like a weird error to have in this case.

I’m not a resolve expert. but I believe you are correct: it’s either

1. the Create Publishable Clip creator doesn’t handle the compund clip properly.
2. or, the clip creator should skip compound clips and we should have another creator for compound clips.

[lucas_pz]

After some fiddling around, I think I found the sources of this issue, and it basically comes down how AYON handles the publishing process around OTIO, which was something I didn’t quite understood before. For our specific use case, this probably mean we need to redo most of the Publish plugins in Resolve to make our use case work as we expected.

Here is a breakdown of what we found with this addon, in case anyone else is having some issues with how Resolve handles data:

Creator scripts

The Creator scripts (called from the Creator plugin in AYON Menu - Create... ) work by getting all video items in the active timeline that are tagged with Chocolate colour. The logic is basically “Create a product to be published for every item in the video stacks that have Chocolate color”.
The Create logic does not check whether or not the item is a regular video clip, or compound clip, adjustment layer, etc (btw Adjustment layers flagged with Chocolate colour cause all sorts of issues but that is a separate topic);

Publish scripts

The key is that the Publish scripts work in a completely different logic of the Creator scripts. While Creator uses basically Resolve’s API to get selected clips, Publish scripts work around an OTIO workflow:

• During Collect Project and Current Timeline AYON creates a full OTIO timeline (stack) will all clips from the current timeline;
• During Collect Shots AYON will get all clips in the OTIO timeline from the step above by calling otio_timeline.find_clips(), which of course filters out anything in the timeline that is not a clip. (This is the first reason why we could not find Compound Clips since they are technically Stacks, not Clips)
• During Collect OTIO Review, AYON looks for the raw video that the Clip uses (in Resolve this would be the clip in the Media Pool, and in OTIO this is flagged as the file in the target_url property of the clip).
• During Extract OTIO file and Extract OTIO review, AYON uses the raw file from the step above to render out the specific time-range for review using ffmpeg. I haven’t look that deep into this but my understanding is that ffmpeg will render each frame separately so it can get the specific frame range of that clip as it was in the Timeline, plus the handle frames specified in the settings somewhere. ffmpeg will convert this image sequence to a mp4 for review.
• In our case, since we use ftrack integration, Integrate frack Component will then upload this generated video to ftrack as well.

(I’ve skipped a bunch of steps just to focus on what is relevant for this issue.)

---

From the steps highlighted above, we found some important factors to consider:

1. AYON is publishing the raw source video into review, using ffmpeg instead of using the regular Resolve rendering workflow as I was assuming. This means that in a way, Resolve is being basically sidestepped in the actual publishing process, to give room for a software agnostic OTIO + ffmpeg render pipeline.

2. This also explains why we cannot find compound clips: they are basically a nested timeline, not a Clip, and AYON assumes we are only handling standard Clips.

3. In theory we can force the Publish plugin to take those Compound Clips into account by appending something like this to otio/utils.py module from Resolve addon:

# add this right after  all_clips = otio_timeline.find_clips()
nested_clips = otio_timeline.find_children(descended_from_type=otio.schema.Stack)
all_clips.extend(nested_clips)

This seems to make that particular issue from the first screenshot go away. However, we get other fails in later publish scripts, since Compound Clips do not have an inherent “source footage” (target_url) because they are a collection of clips. You could technically force AYON to use one of the clips inside this compound clip as your source footage, but we prefer not to go to that route since each video can have a different duration, and dealing with frame range inside a nested timeline is a can of worms we don’t want to deal with for now.

But even if you manage to find a way to deal with that, there is still one big issue:

4. ffmpeg rendering from the raw footage means we lose all effects applied in Resolve. This can be verified by looking into what exactly goes into ftrack vs what is inside Resolve. I applied some random effects in Resolve just to see if they would be carried through into our Review and Publish, and that is clearly not the case:

(it may be worth noting that we got 8 frames of black screen before and after the review upload in ftrack. I assume it is because of handles, that in this particular file we were not accounting for.)

ffmpeg rendering means we need the actual footage on each track to be basically finished, without any editing to be made - which is fine if that’s what you’re going for, but it also means that AYON-Resolve does not support anything other than rendering either the full timeline (editorial package), or the raw version of each clip. There is no support for a VFX / Colour Grading workflow using Resolve.

---

For what we were going for, we can still use the Resolve native workflow of “Render Individual Clips”, and then use a CSV ingest from the Standalone Publisher to find a way to publish these shot clips the way we were expecting. Which is fine, but it was unexpected that basically the only thing that this plugin actually uses from Resolve is the actual cuts in the footage, and nothing else.

We would appreciate to know how other studios are integrating AYON-resolve in their pipeline with these things in mind, since a “Render this clip with all effects and whatnot” is something that would help us a lot.

I will try to create some Publish plugins specifically for this, and share any update that we have!

Thank you!

tl;dr: we realized that AYON-Resolve has a very specific workflow in mind that is not what we were expecting, so we will try to develop a custom publish workflow to take care of what we need.

[jiri.sin]

Hello @lucas_pz ,

Did you consider to use the Resolve native workflow of “Render Individual Clips”, and load & re-conform the renders directly in Resolve? Some might find it more comfortable than CSV ingest. Then you can use OTIO based publisher.

I do use this workflow anytime I need to “transcode” raw footage to Exr sequences for vfx.