AYON Houdini Future

mustafa_jafar · 27 September 2023 18:22

Hello Houdini enthusiasts,

Recently, Some discussions are starting to arise about Houdini Future and Ayon tools and HDAs for Houdini

These ideas are either client requested or reported by the community or ideas that come to my head.

I’ll try to write a brief about these ideas without adding much bias / my thoughts about them.

Houdini Ideas

AYON USD workflow
Procedural publishing
Adopting Houdini’s native workflows
Color Management
Look development
Product types enhancements
Houdini workfile builder
AYON Houdini Clipboard
AYON Cache Node
AYON Load Node

AYON USD workflow

Mostly, you know about it more than me.
Basically,

Supporting/using/installing Ayon resolver to Houdini.
Better publishing UI
Add Helper Nodes e.g. Ayon Lop Loader
Support accessing published assets in Houdini’s Asset gallery while keeping them managed via AYON inventory tool?

You can follow the updates on our workgroup github repository.

Also, there’s a Houdini USD workflow prototype by @BigRoy

github.com/ynput/ayon-core

Core: Implement USD workflow with global asset/shot contributions plug-in

ynput:develop ← BigRoy:enhancement/usd_workflow

opened 10:55PM - 27 Mar 24 UTC

BigRoy

+4363 -1362

## Changelog Description - Implement USD publishing with asset and shot contr…ibutions through the publisher - for Houdini. - Implement a USD Render logic which will submit to Deadline with Husk Standalone (for which you'll need a [Husk Standalone Deadline Plug-in](https://github.com/BigRoy/HuskStandaloneSubmitter)) - Also implements a USD look publishing where textures are being copied along as resource files and are anchored relative to the USD file (similar to maya look publishing) ## Additional info Based on https://github.com/ynput/OpenPype/pull/5925 but excluding the Maya logic (that will be a separate PR) - the description and notes in that PR still apply. That PR description also has some "early WIP demo videos" which may help you along. This is currently a draft refactor from our USD Project's OpenPype logic to AYON: - [x] Published usd layer from Houdini with asset contribution as variant - [x] Published usd layer from Houdini with asset contribution without variant set - [x] Published usd layer from Houdini as shot contribution - [x] Submitted to Deadline Husk Standalone renders (with split export jobs) - [x] Publish a look with textures Note that you currently need to make sure you set your default primitive path on the USD ROPs for correctly outputting the USD files and of course making sure your prims adhere to that hierarchy, e.g. `char_hero/geo/my_cube`. # Tutorial _Click these sections to expand/collapse_ --- <details><summary>Background details + example videos</summary> - [In the original OpenPype PR description it contained quite some 'test' videos](https://github.com/ynput/OpenPype/pull/5925#issue-1995346181) - [At FMX 2024 an OpenUSD presentation was given that showed usage of what is basically the functionality of this PR](https://www.youtube.com/watch?v=1KqrIRCi_EQ). </details> --- <details><summary>Your first USD publish with AYON</summary> ## Creating an asset The producer has his 🔫 pointing at you and needs the **asset** _now!_ When we are referring to the USD asset, we mean the asset with both the model's geometry and the look's materials and textures. The USD structure (simplified) then becomes: ``` asset.usd - look.usd - model.usd ``` Where the look is applied over the model inside the asset. ### Creating the model _From a Maya, Houdini or Blender perspective_ We will want to create our geometry's contribution to the asset. For the asset structure it's critical we work within the asset's hierarchy so that USD referencing works. (_consider that USD mumbo jumbo for now_) As such, we'll always work within a root group with the current folder (asset)'s name. Say our character's folder path is `assets/char_hero` then the root group is `char_hero` We can make a hierarchy like: ``` char_hero/ geo/ body pants ``` And yes, cool heroes wear no shirts. 👕 Only pants 👖 . When done. We publish that `char_hero` as USD contribution to the `model` layer. The publishing logic will make the layering of the `model.usd` into the `asset.usd` for you automatically. So no we have: ``` asset.usd model.usd ``` When loading that in, we'll have our hero wearing pants. But it needs some 🖌️🎨 paint. ### Creating the look _From a Houdini perspective_ 1. Reference the asset so we have the model. _Note the word "reference" here. We are usually not "sublayering" the asset USD file_ 2. Make sure to add a "layer break". This special Houdini node breaks off the USD scene above it and from that moment on any changes on the existing hierarchy are just opinions that are overlaid on top of it. This means that when writing out a USD file now we're not re-exporting the geometry itself, but only any changes we make after the layer break. 3. Apply your materials to the `char_hero/geo` meshes and make sure your materials are within `char_hero` group as well. So usually you end up with: ``` char_hero/ geo/ body <- material 'char_hero/mtl/body_shader' applied pants <- material 'char_hero/mtl/pants_shader' applied mtl/ body_shader pants_shader ``` 4. Publish as the look product type (which is a USD look in Houdini). --- The producer is now happy! ## Creating a shot The producer bought another gun, and is dual wielding 🔫 🔫 now and looks fierce. They need a shot ASAP. _Houdini perspective_ - Quickly, **reference** the asset. - Throw some lights into the scene. - Set up the USD Render Settings. - Publish your USD render submission to Husk Standalone Now the tables have turned! All the producer points at you now is cash dollar bills 💵 and a medal 🥇 You are now the USD champion - until they ask you about asset variants, LODs, purposes and a layered shot structure, and more. Easy, but not for today. Ready to jump to the next level. 3...2...1 - continue `<here>` (next chapter still to write) </details> --- <details><summary>Some technical details explained by @MustafaJafar </summary> ## USD Publish Settings explained ![image](https://github.com/ynput/ayon-core/assets/20871534/a033f951-59b5-4272-a797-75c8aba6195e) I'll just focus on 4 interesting settings. 1. `target-product` which is a user editable text. 2. `target-product_department-name` which is a selection from a drop down list. and users/admins can't change the items in the list. 3. `variant-set-name` which is by default is set to `{layer}` which evaluates to the selected item in the drop down list in Num2 4. `variant-name` which is by default is set to `{variant}` as the arrow in the screenshot points. ## Asset Structure The asset structure is based on: - [ASWF Guidelines for Structuring USD Assets](https://wiki.aswf.io/display/WGUSD/Guidelines+for+Structuring+USD+Assets) - Nvidia/Omniverse's [Principles of Scalable Asset Structure in OpenUSD](https://docs.omniverse.nvidia.com/usd/latest/learn-openusd/independent/asset-structure-principles.html) But, let me summarize the results. #### When enabling Asset Contribution + Add as Variant > [!NOTE] > I love how the asset definition is readable! > Any data in the asset defintion are static. They are computed on publishing. > Paths can change when using AYON resolver. > List of departments, layer-ids and order are hardcoded and not exposed yet to settings. Product: `target-product` ``` # USD Asset {target-product}.usda └── Xform {folder[name]} ├── inherits __class__/{folder[name]} └── payload ./payload.usda # Relative path payload.usda └── mata data └── sublayers └── {target-product_department-name}.usda:ARGS:{layer-id}, {order} ``` Product: `target-product_department-name` ``` # USD Asset Layer {target-product}.usda └── Xform {folder[name]} ├── Variant Sets -> ["{variant-set-name}"] └── Variant Set "{variant-set-name}" └── Variant {variant-name} ├── reference -> Published AYON usd product variant file path └── custom data ├── ayon_order └── ayon_uri -> AYON URI for the published AYON USD product variant ``` Product: `AYON-product-variant` ``` # USD product {AYON-product-variant}.usd # It can be any hierarchy. ``` #### When enabling Asset Contribution without Add as Variant It still add a variant! I think there might be a bug or I did something wrong... _Also, I'll update this section once it works on my side._ #### When disabling Asset Contribution > [!NOTE] > It doesn't affect the version in the latest published `target-product` or `target-product_department-name`. Product: `AYON-product-variant` ``` {AYON-product-variant}.usd # It can be any usd hierarchy. ``` </details> --- <details><summary>Getting an error about "Unresolved reference path" on load?"</summary> See the comment [here](https://github.com/ynput/ayon-core/pull/295#issuecomment-2179046023) </details> --- ## Testing notes: 1. Publish and load USD files 2. Submit USD render to deadline 3. Do the testing notes suck? Check the "Tutorial" section above.

Procedural publishing

After some discussions, it was proposed to implement AYON PDG nodes that’s capable automating the publishing process taking some inspiration from flow(shotgrid)

You can find the early prototypes by @BigRoy here Houdini: Lower level publishing concepts

Also, it’s worth mentioning that we can take some inspiration from omniverse
Animation_38

Adopting Houdini’s native workflows

Also, I’d like t refer to this section as @fabiaserra’s suggestions

AYON Flavored Nodes

Sometime ago he sent a link that includes a code snippet to customize tab menu and wrap Houdini nodes so whenever an artist create a node, it’s created in Ayon flavor. I lost that link.
This can be helpful e.g. Vanilla ROP created from tab menu can be an AYON instance.

AYON flavored nodes should support publishing on demand.

I did some tinkering around it

After some google searches, I found this guide about customising-houdini-nodes.

In essence, we can use any of these solutions to add our custom parameters:
1. Define OnCreate events in HOM location.
2. Override Tab Menu on Houdini launch
3. Using a Shelf
4. Extend OPMenu.xml
More info: System for implementing Houdini Nodes in version controlled and IAC based system · Issue #591 · ynput/ayon-core · GitHub

@BigRoy made a prototype in which we can turn vanilla ROP nodes into AYON publishable nodes. Eventually, that prototype can be added as a command to OPMenu in Houdini .
Here’s the PR Houdini: Lower level publishing concepts and I made a PR to add that command to OPMenu Houdini: Add ‘Make AYON Publishable’ to OPMenu

Vanilla Deadline ROP

In order to use the vanilla deadline ROP node. we need to figure a way that only performs the publishing.
@fabiaserra’s solution was to implement ax_publisher_submitter that creates a json file and submits an AYON publish job (in compliance with our deadline addon)
Also, he override the vanilla rop nodes to add Ayon parameters.
So, the node network actually looks more like this

More info about it in his demo on Github.

I did some tinkering around it

I did quick search about it and I think such an idea requires extending the Deadline ROP itself which I have no idea how it’s achieved.

It seems that deadline rop supports specific standalone plugins, and I couldn’t find a way to extend its functionality to support AYON.
I wish if there are some exposed callback scripts to make extend it. still need to extend my search.
Here are some screenshots with placebo parameters (non-functional ‘prototype’).

HDA Creation

Implemented

Adding another menu option that opens some AYON pop window to create AYONized HDA.
More about fabia’s suggestion in this comment on GH.

Also, It was also requested by one of our clients.
I ended up with this simple implementation.

329596075-ff24a380-5cfa-4b3c-a780-be6623da41a9

Color Management

Look development

Publish materials and their associated textures.
Look assigner/manager similar to Maya’s look assigner.

Product types enhancements

There many ideas.
But for this post, I’ll mention the enhancements that are not related to a specific product type.

Support Multiple Representations for product types.

let’s take pointcaches for example.
how to collapse Pointcache (Abc) and Pointcache (Bgeo) into one creator plugin to include either or both representations in the publish.
Such feature can be useful for Model product type in Houdini.

Support Product aliases in Houdini

Some creators are a lot similar.
e.g. A staticMesh is the same as Model with fbx representation.
Both are a static 3d model. However staticMesh may require more or less of Model validators.

Houdini workfile builder

Similar to workfile builder in other addons. e.g. Maya and Hiero/Nuke Studio

AYON Cache Node

This topic is still in discussions. Any contribution is greatly appreciated.

We had a meeting about it in Houdini wg. and here are some quick notes from the meeting.

github.com

ynput/ayon-workgroups/blob/4c04ff6b3c46fe4b2385c9ff501166eb68e4ebbe/houdini-workgroup/reports/240513-report-2.md?plain=1#L14-L34


      
          ### Meeting digest:
          
          Caching node:
          - Where cache should live ?
          - Local/farm/publish exiting files ?
          - Generic types of cache ? (filecache nodes are using geometry ROPs only inside.)
          - Leverage PDG similar to filecache and labsfilecache nodes
          - Performance compared to regular filecache or labsfilecache nodes
          - Publish on demand.
          - Any necessary of validations ?
          - Cache Versioning ?
          - Should artist care about the cache path ? Note: They are able to find it through load menu.
          - Batch Caching.
          - Cache Pyblish Integration and possible network issues (some studios don't have optimized networking).
          - Setting cache intermediate location.
          
          
          Implementation Ideas: We can create 
          - AYON filecache node
          - Wrap the existing filecache nodes inside some AYON HDA
          - Extend filecache node defintion and add AYON functionality

AYON Load Node

Which is similar to Houdini: Implement Load Asset LOP HDA but generic.

Things to consider:

Add visual feedback if the load node doesn’t include the latest version.
How it behaves with Inventory Tool

AYON Houdini Clipboard

Such clipboard can be cool for Houdini artists to collaborate.
e.g. sharing .cpio
it can be a some buffer per user.
it can be a folder with one file inside. (buffer of size one)

BigRoy · 22 May 2024 19:06

Not sure which topic is a best fit for this - but I’ve set up a concept/draft PR showcasing some ideas for lower level publishing from Houdini (and part of the logic would also be capable of ‘batch ingesting’ files or publishes with a simplified entry point through the CreateContext).

Anyway, check it out here: Houdini: Lower level publishing concepts by BigRoy · Pull Request #542 · ynput/ayon-core · GitHub