Extracting path data

Development
1

Hello,

First of all thx a lot for all the work done ! I’ve been using OpenPype/Ayon for one month now and it solves many issues I’ve faced in the past, it’s a pleasure to work with it.

Now, I’m having a question regarding path resolving : is there somewhere a function/piece of code to extract actual template data from paths ? For example something that would return {“asset”: “My Asset”} if given “/prod_path/My Asset” with a template “/prod_path/{asset}” setup

The question has already been discussed here for representation paths (see this message, thx btw to Roy and Felix for publishing this code it helped me a lot), but for workfiles it seems more complicated as less info are stored in the db for them. I can find a way to have what I want by navigating through the workfile tree (aka getting all successive parents from workfile in database), but I was wondering if there is a cleaner way to do it ?

I must add that I’m not on Ayon, I’m still on OpenPype, so maybe there is already an implementation in Ayon and the ‘clean way’ is to update.

2

From solely the path I’d say you’re out of luck since the paths can be arbitrarily nested. You might be lucky designing an algorithm to work for your use cases and constraints but AYON does not offer that out of the box due to the “limitless” configuration you can do with your path templates that may not be easily reverse parsed to the templates.

The bigger question here is - why do you need it? If you’re in the context of a particular workfile you should have all that data already at your fingertips. So, more importantly how do you get in a position where all you can work with is solely the filepath itself and not e.g. the asset, task, etc. in the context to begin with?

3

As far as I know,

"""
Example: 
Get Template data and resolve work directory template.
"""
import ayon_api
from ayon_core.pipeline import (
    Anatomy,
    get_current_context,
    get_current_host_name,
)
from ayon_core.pipeline.template_data import get_template_data
from ayon_core.lib import StringTemplate


# Get Current Context
host_name = get_current_host_name()
context = get_current_context()
project_name = context["project_name"]
folder_path = context["folder_path"]
task_name = context["task_name"]

# Get Entities
project_entity = ayon_api.get_project(project_name)
folder_entity = ayon_api.get_folder_by_path(project_name, folder_path)
task_entity = ayon_api.get_task_by_name(
    project_name, folder_entity["id"], task_name
)

# Get Template Data
template_data = get_template_data(
    project_entity, folder_entity, task_entity, host_name
)

# Add Roots to template data
anatomy = Anatomy(project_name, project_entity=project_entity)
template_data["root"] = anatomy.roots

# Resolve Path 
work_dir = anatomy.templates["work"]["default"]["directory"]
work_dir = StringTemplate.format_template(work_dir, template_data)
print(work_dir)

# -------------------------------
# Alternativly, for this particualar example,
#   we already have a function to retrieve resolved work dir

from ayon_core.pipeline.workfile import get_workdir

work_dir = get_workdir(
    project_entity, folder_entity, task_entity, host_name
)

print(work_dir)

And, I did a similar thing in Houdini to add folder attributes to my template data.

4

Note that for very simple cases like this then this old python library I made a long time ago called pather might be able to do that for you, see the “Parse data from a path” section.

5

Wow, this is so cool!

6

The bigger question here is - why do you need it? If you’re in the context of a particular workfile you should have all that data already at your fingertips. So, more importantly how do you get in a position where all you can work with is solely the filepath itself and not e.g. the asset, task, etc. in the context to begin with?

My context is that I’m trying to find all the dependencies inside a generic workfile, so afaik I can’t really rely on anything apart from the path of the imported ressource. To be more specific, I have a generic scene that depends on “single files” (exr, pngs, movs, they’re often representations) and that could depend on other workfiles (via any mechanism of import by reference). For all these, I can only access file paths.

From solely the path I’d say you’re out of luck since the paths can be arbitrarily nested. You might be lucky designing an algorithm to work for your use cases and constraints but AYON does not offer that out of the box due to the “limitless” configuration you can do with your path templates that may not be easily reverse parsed to the templates.

Ok I see. Is there a lot of ways to arbitrarily nest templates ? Based on this, I feel like root path and hierarchy are the only ones that hold structure in addition of raw data (str/int). But if this property is not restricted to these two and it’s in the philosophy of Ayon/OpenPype to favor freedom over consistency, then yeah there might not be a simple solution to my problem

It bugs me though that as a result, one path could correspond to several templates. To me paths are natural ids, so it’s kinda surprising that once a path has been generated, it would become ambiguous for Ayon/OpenPype. It kinda means that we cannot rely on raw data paths, even though they are being generated and managed by Ayon/OpenPype. But maybe my use case is too particular idk

7

Thx for the code !
Unless I’m mistaken, the first code solves “getting path from template”, but I was wondering if the opposite “getting template data from path” was possible. Thx for the Houdini-related cpde, I will probably take some parts of it for later use !

8

Your library would be exactly what I’m looking for in OpenPype :star_struck: Thx for sharing I’m looking at it !

9

Anything that’s loaded through OpenPype/AYON is tracked - anything that is not… well, is not. :slight_smile:

When it’s tracked - it should be trivial from a workfile to retrieve its inputs. Actually most hosts, like Maya, Fusion, Houdini already track what was used as inputs in the publish and this is actually also displayed e.g. in the Loader for that publish (see “Dependencies” tab or whatever bottom right on the Loader UI). That metadata is actually being published along already.

Can’t find the right issue/PR that has all the nice images but this may point you in the right direction: General: Collect (generative) input links by BigRoy · Pull Request #3629 · ynput/OpenPype · GitHub

Well, it may belong to multiple templates and it’s unclear which. The issues come e.g. if you were to have a template where multiple patterns can both be nested folders, e.g. {folderPath} and {anotherThingWithMultipleFolders} if you have a pattern like {folderPath}/{anotherThingWithMultipleFolders} then a path like A/B/C you would never be able to identify which of the two starts where. Is {folderPath} just A or is it A/B? In AYON you even have keys that are optional (only if data existed when formatting the pattern). And as such if one of these were optional you can’t even identify whether it was used or not. There are many cases you can still identify the pattern and its values, but by it’s open customizable design we just can’t ensure it’s parseable and thus not ambiguous so we can’t provide you with something solid on that front by its design.

10

Well, it may belong to multiple templates and it’s unclear which. The issues come e.g. if you were to have a template where multiple patterns can both be nested folders, e.g. {folderPath} and {anotherThingWithMultipleFolders} if you have a pattern like {folderPath}/{anotherThingWithMultipleFolders} then a path like A/B/C you would never be able to identify which of the two starts where. Is {folderPath} just A or is it A/B? In AYON you even have keys that are optional (only if data existed when formatting the pattern). And as such if one of these were optional you can’t even identify whether it was used or not. There are many cases you can still identify the pattern and its values, but by it’s open customizable design we just can’t ensure it’s parseable and thus not ambiguous so we can’t provide you with something solid on that front by its design.

:+1:
I think this clearly answer the question asked : the freedom on template design is incompatible with the disambiguity needed to recover a unique template for a given path

Anything that’s loaded through OpenPype/AYON is tracked - anything that is not… well, is not. :slight_smile:
When it’s tracked - it should be trivial from a workfile to retrieve its inputs. Actually most hosts, like Maya, Fusion, Houdini already track what was used as inputs in the publish and this is actually also displayed e.g. in the Loader for that publish (see “Dependencies” tab or whatever bottom right on the Loader UI). That metadata is actually being published along already.
Can’t find the right issue/PR that has all the nice images but this may point you in the right direction: General: Collect (generative) input links by BigRoy · Pull Request #3629 · ynput/OpenPype · GitHub

So in my particular case, for a long-term solution, I cannot rely on independant raw paths but rather focus on getting dependencies tracked by Ayon/OP. Thanks for the link, I totally missed all this handling of dependencies. I seem to lack some of these dependencies in my environment, but it’s very likely to be a local problem.

I think this solves the issue, thank you very much for the help and the clear insights @BigRoy @mustafa_jafar