Restriction of uniqueness of product names

[fabiaserra]

I just recently discovered what in my opinion is a big limitation and a design flaw on the OP/Ayon philosophy (just for the sake of discussion I’m going to try just talk in Ayon terms although it’s the same limitation in OP) of enforcing products to have a unique name under a publish context and I wanted to raise this topic to discuss the reasons behind that and figure out what solutions we have to avoid user errors in the future and perhaps improve the current design.

This discussion started on Discord at this point Discord

Currently we are able to choose the template tokens to use for each product type product names, the defaults are generally along the lines of {product[type]}{variant} but it’s great that we can tweak that to fit whatever studio workflow. However, if instead of using that template a pipeline like ours decides to use a more explicit workflow were artists name exactly how they want their product names and change it to just be {variant}, this shows the limitation that if an artist publishes two different product types using the same name, the system bugs out and ingests that second type under the same DB entry even though they are both integrated in two individual paths.

Real production example when doing this in OpenPype (replacing some of the names by tokens just for security purposes):

• exr representation of a render product type:
  /proj/<proj_name>/shots/410/031/<subset_name>/publish/render/<product_name>/v001/exr/<product_name>_v001.####.exr
• nuke representation of a workfile product type:
  /proj/<proj_name>/shots/410/031/<subset_name>/publish/workfile/<product_name>/v001/nuke/<product_name>_v001.nk

Both were written to individual file paths but they were treated as a single entity in MongoDB due to this code https://github.com/ynput/OpenPype/blob/aecbfa7352d42937bef5ff8929b46ab5be650eb3/openpype/plugins/publish/integrate.py#L412 only getting subsets by name and not validating whether the type being ingested is different than the retrieved subset. This specific use case would be easily fixed by just adding something like this:

        if existing_subset_doc["data"]["family"] != family:
            subset_doc = new_subset_document(
                subset_name, family, asset_doc["_id"], data, subset_id
            )

Just by this single example of a bad ingest, one could argue this is a bug in OP/Ayon since I’m able to break the system by simply tweaking the exposed template. So from this argument alone we should be adding extra validations to avoid an artist/admin doing such publish or enforcing the {product[type]} to be present ALWAYS on the product[name] of all publishes. However, while that would be a small improvement to the current system, I think both decisions would be wrong and limits the flexibility that Ayon should offer.

All the asset management systems that I have worked on don’t have a limitation on the data entries like what this is proposing. Shotgrid, ftrack and other proprietary AMS’s that I have seen in big studios like MPC and The Mill (all of them with proven years of industry expertise and backed up by a lot of artists) don’t have such strong opinions on enforcing uniqueness on the name of an entity. Even OpenAssetIO OpenAssetIO [beta]: Glossary doesn’t seem to be guided at all on doing that… see how all of the examples they provide as entity references contain a combination of fields and not something as basic as just a name!

The database entries already have the information of the {product[type]} or {task} as fields of that entry so having to add that information to the name itself is redundant and we shouldn’t have to enforce that for the names to be unique.

I also mentioned performance implications on my discussion in Discord since theoretically, a system where all product queries are bound to be queried by just the name is a much bigger query space than if you can filter out that space with other fields (i.e. type). I would need to dig deeper in the system to prove this but it would be interesting for someone from Ynput to do the following test:

Create 1000 random product names with different product types and compare the time to do the query of:

• Get product name “x”
• Get product name “x” with type “y”

So… what’s the idea behind this design and what benefits do you get from it? Why would we want to enforce this?

[BigRoy]

Just speaking from “where it came from” let me highlight some name differences for anyone getting confused:

• Before Ayon there was OpenPype, before OpenPype there was Avalon.
• Product name was subset
• Product type was family.

@fabiaserra

the system bugs out and ingests that second type under the same DB entry even though they are both integrated in two individual paths.

I just want to highlight that this is not necessarily the case. Originally product type was not a part of the template defaults. Originally they were publish/{product[name]}/ with indeed the rule that this name was allowed to be anything, as long as unique, and otherwise you’d just be adding a new version into that product entry. That new entry could be anything. This file template for publishes can STILL be customized that way - so that the family type does not become a folder of its own. We have some legacy projects where that is the case - runs just fine.

From that same point of origin, subsets weren’t of a specific product type either - it was the particular version that had the type (=families, at that point in time)) related data. As such, products themselves were typeless.

That historically originated all the way from this object model of Avalon. Note how in schemas the families data is in the Version.

This historically explains why - by initial design - products were not grouped by type and thus also not ‘unique’ by type. They were purely containers of “any versioned data”

@fabiaserra

This specific use case would be easily fixed by just adding something like this:

        if existing_subset_doc["data"]["family"] != family:
            subset_doc = new_subset_document(
                subset_name, family, asset_doc["_id"], data, subset_id
            )

Well - not really as simple. As explained above (and on Discord) the system just wasn’t designed that way. This particular check would even make it so that potentially you’re creating a new subset document each time on publish because existing_subset_doc refers to ONE of the subsets by that name - not all. Suddenly you’d now need to be checking ALL subset documents by that name.

And that is just a small part of it. All logic for products (=subsets) revolves around this idea that the name is unique. There are validators that check that logic, there is custom user code everywhere in production likely assuming this same logic.

Changing this around is a massive undertaking and a massive backwards incompatibility hurdle.

Yes, the database should’ve been guarded against this. But historically almost all publishes came through the pyblish plug-ins and the integrator would’ve ‘caught’ that if not a validator would have. Not saying it’s perfect - but you are intentionally breaking things here - and of course you can break things if you stab right at the heart of it.

---

@fabiaserra

Even OpenAssetIO OpenAssetIO [beta]: Glossary doesn’t seem to be guided at all on doing that… see how all of the examples they provide as entity references contain a combination of fields and not something as basic as just a name!

Funny - I actually don’t see a huge difference with AYON URIs to their examples.

myasset://DVR/ep01/build/prop/table?v=6&t=model&f=alembic

This example quite closely matches that of AYON. A version, a type (subset/product) and a format (representation). It does not have a version + type + productname + representation.

The same goes for:

{'job':'DVR', 'path':['ep01','build','prop','table'], 'version':'6', 'type':'model', 'format':'alembic'}

Yes, it mentions type and format - but it has no mention of a name or subset. Again, the amount of layers in the query matches that of AYON.

• Version / Type / Format
• Version / Product / Representation

Admittedly it’s confusing here that version comes first, but maybe that’s just due to the fact that they don’t have subsets - they just have different asset paths?

@fabiaserra

see how all of the examples they provide as entity references contain a combination of fields and not something as basic as just a name!

I’m still unsure what the difference would be between:

• myasset://DVR/ep01/build/prop/table?v=6&t=model&f=alembic
• ayon+entity://test/hero?product=modelMain&version=2&representation=usd

Their product just happens to be model.

---

@fabiaserra

So… what’s the idea behind this design and what benefits do you get from it?

@fabiaserra

this shows the limitation that if an artist publishes two different product types using the same name

From my examples above, don’t consider the type different from the product’s name and consider the product’s name to be its unique identifier as it was designed to be the human-readable identifier to exactly that.

Yes, querying by type would be faster/easier if the products themselves were typed from the get-go. But then again, if you know you want the main model querying by modelMain I suspect for many databases will be faster than querying by main and model assuming the unique name identifier is a unique index instead of having the name + type together to be a unique indexable entry. I don’t have hard numbers - but as always with databases, it totally depends on the use cases, the queries you actually need to make and actually also end up making

@fabiaserra

Why would we want to enforce this?

Personally, I have no real attachment to it staying the way it is - other than the fact that it’s a major refactor that likely is impossible to do in a backwards compatible manner to existing productions. As such, it’s a major change and I’d even say a next major release even. But, that could still of course be worth it if there are the right arguments to do so.

However, I still haven’t seen pros or cons that make this a hard requirement to actually start changing instead of just “I want the thing labeled differently”. Since the issue you described boils down to the names used for things not resonating with what you’re expecting from it.

As said on Discord, you could just:

• use variant as your visual label for the product (e.g. artist-facing and filepath template name)
• use type as your visual type for the product (e.g. render and workfile)
• use product[name] for what it’s designed to be, a unique identifier.

By doing so, you can query products by variant + type - and you’re doing exactly the query you’re requesting. Artist have full control over variant and there will be no conflict between variants of different types, ever.

The only real change is that you might want to display variant then in artist facing tools instead of the product[name] - but that’d then really be cosmetics at that point.

As a bonus, by keeping this design, it allows e.g. a studio to say - well, I like this never conflicting strategy - I want to even NEVER conflict if someone did the same variant and product type from a different task. Then you can say, make this product type’s product name profile behave like that, make it unique by {task}{product[type]}{variant}. In that sense, the current design might be more flexible even.

[fabiaserra]

Thank you for the quick response and taking your time to explain the history behind this decision.

Suddenly you’d now need to be checking ALL subset documents by that name.

Well, not if to check for the existing_subset_doc you actually queried by family instead! haha the better solution would be this:

        existing_subset_doc = get_subset_by_name(
            project_name, subset_name, asset_doc["_id"], family
        )

Changing this around is a massive undertaking and a massive backwards incompatibility hurdle.

Just to be clear, neither the underlying systems (mongoDB, postgreSQL) or the project managers that Ayon supports (SG, ftrack and Kitsu as well from what I see on their docs?) has a limitation on doing this, it’s only the OP frontend blocking this. I get that from the original design of having everything under a publish folder you’d need to do it this way but what’s the point inheriting that bad decision? If some studio wants everything under the same root publish folder then go ahead and make your product names unique but otherwise what’s the point of having that opinion?

Would you allow a product to evolve its type over time? What production sense does that make? A model all of a sudden could become a geoCache/point_cache, or even crazier a camera?

myasset://DVR/ep01/build/prop/table?v=6&t=model&f=alembic
This example quite closely matches that of AYON. A version, a type (subset/product) and a format (representation). It does not have a version + type + productname + representation.

It’s a bit unclear from the examples but I think table on this case works as the product name and it’s not the equivalent of Ayon’s folder hierarchy so it would be:

table: product name
v=6: version
model: type
alembic: representation

And even if there’s not strictly a name on the example, these are all independent entities split by its type:

myasset://DVR/ep01/build/prop/table?v=6&t=model&f=alembic
myasset://DVR/ep01/build/prop/table?v=6&t=geoCache&f=alembic
myasset://DVR/ep01/build/prop/table?v=6&t=shaders&f=ma
myasset://DVR/ep01/build/prop/table?v=6&t=camera&f=fbx

or
{'job':'DVR', 'path':['ep01','build','prop','table'], 'version':'6', 'type':'model', 'format':'alembic'}
{'job':'DVR', 'path':['ep01','build','prop','table'], 'version':'6', 'type':'geoCache', 'format':'alembic'}
{'job':'DVR', 'path':['ep01','build','prop','table'], 'version':'6', 'type':'shaders', 'format':'ma'}
{'job':'DVR', 'path':['ep01','build','prop','table'], 'version':'6', 'type':'camera', 'format':'fbx'}

But in Ayon URI:

ayon+entity://DVR/ep01/build/prop?product=table&version=6&representation=alembic
ayon+entity://DVR/ep01/build/prop?product=table&version=6&representation=alembic
ayon+entity://DVR/ep01/build/prop?product=table&version=6&representation=ma
ayon+entity://DVR/ep01/build/prop?product=table&version=6&representation=fbx

As opposed to how it could be:

ayon+entity://DVR/ep01/build/prop?product=table&product_type=model&version=6&representation=alembic
ayon+entity://DVR/ep01/build/prop?product=table&product_type=geoCache&version=6&representation=alembic
ayon+entity://DVR/ep01/build/prop?product=table&product_type=shaders&version=6&representation=ma
ayon+entity://DVR/ep01/build/prop?product=table&product_type=camera&version=6&representation=fbx

Another good example of “product type” being a first class citizen distinctive for the “products” is Animal Logic, check their ALab docs https://usd-alab2.s3.amazonaws.com/documentation.html where the fragments have a type first and then a name, so you could have fragment/look/book and fragment/geo/book.

But then again, if you know you want the main model querying by modelMain I suspect for many databases will be faster than querying by main and model assuming the unique name identifier is a unique index instead of having the name + type together to be a unique indexable entry.

Well the example of querying a single entry when you know exactly its unique identifier I think is not a good example of performance… for this discussion we would need to think of queries that return thousands of entries:

SELECT * FROM <project>.products WHERE product_type = 'model'
vs
SELECT * FROM <project>.products WHERE name LIKE 'model%

My SQL knowledge tells me that the former should generally be faster? I know that in most Ayon utilities this is not a concern as product_type DOES exist on the DB and we can directly use the SQL query to filter out by product type but we aren’t really making use of that when we use functions like get_product_by_name.

In AL I also remember that they use the same URI syntax on their USD asset resolver as a means to express a query so they could have a URI like this to reference all of their products of a certain type:
ayon+entity://DVR/ep01/build/prop?product=*&product_type=textures&version=latest&representation=exr

In Ayon since we can’t express the product_type that query would be the equivalent of the get_product_by_name and as I said that’s slower than filtering out by type…

Personally, I have no real attachment to it staying the way it is - other than the fact that it’s a major refactor that likely is impossible to do in a backwards compatible manner to existing productions. As such, it’s a major change and I’d even say a next major release even. But, that could still of course be worth it if there are the right arguments to do so.

Going from my suggestion to the current system would be very hard to preserve backwards compatibility but what I’m suggesting would be completely backwards compatible given that the products are ALREADY uniquely named and we are just reducing the strictness, nothing would break other than the validations and the assumptions of this being the case. On the contrary, if all of a sudden on SG or ftrack we would enforce that all “assets”/“products” need to have a unique name would break a lot of productions that don’t use OP.

Just doing a quick search of get_subset_by_name it’s not even used that much: Code search results · GitHub

However, I still haven’t seen pros or cons that make this a hard requirement to actually start changing instead of just “I want the thing labeled differently”. Since the issue you described boils down to the names used for things not resonating with what you’re expecting from it.

Cons

• Not able to have different types of entities named the same
• You can’t map an ftrack/SG product name directly to Ayon given that in the ftrack/SG databases I could have the different types named the same so when syncing that data to Ayon I need to add a prefix/suffix to make it unique
• Sooner or later an artist will mess with the data if product[type] is not part of the template → If a system has ways that allow you to fail, you will fail.

Advantages

• All the cons negated
• You can’t break the database by changing the template, ALL tokens are valid on the template without any concerns for artists messing up the DB

The only real change is that you might want to display variant then in artist facing tools instead of the product[name] - but that’d then really be cosmetics at that point.

That could be true if variant was actually stored in the DB, variant never makes it to the DB (neither Mongo or PostgreSQL), only name of the product/subset is stored… which shows the problem of this design, if I want to know what the “variant” was I would need to reverse the tokens from the template…

As a bonus, by keeping this design, it allows e.g. a studio to say - well, I like this never conflicting strategy - I want to even NEVER conflict if someone did the same variant and product type from a different task. Then you can say, make this product type’s product name profile behave like that, make it unique by {task}{product[type]}{variant} . In that sense, the current design might be more flexible even.

How is that more flexible than allowing any product to be unique regardless of the name, type, task or variant combination… each product is unique, you are literally flexible to do anything you want whereas the current system isn’t flexible in being able to name your products with just a variant!! The proof of not being flexible is me reporting it! hahaha

[mustafa_jafar]

To be honest I find it hard to jump into this discussion. but, let me try.
So, the discussion is about having product[name] as a unique identifier is not the best practice.

Also, the current design support changing the product[type] while product[name] remains. which can be confusing like in this gif (I was able to produce it by changing the subset field in Extra tab).

---

@fabiaserra

But in Ayon URI:

ayon+entity://DVR/ep01/build/prop?product=table&version=6&representation=alembic
ayon+entity://DVR/ep01/build/prop?product=table&version=6&representation=alembic

I think the Ayon URIs were inspired by the loader where product is the unique identifier product[name]
So, the regular case should be

ayon+entity://DVR/ep01/build/prop?product=cameraTest&version=6&representation=alembic
ayon+entity://DVR/ep01/build/prop?product=pointcacheTest&version=6&representation=alembic

which is not compatible with some subset/product types templates as you mentioned when using {variant} as product[name]

---

I wish if it was as simple as my poor MS paint enhanced screenshot

Having a product group where product name(s) are unique per group which doesn’t conflict with this rule Product Names are forced to be unique under the same parent but it conflicts with product[name] as a unique identifier.

[fabiaserra]

Thanks for looking into this and replicating the same bug I’m experiencing, a product changing its type over versions has zero sense and it shouldn’t be possible.

Having a product group where product name(s) are unique per group which doesn’t conflict with this rule Product Names are forced to be unique under the same parent but it conflicts with product[name] as a unique identifier.

The solution isn’t as complex as this and that would eventually suffer from the same problem. The solution is to simply avoid using get_product_by_name without being explicit of the product type and making sure the integrate creates a new product if the type is different! And removing all the validators that try enforce the uniqueness of product names

[BigRoy]

@fabiaserra

And removing all the validators that try enforce the uniqueness of product names

Well actually - not removing, but fixing them to THEN check by uniqueness of product name + product type. They are still required to be unique - but suddenly by product type and name.

Anyway, I still think the easiest ‘hack’ to get what you want is make your product names be variant.type and you’ve the exact same thing - a unique product variant, and a unique product type and they can live alongside variants of other types, etc. I’m putting hack in quotes there because by design doing that is totally fine.

---

Having looked over the code the usage of get_subset_by_name isn’t that over the top crazy but the areas where is being relied on subset_name as opposed to type + subset name is quite a large area of the code base, e.g. there’s also get_last_version_by_subset_name which is used quite a bit.

Any usages or ‘relying’ on the subset name would now need to ‘update along’ because it may now return more than one entry where it was ensured to return one before.

Some simple examples:

• AYON URIs will need to be redesigned to work with product type as you describe. This will influence anyone actively using the API and its REST API endpoints.
• Switch Asset UI would require an extra field where you also pick the ‘type’ of the representation.
• Push to project tool will need refactoring
• Some Ftrack actions need refactoring, e.g. ftrack integration, etc.
• Delete old versions tool will need refactoring
• Maya look assignments do assignments by subset names in the UIs
• Anywhere where we “in bulk” query subsets we now need to additionally ‘group’ the data by product types, so would need refactoring.

Overall - the usage doesn’t seem to amount to very crazy extends. However, I still feel like there’s a lot that needs re-testing - making it still quite a refactor.

I’ll leave it up to someone else to discuss whether it’s worth it or not.

But this advantage:

• You can’t break the database by changing the template, ALL tokens are valid on the template without any concerns for artists messing up the DB

Is actually not true. You can then suddenly mess up your files on disk if your file templates excluded product[type] - which to me, is totally fine to do. So this con:

Sooner or later an artist will mess with the data if product[type] is not part of the template → If a system has ways that allow you to fail, you will fail.

I feel is more of a con for the requirement of “product[type]” in the output. Note that, even with this changed, there’s also no database requirement that product type + product name is unique. So someone can still ingest that - resulting in the exact same error you’re describing is a con currently. It doesn’t solve the fact that if one is willing to ingest bad data into the database that they suddenly can’t - it just changes what is considered bad data. I personally find two primary indexes needing to be unique as opposed to one harder to keep track of (and a database usually as well).

Regarding the cons:

@fabiaserra

Not able to have different types of entities named the same

They can have the same variants though - they just need a unique identifier. I suppose you’d also be totally fine making all the product names uuids and just relying on variant and product[type] where you need to.

@fabiaserra

SELECT * FROM <project>.products WHERE product_type = 'model'
vs
SELECT * FROM <project>.products WHERE name LIKE 'model%

Yes, this will be quite different in speed with the first being faster - but do note that the top query is just as well possible currently in AYON - however, what you’re getting back will just always end up being unique names (but potentially the same variants across different types).

@fabiaserra

You can’t map an ftrack/SG product name directly to Ayon given that in the ftrack/SG databases I could have the different types named the same so when syncing that data to Ayon I need to add a prefix/suffix to make it unique

This to me is the only real con that is a nice-to-have. Again, variant could be the direct mapping if it were in the database - but it might just be “nice” if that’s call name as opposed to variant or whatever. In practice the difference hardly matters I suppose.

---

That could be true if variant was actually stored in the DB, variant never makes it to the DB (neither Mongo or PostgreSQL), only name of the product/subset is stored… which shows the problem of this design, if I want to know what the “variant” was I would need to reverse the tokens from the template…

This to me is the ACTUAL problem - and a huge problem actually. We should NOT be parsing arbitrary strings to get to actual data we want that MIGHT not be in the string the way we’d expect.

How is that more flexible than allowing any product to be unique regardless of the name, type, task or variant combination… each product is unique, you are literally flexible to do anything you want whereas the current system isn’t flexible in being able to name your products with just a variant!! The proof of not being flexible is me reporting it! hahaha

I don’t understand this comment - The way I understood what you were asking is:

• Allow names to match across different product types.

And as such, it doesn’t solve what you’re describing here. The product is not unique regardless of name, type, task or variant combination. It suddenly is ONLY unique by product type and name. Where “name” is what I suppose is the user typed variant?

whereas the current system isn’t flexible in being able to name your products with just a variant!!

It is - you’re just expecting that can name things based on type - but even “type” is just metadata about the ‘thing’.

---

Anyway, I don’t have any issues with refactoring it. I’m just expecting that we’ll end up with longer code since now we also need to deal with product type everywhere + the queries we’re doing most of the times might get slower, since we can’t rely on the uniqueness of the name - so now need to query for name + type instead of just name. We never really ‘filter’ by type anywhere - since type rarely has been “that crucial” to filter by (or even be possible?)

I wonder what the clever heads at Ynput think.

Also tagging @Danell @Tilix4 @max.pareschi @gabor.marinov to see if they have any opinions on the matter coming from a different angle and production use cases potentially. (Just tagging a few people that I know work in different areas or have had different requests over time).

[fabiaserra]

Well actually - not removing, but fixing them to THEN check by uniqueness of product name + product type. They are still required to be unique - but suddenly by product type and name.

On publish the user chooses the name and the type, you don’t have to validate the uniqueness, if the type and the name is the same, it’s a new version, and we already have version validations? The difference is that here this is not resulting in a completely unexpected result like it is now where a model can all of a sudden become a camera, which is why a validation is very important, and even then, we don’t have enough validations right now to avoid that from happening.

Anyway, I still think the easiest ‘hack’ to get what you want is make your product names be variant.type and you’ve the exact same thing - a unique product variant, and a unique product type and they can live alongside variants of other types, etc. I’m putting hack in quotes there because by design doing that is totally fine.

That beats the purpose of the DB, as you describe as well a bit later where the lack of variant is crucial as you want to have in your DB exactly all the singular fields that allows you to then combine them as you want (a name is not a type!). And as I have said in other places, quite a few of the types in OP are ludicrously named. Artists shouldn’t have to look on the name to figure out what type it is, that’s why there’s a column that shows you exactly the type and you can filter by it, that’s your ground truth data. If a filename has v02 on the name but it’s stored as version 3 on the DB, the version is 3! If that’s the solution you are all proposing, type must be enforced on the name to show clearly to the users this is the limitation you chose to enforce by design.

• Switch Asset UI would require an extra field where you also pick the ‘type’ of the representation.

I mean I would definitely expect that to be the case already anyway, do we allow switching types? You could switch a model by a camera?

• Maya look assignments do assignments by subset names in the UIs

That’s generally fine as in those contexts you already know the types that you allow to make assignments! You don’t do look assignments with cameras or image… it simplifies the problems that you might arise! I’m surprised we aren’t enforcing the types on those situations

Overall - the usage doesn’t seem to amount to very crazy extends. However, I still feel like there’s a lot that needs re-testing - making it still quite a refactor.

Thanks for breaking up what would need to be done, personally I see it very feasible and it’s just a slight change of mind that getting a subset by name isn’t necessarily enough, which is the case in Shotgrid, ftrack and all the AMS’s I have worked on.

Is actually not true. You can then suddenly mess up your files on disk if your file templates excluded product[type] - which to me, is totally fine to do. So this con:

The enforcement of where files live on disk to include type is a far less restriction than the template name for all your products! In fact, in the best pipelines you don’t even store your files on a filesystem but you use object based storage so you abstract the location of the files. On any case, as I said on my post if a person wanted to have that flat folder of all the products, then it’s obvious from that decision that you’d expect all the names to be unique. On the contrary, that decision makes no sense from a DB point of view.

Note that, even with this changed, there’s also no database requirement that product type + product name is unique. So someone can still ingest that - resulting in the exact same error you’re describing is a con currently. It doesn’t solve the fact that if one is willing to ingest bad data into the database that they suddenly can’t - it just changes what is considered bad data. I personally find two primary indexes needing to be unique as opposed to one harder to keep track of (and a database usually as well).

Well if we go to the lowest level APIs of the DB yeah you are right, you could have two entries that look almost the same but a different author… but all the user facing publishers expose Name, Type and Version. Can you describe some practical use cases where a user would be able to create a faulty data like I’m able to now?

Now it’s as simple as doing these two publishes:
name: book, type: model, version: 1
name: book, type: look, version: 2

They can have the same variants though - they just need a unique identifier.

How? That’s not possible now?

I suppose you’d also be totally fine making all the product names uuids and just relying on variant and product[type] where you need to.

Exacty, what’s the point of having uuids if that’s not the only thing enforcing uniqueness?

An actual display of Ayon products table:

This to me is the ACTUAL problem - and a huge problem actually. We should NOT be parsing arbitrary strings to get to actual data we want that MIGHT not be in the string the way we’d expect.

YES

The product is not unique regardless of name, type, task or variant combination. It suddenly is ONLY unique by product type and name. Where “name” is what I suppose is the user typed variant ?

Well you are right that it’s not unique by task and from what I can tell we are not even storing task on the products so for that to be unique per task you’d have to add it to the name template, although I think we shouldn’t be comparing task and type fields, type is far more significant on what the data holds than task. And task is not something we expose on the loaders whereas type is a MUST on any loading mechanism.

It is - you’re just expecting that can name things based on type - but even “type” is just metadata about the ‘thing’.

True but it’s very user error prone due to what I already said

I’m just expecting that we’ll end up with longer code since now we also need to deal with product type everywhere

Did you ever work with ftrack connect or Shotgrid toolkit pipelines? That’s literally how those work, you are explicit on all your queries and ingests

• the queries we’re doing most of the times might get slower, since we can’t rely on the uniqueness of the name - so now need to query for name + type instead of just name.

How is that slower? The search space of querying just by name is MUCH bigger than name + type. Just from a I/O POV, that’s already slower!

We never really ‘filter’ by type anywhere - since type rarely has been “that crucial” to filter by (or even be possible?)

How are product types not crucial in VFX! haha Are we a typeless language like Python? You can even look at that parallelism and the problems of typeless! You don’t want to be using a camera on a render or a model as a shader… literally we speak about types all the time in the pipeline far more than names!

---

Anyway, I’d like to get the thoughts from Ynput and others as to what solutions they propose moving forward. I personally still think it’s an unnecessary constraint to put at this early stages of Ayon and I still haven’t heard any benefits from doing it this way?

[fabiaserra]

Mentioned this discussion on this PR Fixes for "`settings` Add a map for entity's custom attributes #72" by fabiaserra · Pull Request #75 · ynput/ayon-shotgrid · GitHub as this limitation of uniqueness of entity names is now also affecting task synchronization from SG as there it’s possible to have multiple tasks with the same name

[BigRoy]

@fabiaserra

there it’s possible to have multiple tasks with the same name

That sounds like a whole other discussion on tasks having to be unique or not? Are you arguing that basically any name is now suddenly allowed to be duplicate? Or are just products and tasks somehow special? Should version names be unique? Project names?

Currently we have “folder names need to be unique under its parent” - I feel like that being true for tasks makes just as much sense. I wouldn’t even know how to represent on disk two different tasks with the same name? Or are you using that just as ‘aliases’ to track two types of statuses for what is essentially a single task?

[fabiaserra]

I’m arguing that name uniqueness shouldn’t be enforced anywhere as long as we have fields that can preserve uniqueness (id) for entities, which is true for tasks, shots, projects, episodes…

I wouldn’t even know how to represent on disk two different tasks with the same name?

We don’t have the task name as a folder hierarchy and that’s the same argument I said earlier, if the studio did want to use that as a folder hierarchy then you’d make sure to have tasks to be unique or combine it with another field to have unique folders, although I’d personally wouldn’t mind on the case that the tasks aren’t unique that they share the same parent folder. The problem with these limitations is that it makes it very tricky to support the project managers that are flexible like that.

And like I said in a prior argument, AMS ↔ disk storage shouldn’t be driving these decisions, a direct disk storage hierarchy is not even the first choice of most big studios…