New collectors order

Intro

We have few main types of collectors:

  1. General → usually context plugins can run almost anytime, usually are as early as possible.
  2. Create instances → Collectors that create instances, or change their asset or task.
  3. Work with instances → Collectors that use already existing instances. Most of them also expect to have available assetEntity because their logic is based on that.
  4. Others → May or may not work with instances.

Right now we have very small range when collectors of type 3. can run because we have very wide range for collectors of type 2. At the end collector CollectAnatomyInstanceData which fills assetEntity is running very late. So, there is huge gap in the collection phase when nothing happens and then most of the logic happens during range <pyblish.api.CollectorOrder + 0.49; pyblish.api.CollectorOrder + 0.5) .

Sugestion

Dedicate an order “delimiter” before wich should be created instances, and defined their asset and task. After brief inspection of the plugins I think that range of 0.1 should be enough. In that case all instances must be created before pyblish.api.CollectorOrder - 0.4, which would be order of CollectAnatomyInstanceData plugin.

That would open range of <pyblish.api.CollectorOrder - 0.4; pyblish.api.CollectorOrder + 0.5) for instance related collection.

I think that yes, it could be nice if there’s a clearer separation of concerns to make sure there’s some expectancy of what should run roughly where.

However, changing the order is honestly cosmetics until we actually hit floating point precision errors for the ordering.

The much bigger issue is the uncertainty on knowing what data you might need is the correct data at a certain moment in time (at a certain order).

What publish plugin at what order generates what data?

A discussion on this came up e.g. here where I’d argue the biggest issue actually is:

  • Very hard to visualize currently during a publish what data gets introduced and altered where. So even when developing it’s non-trivial to find out where you can put something to rely on it.
  • Since it’s hard to visualize what plug-in touches which data it’s equally hard to really understand what plug-in during the process relied on another plug-in.

For Pyblish I originally wrote this quick prototype debug tool as shown here (bottom of that comment). It shows that it wouldn’t be impossible to have a ‘debug stepper’ UI (if the publisher event callbacks are implemented in e.g. new publisher) to visualize:

  1. What attributes get created or edited on the instance
  2. With some clever ‘python logic’ it could also detect which attributes of the instance would get accessed during which plug-in; allowing to visualize what data might be needed by that plug-in. I’m saying might because e.g. debug printing full instance.data would of course access all of its data.

With just that data in a debug stepper we could also e.g. list ALL plug-in that touched a particular key/value in instance.data


Having said that, I understand why you might want to nudge some order around. But just reading your post I still don’t understand what would end up where - and thus still doesn’t actually solve the issue of “what/where?”. Which in the end just means, yes - some orders shifted around, but nothing really changes in how easy it is to parse.

2 Likes

I wish I could have this discussion.
But let me share my thoughts and I hope they are relevant.

I’m not sure if having standard collector orders will solve the addressed problem (which I’m not sure what is it).
Also, I had a problem once with figuring out the suitable collector order. it hit me hard in this PR

I wonder if standard validator order solved any issues

I’m not tryint to solve all issues. I want to define order after which:

  • you cannot create any new instance
  • you cannot change asset and task on any instance

In other words collect anatomy instance data can be be executed, which is requirement for every collect plugin after pyblish.api.CollectorOrder + 0.49 at this moment.

Having some ‘documented’ recommendations for this I think could be worthwhile - definitely! Somehow I didn’t get this at all from your earlier posts.

It wouldn’t solve all things, but would at least be able to provide some guidelines.