There is a growing need to refactor how farm related stuff is handled across the ayon/openpype - because it is hard to maintain (various pieces duplicated all over the place) and because it is difficult to extend it to other farm management software (Deadline logic is hardwire in the code). With the new publisher UI, there are options to make it easier.
How it works now
Deadline/RoyalRender are defined as modules (addons). They have their implementation and basic API (like abstract job descriptions) and they have their own set of publish plugins (like submit_maya_deadline, submit_nuke_deadline, and so on). Those submitter plugins rely on data set on instances, usually prepared by creators or collectors that are in the core.
What is wrong with it
Many things:
- Required data are over the place - some defined at the Integrator level where it is to late to validate, some are defined by creators, collectors, creators are not consistent on sets of those option on created instances and so on.
- It is difficult to separate Deadline-only logic to easily implement other farm managers.
- Very difficult support for multiple farm managers at the same time (edge case probably)
- Submit plugins can be very complex (like
submit_publish_job in Deadline) resulting in difficult debugging and are prone to errors.
- Publish plugins are registered in every DCC host, so there are âhiddenâ errors when discovering plugins because of native imports.
How it should work
Farm addons should be category in itself, they should implement IFarmAddon (analogous to IHostAddon) so we can simply identify what is enabled and what is not. They should provide per-host publish plugin directories. Creators should be farm manager agnostic - everything there is to be set related to farm should be defined by farm individual plugins. If there is something like minimal set of options needed on instance to be used to create job on farm, those should be added to instance as soon as possible with default values (overrided by settings/artist options if set).
What are you thoughts about this? There is now ongoing effort to handle some of these issues, like moving attributes to submit plugins from creators and so on. But there is still a long path to walk.
Note
This is just a kick-off for discussion. Not pretending to be complete at all
1 Like
Along with abstracting the code, it would also be good to unify the families to have better filtering and easier of plugins. A discussion has already been sparked here; https://github.com/ynput/OpenPype/pull/4623#discussion_r1135668190
As @tokejepsen wrote, a lot of interesting conversation happened here Global | Nuke: fixing farm publishing workflow by jakubjezek001 ¡ Pull Request #4623 ¡ ynput/OpenPype ¡ GitHub .
@BigRoy suggestion is to add <family>.farm and <family>.local families for filtering of instances during publishing.
But some plugins still should be able to know if instance goes to farm without knowing the families - e.g. change task status to âRenderingâ if something goes to farm. Also this approach would work only for already known families, but not plugins in custom addons, which reduce possible customization without changing core addons.
At this moment we mark instances using "farm" key in instance.data. We will need to also add something like "farm_addon" so e.g. deadline plugins will be skipped if royalrender should be used â not sure if we can do that using families?
How about implementing wildcard families?
*.farm > instance plugin would process all families that end with .farm
We will need to also add something like âfarm_addonâ so e.g. deadline plugins will be skipped if royalrender should be used â not sure if we can do that using families?
I mean, youâd either enable Deadline in the project or Royal Render. If the plugins for each are moved into their own module that would kind of solve itself. Right?
But some plugins still should be able to know if instance goes to farm without knowing the families - e.g. change task status to âRenderingâ if something goes to farm. Also this approach would work only for already known families, but not plugins in custom addons, which reduce possible customization without changing core addons.
Iâd actually argue that maybe the issue then is in how weâre defining the families? We have render, prerender, renderlayer, maxrender, etc. But maybe each of those instances shouldâve also been marked render.local or render.farm so itâs targeted at those plugins?
The wildcard families as @tokejepsen mentioned could also work but Iâm a bit worried that it might not be explicit enough as to what itâs targeting.
Or we could turn it around completely and add a custom âis_compatible(instance)â which can just return whether the plugin should show/run or not for an instance. So that we could additionally use that to just hide from the UI if it werenât to be processed at all. The same could then also be used to solve the is_active logic that one of the mixins currently does but still keeps the plugins shown. Then if no instance was compatible at the time/order the plug-in was to be run itâd get hidden from the UI since itâs not to be triggered?
I mean, youâd either enable Deadline in the project or Royal Render. If the plugins for each are moved into their own module that would kind of solve itself. Right?
You may want use royal render for maya but deadline for nuke (we donât have real usecase now). But still we miss the option to disable it per project now.
Or we could turn it around completely and add a custom âis_compatible(instance)â which can just return whether the plugin should show/run or not for an instance. So that we could additionally use that to just hide from the UI if it werenât to be processed at all. The same could then also be used to solve the is_active logic that one of the mixins currently does but still keeps the plugins shown. Then if no instance was compatible at the time/order the plug-in was to be run itâd get hidden from the UI since itâs not to be triggered?
This would be great but I think that would have to be done in pyblish. At this moment we only add abilities for UI, but publish still uses pyblish logic for processing. So we would have to have custom publish function for headless publishing.
Iâd actually argue that maybe the issue then is in how weâre defining the families? We have render, prerender, renderlayer, maxrender, etc. But maybe each of those instances shouldâve also been marked render.local or render.farm so itâs targeted at those plugins?
I agree with this. It is the cleanest way how to filter and more explicit. But at the same time I think that e.g. Submit publish job should not know all possible families.
You may want use royal render for maya but deadline for nuke (we donât have real usecase now). But still we miss the option to disable it per project now.
Ayon would instantly solve that however since it becomes an addon which can be enabled/disabled per project?
This would be great but I think that would have to be done in pyblish. At this moment we only add abilities for UI, but publish still uses pyblish logic for processing. So we would have to have custom publish function for headless publishing.
Yes, this would mean a change to pyblish-base.
I agree with this. It is the cleanest way how to filter and more explicit. But at the same time I think that e.g. Submit publish job should not know all possible families.
Exactly. So say we have an instance that we want to process using Submit Publish Job weâd just add the render.farm family to the instance. We would e.g. have a maxrender instance with families ["maxrender", "render.farm"].
Ayon would instantly solve that however since it becomes an addon which can be enabled/disabled per project?
Itâs in plan to have it, but probably wonât be there at release time.
Exactly. So say we have an instance that we want to process using Submit Publish Job weâd just add the render.farm family to the instance. We would e.g. have a maxrender instance with families [âmaxrenderâ, ârender.farmâ]
Or maybe host specific submitter could add e.g. deadline.publish family (as an example) so submit publish job could just look for that?
Or maybe host specific submitter could add e.g. deadline.publish family (as an example) so submit publish job could just look for that?
Well no - because thatâd mean the submitter would need to know about it being targeted at deadline. It should be generic so that you can switch in/out specific renderfarms Iâd say without then requiring deadline.publish, royalrender.publish, etc.
Well no - because thatâd mean the submitter would need to know about it being targeted at deadline.
But submitter have to know that. The submitter is living inside deadline addon, so the submitter know if deadlineâs submit publish job should be triggered or not â One plugin adds family to instance to trigger different plugin, thatâs completely valid approach, with bonus point that submit publish job does not have to know anything about other families.
EDITED: There is no way we would have universal submit plugin for both deadline and royalrender. Each addon must have itâs plugins for that as they have completelly different logic and probably requirements.
EDITED 2: Also it does not have to be submitter who addâs the family. The idea is that the family should say that âthis instance should have created publish job on farmâ. Can be collected universaly, or by deadline/royalrender collector. Important is that the plugin which cares about publish job should not look for render.farm but something else. The family could be publish.job.farm â deadline.publish was first shot (and an example).
Important is that the plugin which cares about publish job should not look for render.farm but something else. The family could be publish.job.farm
Yes, thatâs what I meant. Anything as generic as render.farm or farm or publish.farm or whatever could work. But render.farm I suspect would basically always mean âprepare it for publishing on the farm if neededâ (but maybe thatâs only true for rendering image sequences)
There is no way we would have universal submit plugin for both deadline and royalrender. Each addon must have itâs plugins for that as they have completelly different logic and probably requirements.
Like said above - itâd just mean they both respond to the render.farm family whether it should run or not - thatâs all.
One plugin adds family to instance to trigger different plugin, thatâs completely valid approach, with bonus point that submit publish job does not have to know anything about other families.
And with the new publisher likely this MUST be the creator so that any attribute definitions on the Deadline plug-ins actually show in the UI as unlike to the pyblish qml post process implementation the logic needs to be in the Creator instead of being able to rely on some âprecollectâ order to define the instance which is a downside to the new publisher system.
itâd just mean they both respond to the render.farm family whether it should run or not - thatâs all.
And thatâs what I disagree with. Submit publish job should not know about render.farm at all, thatâs filter for host specific submitter (Submit Maya deadline, Submit Nuke deadline, etc.).
This is current families filtering in submit publish job ["render.farm", "prerender.farm", "renderlayer", "imagesequence", "maxrender", "vrayscene"] and I think it should change to have only one family in filters, so the submit publish job does not have to know what farm families there are. It is not important for the plugin to know them, he should just create publish job. So I would change the families filtering to e.g. ["publish.job.farm"] to limit his knowledge of families from host specific filtering. Result is that "render" instances must have "render.farm" to create render job and "publish.job.farm" to create publish job.
Advantages: We can create only render jobs without publishing (e.g. renders for tests with low quality). I can in separated addon create my own submitters for my own host with my families and still am able to use Submit Publish job without any modifications. Avoids confusion between developers of different hosts: Can I add this family there or will I break someone else integration?
Another useful refactor of Deadline farm would be the change of how the jobs are sent there. It would be better if jobs are prepared for every instance, even the publish job and then at the end one integrator would send them to farm. This way you it is more consistent when something fails during the publishing and it helps to pull out sumission code from multiple plugins just into one. The same logic (but for slightly different reason is being implemented with RoyalRender).
1 Like