Managing large Addon collection

Hi Gang !

I’m looking for insight on a non OP/Ayon specific topic, but I’m pretty sure this crowd can enlighten me :slight_smile:
Feel free to point me to another channel/forum if this is not appropriate here !

We, at Fortiche, have a pipeline which shares similarities with pyblish, and thus with Ayon.
We have something close to pyblish’s plugins / Ayon addons for Collecting, Validating, Extracting, etc…
Over the years, the list grows – a lot.
It’s hard to get a real number but my rough estimation lands between 400 and 500 plugins.

As you might guess, this is becoming a source of challenges :smiley:
One of them is packaging and deployment, while avoiding dependency hell.
We have a solid multi-repo dev workflow, with 100+ repositories. But adding one repo per plugin does not sound like an enviable future :sweat_smile:

So I’m looking for feedback/experience on how Ynput and Studios manage their Pyblish/OP/Ayon addons collection(s).
Specifically:

  1. Do you have a collection per project, per project type (feature, serie, ad, …), for the whole studio, or a combination of those ?
  2. Do you git/package them separately, with a version number for each plugin, or do you maybe bundle them per department or per DCC and filter them at discovery time ?

Hoping this can start an interesting discussion for all Ayon users :slight_smile:
Thanks in advance !

We’ve been through multiple iterations, but I personally thing that splitting it too much eventually become detrimental to the original aim of making the management easier. It’s a fairly fine line between having enough granularity and not being able to remember where things are and who is responsible for them.

But to your points specifically.

Managing plugins on project and studio level

If I was considering splitting code up based on project it would definitely be project type, but the production reality often shows that shows need individual overrides. That’s exactly why we have, studio and project level cascading setting overrides to cover both cases.

For pyblish plugins for example we have vast majority in the same place grouped by addons, which can be a dcc, or a different integration. That’s the default layer, studio can then add (or technically also override) plugins on a studio level and lastly on a project level, with different location to source them from. In practice, the project level additions and overrides are extremely rare.

I’d argue that if many project level plugins are needed there might be something wrong with the overarching pipeline design. There is of course a value in very specific workflows for a give project, but the expense of managing almost the same, but not quite the same pipelines for various projects grows really high really fast. So we always try as hard as possible to keep the code universal and only abstract out settings that affect it in a reasonable way into the project scope.

Version control of a large plugin collection

We bundle by addon, so essentially by DCC-ish and keep that collection together. At the beginning we were considering versioning individual pyblish plugins too, but truth be told that management overhead just wasn’t viable. Especially considering we’re versioning the full product (or in AYON the individual addons, and their pyblish collections), there’s little value in versioning each tiny piece of code.

That depends how much responsibility those plugins have though. Our pyblish plugins are for the most part fairly agnostic to the rest of the pipeline version. There are data and API dependencies of course, but as far as we keep all the maya plugins together, we’re good.

Also, quite importantly, we practically never use individual plugins in isolation. They need other plugins around them to be useful, hence testing them individually is sometime not possible (outside of unit tests) and probably not very useful.

With move to AYON, we’ll be splitting each addon into it’s own repository, so we’ll end up with around 30-40 repos, which they have their dedicated owner for tracking responsibility. If someone uses the full addon, they should be good to got and it would be possible to have different ayon-maya versions in different projects (even though it’s not pretty), But if you take out and singular pyblish plugin out of that collection…you’re on you own I’m afraid.

Also to this. Yes, we filter everything at discovery time, sometimes even at runtime to some degree.

Hey Milan :slight_smile:

Sorry for the late reply !

Thanks a lot for your detailed answer. You confirm most of my feelings on the subject, and I’m glad to see we have similar points of concerns, even though the challenge is not the same with our tonload of plugins :stuck_out_tongue:

I guess we’ll explore the idea of packaging by topic/feature set, inspired by own you group by integration.

Cheers !