To Pyblish or not to Pyblish?

Ideas
8

I remember when initially Marcus and I sat down to discuss the design for Pyblish when he started building it (partially based on some funding from our end actually). It’s a long time ago. Initially we prototyped the idea with a node based system - since everything around us was node graphs in our work - but felt that it’d hold back on the simplicity of Pyblish design philosophy. It would have e.g. required at least a UI tool to manage plug-in dependencies since it’s non-trivial to describe node graphs by code.

Pyblish define order by dependencies

By the way, nothing is actually holding you to set the `order plugins based on their dependencies. Consider e.g. doing this:

Say we define plug-ins like this:

class A(pyblish.api.ContextPlugin):
    order = pyblish.api.CollectorOrder

    def process(self, context):
        context.data["A"] = True
        print("A")

class B(pyblish.api.ContextPlugin):
    order = pyblish.api.CollectorOrder
    dependencies = ["A"]

    def process(self, context):
        assert context.data["A"] is True
        context.data["B"] = True
        print("B")

class C(pyblish.api.ContextPlugin):
    order = pyblish.api.CollectorOrder
    dependencies = ["A", "B"]

    def process(self, context):
        assert context.data["A"] is True
        assert context.data["B"] is True
        context.data["C"] = True
        print("C")

class D(pyblish.api.ContextPlugin):
    order = pyblish.api.CollectorOrder

    def process(self, context):
        context.data["D"] = True
        print("D")

And use a sorting algorithm, e.g. this:

from collections import defaultdict, namedtuple

Results = namedtuple('Results', ['sorted', 'cyclic'])


def topological_sort(dependency_pairs):
    """Sort values subject to dependency constraints"""
    num_heads = defaultdict(int)  # num arrows pointing in
    tails = defaultdict(list)  # list of arrows going out
    heads = []  # unique list of heads in order first seen
    for h, t in dependency_pairs:
        num_heads[t] += 1
        if h in tails:
            tails[h].append(t)
        else:
            tails[h] = [t]
            heads.append(h)

    ordered = [h for h in heads if h not in num_heads]
    for h in ordered:
        for t in tails[h]:
            num_heads[t] -= 1
            if not num_heads[t]:
                ordered.append(t)
    cyclic = [n for n, heads in num_heads.items() if heads]
    return Results(ordered, cyclic)

Then we can register a discovery filter to do some magic.

def shift_order_for_dependencies(plugins):
    offset = 0.0001  # amount of offset applied from dependencies

    def get_dependencies(plugin):
        return getattr(plugin, "dependencies", [])

    plugins_by_name = {plugin.__class__.__name__: plugin for plugin in plugins}
    dependency_pairs = zip(
        name, get_dependencies(plugin)
        for name, plugin in plugins_by_name.items()
    )
    result = topological_sort(dependency_pairs)
    assert not result.cyclic, "Not allowed to have cyclic dependencies"
    
    # Apply offsets in sorted order
    for plugin_name in result.ordered:
        plugin = plugins_by_name[plugin_name]
        for dependency_name in get_dependencies(plugin):
            dependency_plugin = plugins_by_name[dependency_name]
            if dependency_plugin.order > plugin.order:
               # todo: we might want to validate it does not move into another order
               #   e.g. avoid pushing it from Collector to Validator
               plugin.order = dependency_plugin.order + offset

    return plugins

pyblish.api.register_discovery_filter(shift_order_for_dependencies)
# Now publish away

Note that I didn’t test run this code at all - so consider it quick pseudocode. :slight_smile:

Visualizing data during publishing

Having said that - I think the bigger issue really is instead more of a visual debugging problem than anything else. It’s too bad it’s just non-trivial to see how values change over time, by what and what uses what data, etc. - Which actually isn’t even that hard to do - look for example at this old prototype I had quickly put together that shows what data was added and changed by a plugin.

Nothing is holding us from swapping out the actual instance.data behavior with something that also adds a notification if any data was accessed - so, that also that could be visualized.

1 Like