Publishing: better error reporting

General
1

Better and more user-friendly error reporting

Sometime error popping up from Pype are quite intimidating. Errors on user end ending up as scary Python tracebacks are not helping end-user to underestand that he made some mistake in scene and this redirect him to our support with often trivial issues.

Better error reporting and documentation can lead to overall better feeling coming from working with Pype.

GUI changes

As a first step we can modify main GUI user usually encounter errors - publishing. This is mockup how error coming from validator can be presented to user:

better_errors
better_errors801×539 138 KB

Technical aspects

This will require changes is publishing plugins, more try / except instead of asserts. We can create our own custom exception that can handle links to web documentation and markdown formatted help text (possibly taken from plugin docstrings itself as is done now for plugin description).

2

Instead of try/except why not custom assert methods?
Miniman change to the code needed.

3

So even though we’ve implemented new publisher with its new reporting, there are several issues with it:

  • The content of the messages presented to artist is wrong or missing. This needs some effort to go over all publishing plugins and in sync with documentation change/enhance.

  • Include description of the plugin. Description was taken from the plugin class docstring. It was easier by developers to maintain but it lacks more advanced formatting options. With markdown support and some simple docstring parsing we should be able to differentiate between artist related description and developer based one. Something like:

class ValidateSomething(pyblish.api.InstancePlugin):
    """This will serve as developer API documentation.

    Following classic google-style docstrings, this is easily parsed by Sphinx
    and turned into API documentation, but this shouldn't be presented to artist as it
    may include developer specific mumbo-jambo.

    Artist:
        But this way we can parse "Artist" block and render only that in publisher report
        pane, even use some **markdown** formatting. This block should describe
        for example what the validator is trying to validate and it's results.
    """
    ...

  • XML side container to hold formatted text data. Ideally with the hypertext links to documentation. Maybe the XML is overkill and we should change it to inline markdown for more easy management. Whether XML or markdown, it should be easy enough to formatting data to print it in console friendly fashion.

  • Logged messages are in separete place. Logged messages should be IMHO part of the report as in the mockup I’ve made here. We should curate these messages too to make them as much informative as possible.

  • Link to documentation, link to these community pages. If there is related page in documentation, help artist to get there directly from the publisher. Second button could automatically create new topic here. Nice to have, but the points above are more important.

Links to @BigRoy notes about publisher reports:

Additionally:

4

I think asserts should be used only in tests or some carefully selected places. The trouble with them is that they can be disabled by setting __debug__ to False or by PYTHONOPTIMIZE env var or with -O flag during the execution. The other thing is that they will stop the execution completely - that might not be such an issue in Validators, because you can look at them as special case of tests, but in some cases you might want to collect all errors before bailing out or similar. Exceptions are more verbose, but also more cleaner.

5

This is very true, but I’d say we should pick this up after the UI has settled into something approved with the right design containing the plug-in docstring, the collapsibles, the logs being accessible, etc. Because that will help define what information is best to put where in the design.

So even though this is the biggest issue for an artist - we’re massively doubling the work if we try to do this now first.

Yes. :+1:

but it lacks more advanced formatting options.

It does? I actually prototyped this last week and it seems to support most markdown types of formatting already in the new publisher. It gets it wrong sometimes but seems ok most of the time.

some simple docstring parsing we should be able to differentiate between artist related description and developer based one.

I don’t think necessarily parsing an Artist section is that necessary. We can always put developer notes at the top of the plug-in file or on e.g. the process function if it’s related to that.

:trophy: :+1: 100%. The logs shouldn’t be discarded. They should be prominent, they should just be way more informative than they were in Pyblish QML/pype.

Previously you had to oddly expand them before they came fully readable, etc. There were so much debug logs that you hardly got to the info you actually needed. Sometimes it was worse than reading through a Deadline render log. It looked nicely condensed but usually just meant you didn’t see the information you actually needed in a decent way.

We might want to much stricter about what we log as “debug” and maybe hide those by default so we only get info and warning logs that are actually user friendly and report the right information.

I do want to mention again that being able there to see what line of code logged it was massively useful with the old publisher. Also, the logs being color coded also really helped understanding what was really bad information and what was just some debug or info message.

Yes, having links to more thorough artist documentation I think could work great for complex cases and plug-ins. I’d say being able to just put in a URL and it’d allow formatting it as a link and is clickable (UI doesn’t support that yet) is quite a high priority.

The “Create forum post” button - I’d definitely put that low on the backlog. It’ll be hard to maybe avoid conflicts of people reporting the same invalidation error unless it could detect that, e.g. same OP version (or close OP version), same plug-in and same or similar error message. It’d be epic if it could bring you to e.g. a known forum topic about the error message or the plug-in if it already existed.

Overlapping forum topic Topics are now merged into this topic.

I wonder whether this topic overlaps too much with New Publisher: Report structure too verbose and loose? → This is merged now into this topic.

6

I’d say it’s overlapping 100% actually and should be merged into a big " publishing reporting topic"

7

So I’ve been working across different hosts to implement the New Publisher in the integration. I’ve fiddled some with the new publisher in Houdini and did some work in:

  • Fusion New Publisher #3892
  • Maya New Publisher #4388
  • Substance Painter Integration #4283

These are all working with the new publisher system.

Based on that development time spent + the UX of testing the publisher itself I wanted to raise a discussion about the “Publish validation report” and the way it’s structured. To me personally it’s a big bloat and I’d argue that for many cases it’s even harder than before to quickly read up on what actually is wrong with the scene.

Examples

Here are some examples:

afbeelding
afbeelding1294×420 12.7 KB

afbeelding
afbeelding1330×358 9.9 KB

afbeelding
afbeelding1302×263 7.83 KB

afbeelding
afbeelding1270×997 21 KB

What these do nicely is ‘explain’ to the user what is being validated however I’d argue that if a user publishes 10 models a day and they’ve worked with this system for a few weeks they don’t care about any length explanation - they care about the content that is invalid and especially rapidly seeing all that is wrong + repairing/selecting the invalid content as fast as possible.

Amount of extra code

Even worse maybe I think is the amount of code involved to ‘report’ it as verbose as these screenshots. For example take this commit to rewrite the Maya unit validation in such a way that it generates this new report:

afbeelding
afbeelding1480×472 23.1 KB

The old code would have generated this report:

afbeelding
afbeelding730×461 17.7 KB

To me the old structure albeit slightly more technical looking directs me much better as to where I should be looking. I’d even argue that if those old log messages were slightly more verbose in their wording that they’d appear quite a bit less technical.

Extra problematic now is that we’ll need to spend much time in generating a report that looks good in this UI but also spend the same amount of time getting structured output which provides the exact same information in a log for e.g. for command line publishing or deadline rendering.

Old reporting from code more pythonic

The old plug-in code somewhat forced a plug-ins’ source code to itself be a bit documented. Since the docstring would be what is shown at the top as documentation. To me that’s a plus. When I look at the plug-in in code I can read up what it’s about and there’s much more chance developers keep the information in sync with what the plug-in does. And even then they did not!

The same goes for the error logging. The new system requires to spend coding time to ‘format lists of data’ into bullet points of node names. Again, see the commit I mentioned earlier. Maybe it’s just that I have less experience with the new system but I have a hard feeling that it’d be easier to write the new code than the old - even though we should have a huge focus on making the development as logic and simple as possible so that it’s very easy to generate a great report and output to the end user without getting in the way of writing small plug-ins.

Lack of ‘default layout’

A big issue with the new reports is that they lack structure. The report can have any amount of headers, any amount of hints, etc and there doesn’t appear to be any best practices.

I think personally to me the clarity of the old publisher UI helped a lot. There’s documentation, there’s a specific log with more details.

I’d actually argue that the new publisher should mimic that. Each Report should just have a “Plugin documentation” area, a “Report” of the actual issue and a “Log” of the output.

Currently each report I check out for something that got invalidated has me looking around across walls of text to find the actual information I need. It needs more consistency and I’d argue more explicit sections. Having a clear ‘report’ versus ‘documentation’ structure I think personally would help a lot.

Other notes

I wanted to also copy-paste this rant I posted elsewhere during a moment of frustration:

Admittedly I’m close to a point where I’d rather go back to the old publishing system. In my test runs of creating and publishing I was able to go through the old publishing workflow quicker, with clearer visual feedback on what was actually getting processed from my scene (and also what actually finished!) and if an error occurred I honestly believe the old way was much better. Yes, for newcomers it’s nice to get a thorough documentation and report about a certain validation that failed… But if you are publishing 15 models a day you really don’t care about a lengthy explanation of the normals being invalid. You care about seeing which instance, which model, select it rapidly and fixing it. And if a validation didn’t pass that I haven’t seen before then and only then I would care about getting a lengthy report on how to solve this issue. The new system feels like it might help someone who publishes a few models in a system they are new to but it’ll just be much of a slowdown to anyone who has worked with the system for years.

Just selecting the invalid nodes for five failed validations individually took me 40 seconds with the new system on four instances. With the old it took me about 6 seconds (with the potential downside that I couldn’t filter the selection per instance but in practice I didn’t really care too much. If it didn’t pass a UV validation I just quickly want to see which meshes didn’t pass - not necessarily which meshes per which instance)

I think there are features the new system brings are very welcome, like managing creators and the settings too. The still lacking stability of the UI is something still forgivable too but the UX of the actual publishing itself to me is actually a step back.

Also wanted to note that most creators can work with your selection “Use selection” and add the currently selected nodes or things into the instance but the Publisher UI being this massive I often find myself spending a lot of time moving the UI to the corner of my screen, scaling it down, checking my scene selection, scaling/moving the UI back and the clicking create. It feels like the UI is about 75% of my screen space by default, I even have the tendency to make the window even bigger to visually parse what all the data tells me on the pages because everything is super big. The old create UI was so small that it didn’t have this issue.

With that it appears the new system appears to be designed from the ground up with a heavy focus on “ensuring no one needs to explain ever to a user how the publishing failure works” so that potentially less newcomers will ask questions about Hey, thing is red during publish - what’s up? and with that there’s been close to no focus on making sure that once those questions were answered that these same artists can go through publishing report rapidly, find the relevant data and continue to publish 15 models a day.

Yes, clear instructions and documentations on WHAT was validated and HOW that could be repaired are very nice - but once you’ve had a Missing UVs error twice all you want next time is just directly click a thing and see which nodes in your scene are missing UVs. Previously I could right click Select and voila, I had them all. Now I need to go into the report (all the way on the left) then move cursor all the way to the right to click on the select action. Imagine this for five validations going wrong on five instances - I have to do this 25 times. Previously I could’ve done this five times rapidly within a few seconds and quickly get to investigating my actual scene.

afbeelding
afbeelding748×920 141 KB

afbeelding
afbeelding1518×1080 49.4 KB

Discussion

A large part of the issues with the older publisher which I suppose was considered too ‘technical’ could have been solved e.g. by already Hiding all plug-ins which passed so that the user would only visually still see by default “what went wrong” - that to me would’ve been a better version of what “report” is now already.

Then the next step would’ve been making it clearer that the failed things actually provide actions and information. Honestly I’m perfectly fine that this might have been a “one time learning” ordeal for new user, like “press the right triangle for more documentation/information” and if it shows the “A” icon you can right click for special actions.

I feel like there should be more structure to the new publisher so that it’s clear, for a report, where the user should look for what information. It should be quick and easy to get to see verbose logs for that issue at hand if they wanted sure.

I’m very worried validators will turn to be huge plug-ins, each with an xml file even and spending a lot of writing (+ computational time) on formatting some data into a report that’s marginally better likely for new users and probably overly verbose for existing users.

8

Thank you @BigRoy for the detais. Let us chew through the suggestions and come back with our side. A lot of your points are spot on and need to be addressed, but a few feel like a point of few of developer vs user, while the user has to come first for vast majority of the cases.

Let me go through this properly first though.

9

Our experience is that the minimalistic approach can be very scary for most of the artists and it will in fact generate many requests for support either from the studio TDs or us even though those validation errors can be easily fixed by artists themselves if they were not that scary.

I get that the “old” way could be easier to orient to more experienced artists. Maybe we could slightly modify the reporting so it will behave in the “Detail” panel will show similar information as before (now it show exception):

image
image1159×818 119 KB

And you might configure it in local settings if you would like the “new and simple”, or “old and h4xx0rs fr13ndly”…

10

Just wanted to push here that the new publisher is still the main road block for us for updating to a newer version of OpenPype since Houdini was refactored to the new publisher. We’ve remained on an OP version prior to that.

I’ve elaborated some more on the issue with the new publisher’s design in the Maya refactor to new publisher PR and mentiond more Houdini specific validation design issues in a dedicated issue and discussed some of the issues with the new publisher’s design here on Discord.

In essence it boils down to what @fabiaserra wrote nicely succinct:

the current framework feels a bit too low level and requires too much code to write a useful dialog

There’s so much work involved per plug-in per validation to only result in a ‘mediocre report’ + most of the reports currently are much worse than what they were previously (especially looking at Houdini here) even after now having more lines of code.

I’d say this is high priority to get solved before e.g. the Maya refactor to new publisher PR can really be finished with elegant code design in it and likely also needed to get Houdini publishing patched up in a usable manner and wrapping up new integrations like Substance Painter.

11

We’re going through all the feedback now and distilling it into an enhancement proposal. In Essence we’d like to keep 2 sections in the report for each instance.

Structured list of issues for the lack of better therm, and a nicely formatted, description and details. There should be a place for both. And ideally a power user should be able to say they no longer need to see the verbose description, but are only interested in the structured list of issues.

We’ll come back here with a proposal

12

I’ve merged the two overlapping topics about publishing error reporting and validation reporting, as they are a major overlap. Let me chime in with my view of the matter.

When it comes to points raised by @BigRoy and @antirotor higher up here, I agree with most of the functional ones, however we’re somewhat mixing multiple topics. Artist facing report, detailed report for developer and for debugging and development and maintenance of the validators and plugins. Let’s try to break it down.

Artist facing validation report

From all the feedback it seems we need to

  1. provide a way to show simple structured logs, that could be generated from multiple places in the validator, same way as with all logging.
  2. provide at glance nicely formatted and human readable information for the artist
  3. Hide any information that is irrelevant to the situation (like all the successful plugins, that were previously visible

i feel that would be solved by simply adding an option to show info and warning logs in the report, next to the description. In that case it could even easily aggregate and show logs from all the instances processed by the same validator when it’s selected. Similar to this mockup.

I would be strongly against showing actual traceback anywhere in these logs though. There’s absolutely zero value in the for the artist. And it’s what the details page is for.

report Page - notification colapsed
report Page - notification colapsed1099×700 49 KB

Success report and error report can then take the same look, just with different colour and slightly adjusted layout

report Page - notification colapsed (2)
report Page - notification colapsed (2)1099×700 24.6 KB

report Page - notification colapsed (3)
report Page - notification colapsed (3)1099×700 26.1 KB

Details

I think this is only missing some nicer formatting of the logs same as we had in the previous UI. Artist should never feel like they need to actually go there, while TD might spend most of their time here.

It would be valuable to add option to filter out all green plugins if the TD/dev/power user chooses to.

13

As far as we never ever include traceback in the artist view, we can show simple logs.

This is super important and was the main reason for splitting out artist facing description to the xml (but the form is not important right now). Using dosctring for the artist message is terrible practice. I either exposes them to irrelevant dev information, or it leave plugin with not dev docstring at all. We simply shouldn’t mix the two.

Agreed 100%, but not on the report page. Artist should not give a damn about what line of code did what. If you’re a developer and can act on it, use the details page instead of report. (should make it possible to default into it though.)

I’d argue we need just two.

  • Plugin documentation a.k.a human readable description, with potential of extra details section that’s collapsed by default.
  • List of issues that look the same as the previous logs. Probably actually logs so they can have type of info or warning. They have to be minimal, to the point and never include actual technical information I believe.
14

With the docstrings - I think the only way to maintain ability of Sphinx to generate documentation from publishing plugins and ability to show curated information to artist is to combine them together, let Sphinx to eat it all (because I don’t see any issue about including artist based documentation to dev docs), but parse from it only artists related parts - that’s why I am suggesting the artist block. It is minor technicality in overall discussio n, but… :slight_smile:

15

If we ignore even trying to parse it - wouldn’t it maybe just make more sense to allow a plug-in to implement a classmethod that just returns the plug-ins artist facing documentation?

Less complex logic involved and seems just as simple to write (if not even more explicit)?


class Validator(pyblish.api.InstancePlugin):
    """Simple example"""
    order = pyblish.api.ValidatorOrder

    def process(self, instance):
        pass

    @classmethod
    def artist_docs(cls):
        return """
        Validate whether the scene contains a cube.

        It's not allowed to have cubes since the prince
        of spheres took over. As such, it's crucial that
        nothing in the pipeline *ever* is a cube.

        This can be automatically repaired but it will
        turn your cube into a sphere.
        """

With it being a function it could even allow the artist facing documentation to be tweaked based on studio settings, if needed.


class ValidatorWithSettingsBasedReport(pyblish.api.InstancePlugin):
    """Simple example which tweaks artist docs based on settings"""
    order = pyblish.api.ValidatorOrder

    location = "origin"

    @classmethod
    def apply_settings(cls, system_settings, project_settings):
        cls.location = project_settings["pivots"]["required_location"]

    def process(self, instance):
        pass

    @classmethod
    def artist_docs(cls):
        return f"""
        Validate all pivots of objects are at {cls.location}
        """
16

:trophy: Let’s do it.

I just wanted to ask, how do you envision e.g. reporting which 10 objects are invalid. Say we want to let them know which objects failed to match the required naming convention or which objects were missing a required UV set.

Any ideas on how we could keep the logs succinct and not huge in size but still allow to report e.g. the actual node paths, etc. I believe that’s largely why Pyblish QML originally ‘collapsed’ log entries so logs like the invalid nodes wouldn’t show up huge by default if there were say 50 invalid objects.

Edit: I see now you’ve included design mockups in this post 2 hours ago. I had missed those. Sorry.

17

Yes, that sounds great. At some point we could be as evil as add it to abstract validator class to have it always implemented.

18

@milan Any ETA on when the work will occur for this refactoring? I’m wondering what input you’d still need to get this in progress? Is there still specific feedback you’re looking for?

I really like the design that the logs are getting more prominent again - even though we should be putting a lot of effort in making sure the logs are clear messages usable for the artists.

19

Yeah that’s simply elbow grease type of work. We’ll be starting on this probably tomorrow.

20

Yes, or when lazy the method itself could just do so it’s still just the top level docstring:


class ValidateEvilThings(pyblish.api.InstancePlugin):
    """It is not allowed to have evil things."""
    order = pyblish.api.ValidatorOrder

    def process(self, instance):
        pass

    @classmethod
    def artist_docs(cls):
        return cls.__doc__