Publishing: better error reporting

General
21

Once the UI design is ready I’d be happy to help with refactoring the validators, etc. We’ll first need the UI prepared to be able to implement and also test whether what we’re refactoring in the messages presents nicely to the end user.

22

If you can point out a bunch of worst case validators on both sides (super large and super brief), based on your maya refactor, that would help. We’d use those for testing

23

Some that come to mind are:

  • Maya Render Settings validator massive beast of a validator with lots of logging. Also has tons of things to actually validate, thus potentially hard to write an informative documentation or report for too. Also what it actually ends up validating is heavily dependent on studio settings.
  • Maya Rig Controllers validator less big than the render settings one, but also different things to log and validate.
  • Maya Validate Frame range even smaller but also builds up a report of errors. The errors seem to be are already lengthy and detailed so I think it’s a decently simple case to test how well the new design holds up.
  • Houdini validator which reports the error through logs and returns the invalid node.. Decent test case to see how much really needs ‘code changes’ for a validator like that to make for a good report.
  • Maya Validate Transform naming suffix is a good case where reporting could be much better by maybe also reporting what part of the name is invalid. For example it could log per node what suffix it expected instead of just saying it’s invalid.

But these are great contenders too: bug: Houdini: New Publisher validations unusable validation reports · Issue #4817 · ynput/OpenPype · GitHub

Does that help?

24

Absolutely!

@iLLiCiT just FYI when you move to the adjustments of publisher

25

Wow, so much lines to read… Almost everything I had in my head was already written.

I’ll try to sum my thoughs, will try to organize them but something may not make sense without reading whole message.

What I think should be done first is change of stylesheets, make smaller texts, paddings, etc. That should help make the window smaller, which to be honest is for me the biggest issue. It works perfectly in TrayPublisher, but I want to shoot myself when I have to use it in Nuke or TVPaint because it’s just too big. Also making it smaller should give us idea how we can work with the layout a little bit more as it may change our thinking about the layout composition.

Next to make the overlay window smaller when publishing stops (but don’t when is paused) → first thing everybody does is to click on it.

These are small quick changes that does not change anything but helps with curent every day of artists.

Report page

Few facts & notes

What we were thinking when we decided for xml (as I remember it):

  • dev/artis docs difference is just PITA, nobody writes nor reads the top part of plugin file, it’s just out of the world
  • single validator can trigger different descriptions during single validation → when there are 3 checks in the plugin they can show different description messages grouped by title → title is at this moment important
  • we we’re thinking about merge of some validate plugins into single plugin → change message based on failed parts
  • change few parts of messages based on settings
  • *possibly multi-language support

Right now each instance may show different message. That is because we have no other way to show what’s wrong per instance, with logs view we could have the way, but does it handle are mentioned cases? Where to show those 50+ node names, in logs, in description, elsewhere?

Validation report and success/error report from suggestion look similar, but display different things. Success/error report show instances & context + their logs. Validation report shows validation errors by title + plugin logs + description. There are missing logs for “instances” in validation report page.

Action buttons are UX wise “unreachable” or “not related”. They should be moved to the left side as close to instances as possible( + maybe add actionable icon directly on the cards like in pyblish qml/lite/pype).

How to filter relevant artist logs

We were thinking that we could add another attribute to plugins that would work as “artist report logger”. With that we could separate logs that are meant for user.

def process(self, instance):
    self.log.info("My dev message") # Only in Details
    self.report.info("Your scene is a mess") # In Report and in Details

We can discuss what would be the name of the attribute (report in example) and what methods would have → same as Logger objects have or less? The idea is that it would use plugin’s logger (so the message will be in Details too), but also store the message for simple report shown in mock-up. This is idea which was not yet tested if is possible! I would say this is the easiest way how to differentiate logs meant for artist and for detailed information.

Idea: We can also add switch to show artist logs/all logs.

1 Like
26

I think it’s an interesting thought - however, I think we might just as well just keep log.debug for developer log.info (or whatever is above a certain level of ‘informative’) as messages for the end user. Having the report logger separation can definitely be beneficial but I admit I haven’t seen the case where debug messages were insufficient for developers and anything above it (e.g. info, warning, error should have been clear informative message that even a user understand.)

I think it’s a much better push to ensure all info logs and above are human readable instead of separating it out to separate logging handler. I don’t see much of a reason e.g. for self.report.debug calls either.

I’m not fully against separating the two but I don’t see yet why we’d require them to be separate - and if there’s no need then I’d say we shouldn’t just to keep things simpler.

100%. The A icon was far from perfect but it was a gift that you could just right click and fix, right click select instead of having to go into the details of each one.

That’s one of the things I disliked about @milan 's latest design mockups. Each report page behaves differently, one shows plug-ins left side (on invalidation), other shots instances left side (on success). Then on creating instances they are right hand side but in publish page left hand side. I’d much rather have fewer unique designs for each

It feels a bit confusing that there’s so many ‘different layouts’ to learn. Additionally I feel like there’s so much overhead to ‘designing’ all of these pages/widgets too even though it feels like they aren’t really needed to be that different. I keep getting confused on where something is in the designs/layout.

27

I don’t have strong opinions about it either. I would say it’s more about developers mental thinking, when self.report.warning is used, user will see it, so I can use self.log.warning for development and don’t have to worry about it, if I’ll forget to remove it.

I don’t think this is confusing, I would say it’s 100% right approach. In create tab you see instances on right side as a result of actions on left side, and in publish tab you select on left side source for right side.

But the fact that validation report does not show the same thing could be confusing. The only way how it could be solved is a tab switch between the 2 pages: 1st page for plugin validations, 2nd page for instance view?

Another question is if it’s a problem, that instances are not visible on validation crash? When validation fails there’s maybe not a need to see all instances as it would be unwanted noise at that moment?

28

I think being able to see at least that is beneficial. When an artist starts reporting then at least they can say my publish failed for creating a pointcache. We shouldn’t keep them in the dark completely as to what’s going on. Being able to see what failed might also allow them to say, disable that one instance for now but do publish the others if they are in a rush.

I understand the logic - but I’d say it’s just a matter of focusing on cleaning it up. For future PRs we would then take extra attention to whether the logged messages are clear - etc.

I’d also rather have one message to many visible that is ‘somewhat confusing but critical to the validation info’ to the end user and hearing about it then having the logged message lacking at all and the user being in the dark completely. (Which is kind of what’s going on now.)

29

Any ETA on when we can see a draft PR for the new design?

30

It’s been some time since (some of) the Publisher tweaks discussed above have been implemented that made the logs visible in the validation reports. However, there are still improvements that can be made.

To be sure not too much is lost through time I wanted to reference some discord discussions here:

@Innders did come back with some good pointers too and the community brought in some decent of their input too.

It’d be great if this could be used to steer the design into the next iteration and improve usabilityin production.

1 Like
31

Adding another Discord discussion related to the source of this issue Discord

If the central logs stored in OP that can be accessed from the Log Viewer showed error messages this would reduce drastically the need of artists needing to know what they need to report back because as soon as someone mentions “error” we would be able to just go to the log viewer, filter by error and see the error from the artist, without any need of copy pasting. Moreover, we would be able to periodically check the logs to have a sense of the overall health of the system by seeing how many errors artists are hitting and also tackle those errors from the technical artists that are embarrassed to share they hit an error and try fire fight it themselves…

32

You might even be able to run some data collection on the logs to find bottle necks in the publishing pipeline.

33

I created this issue about Publisher event system / callbacks to get feature parity with what Pyblish offered originally. That’d make it trivial again to do so since any studio could do anything with the publish context at any point in time when a certain event occurs - this could e.g. include posting whatever information to a dedicated place.