Kitsu Sync Service

Development
, ,
1

In one day we have now gotten 3 PRs to get automated sync between Kitsu and OP!
I wanted to create this thread to let us better communicate on how to move forward and not work over each other.
All 3 does things quite different so I think it’s best to open a communication here so we don’t do a bunch of development in vain.
Here is the 3 PRs:

Leecher Service by @Danell (me): Add leecher service (gazu event listener) by EmberLightVFX · Pull Request #25 · ynput/ayon-kitsu · GitHub
Ayon-Kitsu + Auto-Sync by Ahmed Gamal : Ayon-Kitsu + Auto-Sync by Agamal17 · Pull Request #26 · ynput/ayon-kitsu · GitHub
Create, Update and Delete of folders and tasks by @scott_mcdonnell_jam: Enhancement/delete kitsu entities by scottmcdonnell · Pull Request #28 · ynput/ayon-kitsu · GitHub

From my understanding, these are how each PR does the syncing. If I have misunderstood anything, please correct me:
Leecher Service adds a new kitsu service called Leecher. The Leecher service handles all the gazu listening and creates an Ayon event when catching something.
The event is then picked up by the Processor service and passes it over to the correct handler that creates everything for both Ayon and Kitsu as needed. This means we can add more custom events and handlers in the future if needed. It also means we will have a two-way communication between Kitsu and Ayon.
It’s 100% inspired on how the Shotgrid and Ftrack services work. If the Processor service would go down or get updated the Leecher will still fetch all events and queue them up for when Processor gets up and running again.
The services uses Gazu API for Kitsu communication and ayon_api + ayon entities for Ayon communication
Currently there is no update with the UI but in the end the user will be able to select what projects to sync, or sync all open projects from Kitsu

Ayon-Kitsu + Auto-Sync removes all frontend and instead adds the gazu event listener. When something gets added, updated or removed a full sync happens. The sync-code is the same as the old full_sync that uses zou API instead of the gazu API to fetch data from Kitsu and uses postgres commands to modify the Ayon code.

Create, Update and Delete of folders and tasks adds a bunch of pytests to test all possible syncing. It also splits out the full_sync into individual functions for assets, episodes, sequences, shots and tasks. It communicates to Ayon over ayon_api.post.

My thoughts on each PR:
Ayon-Kitsu + Auto-Sync updates the current code quick and dirty to add automatic synchronization. It does one thing and it does what it’s suppose to do and not much more.

Create, Update and Delete of folders and tasks gives us a bit more flexibility compared with the current full_sync code. Here each event gets its own function that handles the sync. It makes it easier to implement future features from Kitsu.

Both of these uses the Processor service to directly use server code to modify Ayon. There is no two-way communication.

Leecher Service is based on how the Shotgrid and Ftrack addon is built. The code is more split up compared with the other two PRs and allows for great flexibility in the future. It also only uses the services to code to communicate with Kitsu and Ayon and doesn’t touch the server code at all.

My conclusion:
I think Leecher Service would be the best way forward as its built exactly how the Shotgrid and Ftrack addons are built. It will give us a two-way communication. Any update to Ayon that might change how the Shotgrid and Ftrack addons work will easy be added to the Kitsu addon. The use of handlers opens up special edge-cases where external tools can modify both Kitsu and Ayon at the same time.
Create, Update and Delete of folders and tasks gives us great pytest-functionality and I would love to have that in the final product so we super quick can test any future development. (Currently I have setup a mouse-auto-clicker that creates new projects, tasks, comments etc for my tests lol).

I would love to hear both from Ahmed Gamal and @scott_mcdonnell_jam l but also Ynput themself on your thougts for moving forward with the Kitsu sync integration.

1 Like
2

There is one big advantage with doing it with a background process as we do it in the simple implementation now… no extra service management for the admins.

That being said, my general feeling is that leecher and processor is the way to go.

3

Its a shame to have worked separately on this and not being able to combine code as yet.
With the leacher PR is it working now to fullsync and update ayon with kitsu changes or is there more work to be done? Can you list the tasks done and any remaining?

4

Good idea! I’ll update the PR with this.
There is quite some work left to be done but it will also cover all cases and also sync user assignments etc.

5

Can you explain how the PR now changes the process - when a gazu ‘task.create’ event is fired - what is the flow now?

6

The leecher is listening for any Gazu events. When it gets an event it cleans up any data needed (currently create a legal project name and project code).
It then dispatches the payload over to the processor.

The processor gets the event from the leecher, checks the payloud what kind of gazu-event happened and send it to the correct handler and it process the package accordingly.

In your example this happens:

Gazu (task:create) → Leecher (cleans up package) → Processor → Handler (Creates, Updates, Delete everything for Ayon or Kitsu using a ayon_kitsu_hub)

7

and the ayon_kitsu_hub?

8

ayon_kitsu_hub contains all processing code for both Kitsu and Ayon.
Each handler initializes a hub and then executes the needed functions for its type of handling.

A Hub to manage a Project in both AYON and Kitsu

    Provided a correct project name and code, we attempt to initialize both APIs
    and ensures that both platforms have the required elements to syncronize a
    project across them.
9

Thank you all for the work. I’m studying the PRs and i’d like to break it down a little:

Ahmed’s approach is super simple and with some tweaks, it could be actually implemented really quicky. I don’t think removing the frontend part is a good idea in favor of “just syncing all”, but we could have a setting directive to do exactly this - boolean switch “Auto-sync all”, which would enable the gazu listener in a different thread of the processor service and when an event arrive, just sync everything (including project creation if needed). Some event debouncing should be implemented, but during my tests, full sync rarely took more than a few seconds.

I really like Scotts work on tests and extending the server part with /remove endpoint. I think his PRs could be merged regardless the decision of the service architecture. The same goes for the processor counerpart (update/delete functions).

Honestly, I’m not a huge fan of a separate leecher service, which would just collect kitsu events, store them in the ayon database and then let the processor create another event. It feels like a huge overhead when compared to listening directly in the processor. In FTrack, leecher was introduced as a means of the listener’s high-availability (not to miss any of the event), but in case of service failure/restart/API unavailability, we could actually list events based on last timestamp and process them ex-post and reduce the number of event records in the ayon database by 2/3, which i believe is nice in the long run.

10

How much overhead is there for the shotgrid and ftrack addon with two services instead of one?

No matter what PR we go with in the end, three features I find very important to get working:

  • Use Kitsu IDs for everything so renaming objects works as expected
  • Task assignment
  • Task status change-sync
  • Pick what projects to sync

Good to have features would be Ayon to Kitsu sync.

Nice to have features would be to assign icons and statuses to task types, statuses, shots, assets etc

11

just to finish up where I was, I have added the kitsu events to the processor in PR #31.
Testing with a live Kitsu project and folders and tasks create, update, delete work for me.

Version is working well enough for us to work with in staging. We are going to test with this for a while and see where this repo develops.

12

On my forked repo, I was working on TODO functions to update and delete entities so it is still WIP.
since the leecher service is what you will be using, I will be happy to further contribute to it.

13

As ayon_api doesn’t have all the functionalities that is needed (eg change tast status etc) I have decided to abandon my PR is the code would end up being very non-consistent between using ayon_api, REST etc.

I’m moving over to @Jamscott 's code. I have made a PR on is fork where I have fixed a bunch of bugs, added some small features etc to make it work as expected.

When this is done we could merge his PR and after that anyone can create new PR’s to improve the code even more and add future features.

14

I’m done with my PR for @Jamscott 's PR: Features by Jacob by EmberLightVFX · Pull Request #1 · scottmcdonnell/ayon-kitsu · GitHub
These are the things I have added:

  • Merge Ynput develop branch and resolve conflicts
  • Format code using Black
  • Remove unused imports
  • Add correct types to functions and return values
  • Fix spelling on logging.warning
  • Clear duplicated task types
  • Allow auto-upload and server reload when building the addon
  • Set icons, state and short name for tasks types and statuses
  • Automatic short names like from Ayon server for task types
  • Added fully working Edit and Concept processing
  • Sync assigned users
  • Sync description
  • Calculate and sync frame length

Things I would love to add but in a future PR as it isn’t super important:

  • Asset type icons
    The current icon for all asset types is a robot as that’s the icon for Assets
  • Move everything in constants.py info to Ayon settings
    The stuff inside constants.py just covers the default values in Kitsu. Being able to set icon, state and short name in Ayon settings could be a good addition but not that important if you ask me.
15

After 3 more days the PR is now done done :smile:
I pushed it to Ayons repository: Fully working sync by EmberLightVFX · Pull Request #32 · ynput/ayon-kitsu · GitHub

1 Like
16

Hi all, I was away for the last week, looks like this has progressed well. Good work on this Danell. I am going to catch up on the changes today and test locally.

2 Likes
17

Hello,
I added shipped tag Since the PRs related to this post are merged.

They were presented in the following ayon-kitsu release.