Shotgrid Integration addon

[Minkiu]

Hello all,

In this thread we can discuss everything regarding the Shotgrid Integration in Ayon!

I’ve just recently updated the README in the Addon repository with the latest instructions to get it up and running.

It currently supports importing projects from Shotgrid, syncronizing projects already imported (Shotgrid → Ayon) and listening to Shotgrid event and create, update or delete entities.

Have a read through the README and don’t hesitate to ask any question or flag any issue while testing it.

It will create the following fields in your Shotgrid instance:
For the project Entity it will create the fields: sg_ayon_id, sg_ayon_sync_status, sg_ayon_project_code, sg_ayon_auto_sync and sg_ayon_server_url

Then for Episode, Sequence, Shot, Asset, Version and Task it will create the fields: sg_ayon_id and sg_ayon_sync_status.

Note that we are changing the ayon-python-api to read the environment variable from AYON_TOKEN to AYON_API_KEY so depending on when you are trying, the step where you create the .env file might need to sill use the AYON_TOKEN.

Cheers!

[matteo]

Hi,

I see the ShotGrid integration is very basic, looks like it’s just project (and contents) orchestration on ShotGrid which then mirrors into AYON.

Any plans to reinforce this?
SG is far more widely used than Ftrack and not having simple features like media uploading back to SG for review or right client menu actions (for example, creating a delivery) is a huge deal breaker.

Overall you’ve built an incredibly compelling pipeline here that I and many others are exceedingly interested in and would happily pay support for but it feels like even Kitsu has more integration.

[tokestuartjepsen]

@matteo

Any plans to reinforce this?
SG is far more widely used than Ftrack and not having simple features like media uploading back to SG for review or right client menu actions (for example, creating a delivery) is a huge deal breaker.

@Minkiu can correct me if I’m wrong, but looks like media uploading is handled when publishing; ayon-shotgrid/client/ayon_shotgrid/plugins/publish/integrate_shotgrid_version.py at 9ab760ffc0f94e4d97f47d0fea0af4daa3122bb7 · ynput/ayon-shotgrid · GitHub

[milan]

That’s correct. We support also creating reviewable versions on shotgrid.

Regarding other improvements. It’s a fairly simple approach we take. If studios use part of AYON and report back what they would like improved, we’ll usually put it on a backlog. However we get to it almost exclusively if a studio (or more) puts some funding into the requested feature. There are 100s of feature requests and issues by now, so we work those that are prioritized by studios paying for some kind of support. They deserve to have the biggest influence on the product.

SG addon is not yet used by any of the existing clients (in the on-boarding process though), hence not many specific funded requests for it yet.

[matteo]

Hi @milan,

Thanks for getting back to me and explaining your process, the approach totally makes sense.
No doubt we’ll be in touch in the coming weeks

[matteo]

Hey @milan,

Just coming back to this, are there any plans (maybe the add-on already does this) to use the ShotGrid add-on for AYON as a handler for ShotGrid event triggers?
Usually I’d set up a flask server for this type of thing, just wanting to look ahead if I should do that or wait to implement in AYON.

[Minkiu]

Ayon Shotgrid Version 0.2.0

Hello all, we’ve been hard at work polishing some rough edges, bug fixes and adding the option to sync/push from AYON to Shotgrid!

New UI

On the surface you’ll find that the “Shotgrid Sync” page has been modified, it now looks like this:

Under the hood we now query the Shotgrid API and create the AYON events to sync via Javascript directly in the frontend, instead of having to go through a custom addon endpoints.

AyonShotgridHub for services

We created a new class called AyonShotgridHub which allows for easy interaction between the two platforms, bootstrapping common actions to avoid repetition, and it’s where logic like creating, synchronizing projects lives, and it’s what we now use for the services.
This involved a fair bit of re-factoring, so I apologise to anyone who was becoming familiar with the processor code base… a lot has been reuse, and mostly re-shuffled.

Syncronize from AYON to Shotgrid

Mentioned in the first paragraph, we can now synconize a project from AYON into Shotgrid, this requires the processor service, and the logic is found in the AyonShotgridHub.

The transmitter service

It’s a new service that will keep an eye on AYON events, and when we create, modify or delete an entity, we attempt to transmit the change to Shotgrid, so the opposite of the leecher.

You can see the changes here: Comparing 0.1.0...0.2.0 · ynput/ayon-shotgrid · GitHub

Next Steps

Being a .0 version chances are there are unforeseen bugs, so your feedback is greatly appreciated, either here or on the repository issues, as well as any contribution, feel free to ping me on Discord and I’ll be more than happy to show you around.

There are know issues or things that are not really as they should, a non complete list would be:

• Update the README to reflect these changes and provide a better overview of how things interact in the Addon.
• Use the secrets back end to handle the Shotgrid API Key.
• Synchronize more attributes, currently is really basic.
• Structure/entity mapping between the two platform is fickle at the moment.
• We should be using “Pipeline Steps” instead of the task names.
• When creating a Shotgrid project (from an AYON one) modify the “Tracking Settings” and “Navigation” so the project looks good (this is probably subjective).
• Something you want?

Cheers!

[Minkiu]

Hey @matteo ,

I believe what you describe is what the leecher service does, it looks up at the Shotgrid Events table and feeds them to AYON, which then we process via the processor service.

Cheers,
Oscar.

[Veena]

Hi there,

I have been testing out the Ayon-Shotgrid integration following the steps given in the README and I’ve had a couple questions come up. Currently, I am trying to import one of our Shotgrid projects into Ayon using the synchronize shotgrid → ayon button, and the event spawns, but it does not complete successfully.
Looking at the Event Viewer, I see some log.errors, but the messages are not too descriptive as to what’s going wrong. In general, I was wondering what is the advice for approaching and debugging service errors that we may encounter in the integration?

thanks!

[Veena]

Possibly related to the above question, after adding the shotgrid-ayon addon to the server and configuring it using the shotgrid script key, the authentication in the desktop launcher does not seem to be working: it’s giving an error of “cannot authenticate shotgrid user” although the same username and password credentials work in the browser.

[Minkiu]

Hi @Veena ;

Do you have the shotgrid-processor service running as well?
That’s where the actual errors would be displayed (in the logs of the service); if you provide them I could point you out where is it failing.

If you are running it via ash, you can login into the machine where ash is running and do (assuming Unix):
docker ps -a | grep processor
docker logs --follow <container_id_from_previous_command>

From experience, the Shotgrid API will fail to create fields on entities which if you ran it again it will succeed; another potential gotcha might be the “Shotgrid Project > Project Actions > Tracking Settings”, do you mind sharing what entities you have enabled/disabled?

We attempt to create the fields on the below (the keys of the dictionary) (checking if they are enabled but might be what’s tripping it…)

AYON_SHOTGRID_ENTITY_TYPE_MAP = {
    "Project": "project",
    "Episode": "folder",
    "Sequence": "folder",
    "Scene": "folder",
    "Shot": "folder",
    "Asset": "folder",
    "Version": "version",
    "Task": "task",
}

Let me know how it goes!

[Veena]

Hi @Minkiu thanks for your reply!
I checked the logs and it’s true that we did have one warning about failed field creation on an entity type we weren’t using (Scene). Even with scenes enabled though, we’re hitting a connection error with the ayon server,as you can see in this log snippet:

[jackiezxt]

I successfully connect ayon to shotgrid now, here’s need something to do

1. Create a user on AYON web ui, any name is ok, but the Access Level must be service, and need to remember, the password will be called AYON_API_KEY below.

2. Setup the ASH docker, run the server. I’m not familiar with linux and docker command, but you can use chatGPT to generate the command code.

   docker pull ynput/ayon-ash:latest
     
   docker run --rm -ti \
     -v /var/run/docker.sock:/var/run/docker.sock \
     --hostname worker01 \
     --env AYON_API_KEY=verysecureapikey \
     --env AYON_SERVER_URL="http://<your_url>:5000" \
     ynput/ayon-ash
   

3. Since the github repo says there’s need Ayon Auto Sync field on shotgrid project entity, but you don’t need to create the field by yourself, it will be automatically generated. But if you created by hand, that doesn’t matter
   

   

4. Set the script api key in shotgrid, and remeber the name, fill the name into Secret page on Ayon, like the pic below, the name is ayon, but I filled another name “shotgrid_api_ayon”, it will cause authenticate errors. And also need to set up the user password and personal token key, that would bring to successfully logging state in AYON desktop app.
   

   
   
   
   
   
   
   

5. After test, the better is create a project on shotgrid first, and you have to fill the code in “code” field in project entity. If it is empty, you can’t sync the shotgrid project to AYON site. Also, you need to create the assets and shots, sequences, tasks on shotgrid first, and the folder structure will be automatically created in AYON. Otherwise, there’s will some weird things
   
   

[jackiezxt]

But I don’t know why this error happens

1. I launched maya 2024 in AYON desktop app, it has error like this
   

   

2. I publish the file, it happens like this
   

   

[BigRoy]

I know that the first issue is a bug fixes some time ago. So please make sure to update to include it on your end.

The second, I don’t know about myself. Will leave that to someone else.

[Minkiu]

Ayon Shotgrid Version 0.3.2

Hello all, a new release is out, with a plethora of fixes and improvements, most noteworthy ones are:

• Bump the ayon-python-api version to be 1.0.0
• Fix bug and implement safeguards for the transmitter: it was previously reacting to all and any event produced by the service AYON user, there’s a new setting where you can define which user is the one creating stuff via the ayon-shotgrid so that we can ignore those events.
• Added a setting to control the usage of Shotgrid’s Local Storage.
• Reduced the overall noise of logging, so that when you are looking at the service logs is mostly relevant information.
• Better client side code, handling errors better, and removing the obsolete “login” dialog, we will now use the API key defined in the Secrets page.
• Bugfixes in the ayon_shotgrid_hub

The settings page

And probably more that I am missing now.

Before trying it

TO note that the previous iteration of transmitter was misbehaving, and we found out that there might be new entities in your project that have empty names in SG, do look through the entities created by your defined Script and make sure any odd looking ones are removed before trying to sync again!

I would like to thank @fabiaserra for the immense help to get this version out and nailing a lot of fixes while providing great feedback.

Please get it while it’s hot and let us know if you have any issues.
(It might still not be available from the marketplace, so you can build it from the code with python create_package.py if you want to try it already).

[Minkiu]

The new version has had the local storage logic slightly modified, so probably worth a try to see if it’s still happening.

[Minkiu]

I though it wasn’t necessary, but it might be, that the secret name in AYON has to match the Script Name in SG, i.e.:

# Shotgrid > Scripts
* "ayon_connector"

# AYON > Secret
* "ayon_connector": <api_key>"

I believe that’s why it fails, try that and let me know, otherwise you have everything else correct.

Also, I’d say to just start with the transmitter and then once everything is in place try the transmitter or leecher (or both).

Cheers,
Oscar.

[Aivo]

Great work Oscar. Thank you for the update.
A small question concerning the sync from SG. Is it possible to Sync the Sequences and Shots into a specific AYON Folder? Or is something like this planed?

Thank you in advance and cheers

Aivo

[Minkiu]

Hi @Aivo thanks for the kind words and welcome to the community!

You would like to define an arbitrary folder in AYON where to put only the Sequences and Shots?
I guess it would be possible, but I am afraid it would add complexity to the already (fairly) complex logic, since we can’t do an exact one to one map of a SG project; something on the lines of a setting that allows you to define hierachy?

At the moment, the sync tries it’s best to replicate the Shotgrid structure in AYON, with two caveats (that I reacall from the top of my head):

• In AYON we do not allow Tasks at Project level, so I’ve been thinking that I should probably implement a top level folder “project tasks” or something on the lines of that.
• We create folders for Assets categories, since these are not entities per-se (or at least not queryable) in Shotgrid.