Automate project folder structure creation on creating project on Ayon

Hi,

I currently have this. It works but not sure it is the desired way:


from ayon_server.addons import AddonLibrary

addon_library = AddonLibrary.getinstance()
core_addon = addon_library.get("core") settings = await core_addon.latest.get_project_settings(project_name

)

I’m looking into automatically create the project folder structure once a project on Ayon is created. I’m using the event system but found out the server can’t create folders on the network drive. Makes sense but for me it’s the first time to work on the server side.

So I have to go back to the drawing board anyway. Looking into webactions now.

Thanks for your help!

Cheers,

Ralph

Besides web actions, you can rely on services. you’d need to mount the project storage to the service.

Right…

I need to look into services and how they’re set up. I do have ash running. Still not 100% wrapped my head around it how they work and setup.

I think that would be a better option than web actions

Thanks!

Ralph

2 Likes

I guess as a start you can check existing services in different addons such as kitsu, ftrack, and etc.

Mostly, we have 2 folders in repos:

  1. “services”: where services implementation live. (basically, they are all python scripts + docker file for builder docker image where the code will run eventually)
  2. “service tools”: which includes helper tools for development like asking it to run a particular service.

I didn’t dig much in creating services before, but let me share some findings. Hope it helps wrapping your head around it.

Personally, as a start, I believe for a far very minimal service, one can start by writing their python implementation.

Here’s a very rough example which should outline the code flow.

Rough example
import ayon_api


def main_loop():
    # while service is not stopped
    ...

def main():
    try:
        ayon_api.init_service()
        connected = True
    except Exception:
        connected = False

    if not connected:
        ...

    # Register interrupt signal (it includes a clean up process)
    ...

    main_loop()


if __name__ == "__main__":
    main()

once this is setup, one can start exploring working with events:

from ayon_api import (
    get_event,
    get_events,
    update_event,
    dispatch_event,
    create_event,
    delete_event,
    enroll_event_job,
)

I don’t think there’s a way to subscribe to events to act upon them in Realtime. Alternatively, I think it’s the other where you implement a listener and just check the server events periodically for the event of interest.


Tip: In addon server side you can implement a web hook to be run when a specific event is created. (which sounds like subscribing to events)

Register Event Hook
class MyCooolAddon(BaseServerAddon):
  async def setup(self):
    register_event_hook("entity.folder.created", self.on_folder_created)

  async def on_folder_created(self, event):
    """put a comment on every new folder"""

    await create_activity(
        project_name=event.project, 
        activity_type="comment", 
        body=f"Hoooo, i have been created by f{event.user}!", 
        entity_references=[
         {"type": "origin", "entity_type": "folder", "entity_id": event.summary["folderId"}
        ]
        user_references=[{"type": "author", "name": event.user}]
     )

You can also stop listening via:

# Stop listen
event_callback.deregister()

Thanks Mustafa,

I’m currently struggling to see where to implement your suggested code and what it does precisely.

I’ve managed to setup a hook that triggers the launcher cli to run some actions.

I’ve tried using the webhooks for that. I’ve set up the ActionExecutor and when it runs it doesn’t throw an error. However it doesn’t properly triggers the action like the web action does when triggered from the Web UI. Biggest difference is that when triggering from the UI I get the [EVENT CREATE] action.launcher and [EVENT UPDATE] action.launcher feedback when I check the docker server logs.

However when I trigger my custom code it only triggers the [EVENT CREATE] action.launcher. So somewhere it doesn’t properly call the launcher from this code.

Is this the proper way to set it up or would you suggest a different way. This is my code block btw:


    async def on_project_created(self, 
        event: EventModel,
        ) -> ExecuteResponseModel:
        
        user = event.user
        project = event.project
        entityId = event.summary["entityId"]
        sender = event.sender
        sender_type = event.sender_type

        logger.info(f"sender:{sender}")
        logger.info(f"sender_type:{sender_type}")

        context = ActionContext(
            user=user,
            project_name=project,
            entity_type= "folder",
            entity_subtypes = ['Asset'],
            entity_ids=[entityId]
        )
        addon_name = self.name
        addon_version = self.version

        logger.debug(f"addon name:{addon_name}, version: {addon_version}")

        executor = ActionExecutor()
        executor.context = context
        executor.access_token = "1441fabe2a78afe364b2a5c8f2256bd2a6f722e6b1e71a4a76d3f0679c164e4a"
        executor.addon_name = self.name
        executor.addon_version = self.version
        executor.variant = "dev_bundle"
        executor.identifier = "event_handler.create_project_folders"
        executor.user = await UserEntity.load(user)
        executor.sender = sender
        executor.sender_type = sender_type
        executor.server_url = "http://server01:5000" 
                
        if executor.identifier == "event_handler.create_project":
            logger.debug("creating project folders")
            return await executor.get_launcher_response(
                args = [
                    "addon", "event_handler", "show-selected-path", "--project", project
                ]
            )

        raise ValueError(f"Unknown action: {executor.identifier}")

After more digging is it correct you always need an addon service to make this work?

You can’t manually trigger a webaction the way the UI triggers the webaction.

@ralphymeijer I’ve moved this discussion into a new post if you don’t mind. I just realized this is different from getting project settings.

Could we take a step back and revise the goal you want to achieve?

Which on of the following describes your goal better:

  • Create project structure on AYON server.
  • Create project structure on your file system.
  • Create additional folders in your root folder on your file system.
  • Create additional folders in the work area of tasks.

Hi Mustafa,

Thanks for moving the post to a new topic. It might become a bit confusing.

So here’s one of the few things I want to do with my custom Addon.

I would like to create a folder structure on disk the moment a project is created in Ayon.

Initially I wanted to do everything on the server side but I quickly found out the server doesn’t have a direct access to the file system which makes total sense.

My second try was using the Web Actions that already exist. There’s already a web hook implementation that creates a folder structure. So my ignorant mind thought I can just use that logic but then trigger that when the entity.project.created event gets triggered. That way the launcher cli would get triggered on the client side and create the folders.

I came pretty far by custom calling the ActionExecutor class. Initially I got everything working and the addon would call an [EVENT CREATE]. However it doesn’t trigger the launcher cli.

When I compare the server logs when calling the create folder structure action on the Web UI it also returns a [EVENT UPDATE]. One that I’m missing when calling it direct from server code. This is the code I’m calling when I get the entity.project.created event:


async def on_project_created(self, event: EventModel):

        user_name = event.user
        project = event.project
        entity_id = event.summary.get("entityId")
        sender = event.sender
        sender_type = event.sender_type
        
        logger.info("=" * 80)
        logger.info("MANUAL TRIGGER - Analyzing event creation")
        logger.info("=" * 80)
        
        # First, let's look at what a web UI event looks like
        await self.analyze_recent_web_ui_events()
        
        # Load user
        try:
            user = await UserEntity.load(user_name)
        except Exception:
            user = await UserEntity.load("admin")
        
        # Get service token
        service_token = "d1e3380af352955a2311551b4fb75d5e4060472759b3349c0413450d2753922"
        
        # Create context
        context = ActionContext(
            user=user,
            project_name=project,
            entity_type="folder",
            entity_subtypes=["Asset"],
            entity_ids=[entity_id]
        )
        
        # Create executor
        executor = ActionExecutor()
        executor.context = context
        executor.identifier = "event_handler.create_project"
        executor.addon_name = self.name
        executor.addon_version = self.version
        executor.variant = "dev_bundle-1"
        executor.sender = sender
        executor.user = user
        executor.sender_type = sender_type
        executor.access_token = service_token
        executor.server_url = "http://server01:5000" 
        
        logger.info("Executor configuration:")
        logger.info(f"  identifier: {executor.identifier}")
        logger.info(f"  addon_name: {executor.addon_name}")
        logger.info(f"  addon_version: {executor.addon_version}")
        logger.info(f"  variant: {executor.variant}")
        logger.info(f"  sender: {executor.sender}")
        logger.info(f"  sender_type: {executor.sender_type}")
        logger.info(f"  access_token: {executor.access_token}")
        
        # Call get_launcher_response
        logger.info("\nCalling get_launcher_response...")
        
        response = await executor.get_launcher_response(
            args=[
                "addon",
                self.name,
                "show-selected-path",
                "--project",
                project,
            ]
        )
        
        logger.info(f"Response: {response}")
        
        # Now compare with what web UI created
        if hasattr(response, 'event_id'):
            await self.compare_events(response.event_id)
        
        logger.info("=" * 80)
        
        return response

Now I’m wondering if this is even possible this way.It seems I need to implement a service in my addon but I was hoping to stay away from that.

So to sum up. Is this even possible to do directly from the server code without a service, and if so how would you suggest implementing. I thought it would be relatively straightforward since we have the events and we have the code already that creates the folder structure. I just need to bring them together.

Thanks in advance!

What better suits this description is: AYON Service with your studio storage mounted (so that it can access your file system and create folders).

Alternatively,

  • use a web action to create folders.
  • OR use a web action to run piece of code that keep looping e.g. every 10 minutes check the project hierarchy on server and create them on disk.
  • OR add a tray button to the launcher that create one time or keep looping.
  • OR add a custom tray code that on initialization maybe inside tray_init see example, where you can open a thread and keep looping. this is inspired by this (but it needs to to be limited, e.g. make it work with one user only, as you only need it once).

I’m not making recommendations. I’m just writing ideas that came to my mind

Let’s move to the next part, you want to create the work directories.
It’s worth mentioning that AYON provides a flexible settings for work directories: work templates + work template profiles + extra work folders. These configurations are very context aware (project, folder, task, application)
So typically, you should do something along these lines.

from ayon_core.pipeline.workfile import (
    get_workdir,
    create_workdir_extra_folders,
)
from ayon_core.pipeline.context_tools import (
    get_current_project_entity,
    get_current_folder_entity,
    get_current_task_entity,
    get_current_host_name
)

# Get Current host name
host_name = get_current_host_name()

# Get Entities
project_entity = get_current_project_entity()
folder_entity = get_current_folder_entity()
task_entity = get_current_task_entity()

#  we already have a function to retrieve resolved work dir
work_dir = get_workdir(
    project_entity, folder_entity, task_entity, host_name
)

if not os.path.exists(workdir):
    os.makedirs(workdir, exist_ok=True)

# Create extra folders
 create_workdir_extra_folders(
    workdir,
    host_name,
    task_entity["taskType"],
    task_entity["name"],
    project_entity["name"]
)

Also, you may need to care a lot about caching.

I’m aware you can simplify this code a ton by hardcoding many parts which will mostly for your studio specific configurations.

Tbh, everyone wants to stay aways from that. This is most likely because the lack of documentation about it and there are not examples that made for the purpose of learning how services are developed. And, production examples can be overwhelming if you are not familiar with them. you can take this post as an example How do I develop services? there isn’t much info there :smile:
Maybe this can be a topic for AYON workshops on Ynput Summit 26.

Thanks for your input Mustafa. I think I just have adjust my ideas a bit. My idea was to tie everything to the creation of a project but that seems to be a bit complicated.

I can create an additional web action but that kind off defeats the purpose since it requires an addition action from the user. It’s not bad it’s just something I rather not want.

An additional service with a mount to the drive is a good option, but eventually I see the server running in the cloud where it doesn’t necessarily has a connection to the main project drive.

The launcher option I will look into but preferably want to have the folders ready right when the project is created.

I was hoping you could trigger web actions from the server side and using the launcher cli seemed like a great solution for my idea. But after investigating the system a bit more it kinda makes sense this isn’t possible.

Thanks again for your suggestions!

Cheers,

Ralph

This is the same situation as Web publisher addon. where the documentation mentions that with cloud instances, you should run a new worker on your studio side so that it can access the storage.

https://help.ayon.app/en/help/articles/4834329-web-publisher#ottdleet46i