🤖 OpenPype to AYON core addon conversion

iLLiCiT · 5 February 2024 12:37

Introduction

AYON release v1 is out. Now is time to begin with conversion of OpenPype into AYON core addon, and split existing hosts and remaining modules into their addon repositories.

Current OpenPype repository remains dedicated to OpenPype, while a new repository ayon-core will emerge to house the core addon. That gives us big opportunity to start from scratch.

Plan

Let’s dive into the details of our action plan to smoothly transition OpenPype into the AYON-Core addon.

Initialization of core addon repository - first phase

Steps that will happen before any first PR as initialization of core addon repository. Should be done as fast as possible, and some of the steps will be prepared in advance.

Push develop from OpenPype into the ayon-core repository to keep commit history.
Remove all OpenPype related code from root of repository (igniter, pyproject.toml, tools, ect).
Move content of ./server_addons/openpype/ and ./server_addons/core/ into root to match addon structure, and move openpype content into ./client/.
Prepare create_package.py script.
Remove all code that is only OpenPype related (not needed in AYON). Like tools, mongo code, OP settings backend, all AYON_SERVER_ENABLED conditions, modules that are already separated (some of this may happen later in PRs, based on complexity).
Rename openpype to ayon_core and update all related imports.
Implement backward compatibility for importing ‘ayon_core’ as ‘openpype’.
- Introduce AddonsManager in ayon_core/addons/ as replacement for ModulesManager with backwards compatibility.

Remove legacy - second phase

Use AYON settings structure → remove settings conversions, and system settings.
Use AYON entity structure → remove entity conversions. e.g. use folder over asset doc.
Use AYON environment variables → replace AVALON_ and OPENPYPE_ environment variables.
Change structure of container and instance in each host (“ayon.instance”, “ayon.container”).
Change keys used during publishing → Use folderPath, folderEntity, addonsManager etc…
Stop using legacy_io.Session
Anatomy can use template categories

Separate addons - third phase

Split hosts and remaining modules into their addon repositories.

Plan conclusion

This transition will require time, testing, and verification, especially during the second phase.

Migration of existing PRs

Due to the extensive changes, migration of existing PRs is not possible. All PRs must be manually converted into new one.

Conflicts

Certain changes may affect existing addons, such as ftrack, kitsu, and shotgrid. Preparation for conflicts, especially related to entity structure and publishing keys, is crucial. Entity structure (using folder instead of asset doc) and keys used during publishing are conflicting every named addon.

Possible solution

We might define a constant USE_AYON_STRUCTURES in ayon_core that would hint addons if they should expect assetDoc or folderEntity. The addons could use it as forward compatibile value.

Day D

Day D determines when initialization of core addon begins. Any PR that should be in ayon core should not be merged in OpenPype repository, unless is CRITICAL for production. When initialization finishes all PRs that should be in OpenPype and ayon-core must be done to both repositories.

This work is starting on 5.2.2024 and we will be updating this topic as work progresses.

milan · 5 February 2024 12:41

This change will cause one very important side effect.

We’ll be closing PRs to OpenPype that are not targeting AYON and kindly ask anyone who is actively contributing to migrate them to the new ayon-core repo, or individual addons once they are up and running.

The aim is a massive PR cleanup of stale work and a nice fresh slate for the upcoming work in AYON. If your PR is closed, please don’t take it personally, we’ll always make a note what is the suggested next step with and hope to see majority of the work make it’s way to AYON.

iLLiCiT · 5 April 2024 13:00

Introduction

The new version of AYON core 0.3.0 brings many changes. Most of the changes are related to getting rid of OpenPype compatibility. That led to many changes in the code. Important changes include that OpenPype naming in classes and functions now have AYON naming. The same applies to environment variables. The most important change is that the AYON entity data structure is now used. That led to most of the changes and sometimes required removal of what we had. Sometimes it revealed a requirement for refactoring and thinking about changes that should have been done earlier.

Changes in names, as mentioned, OpenPype is now AYON, which is the first and important change. The entity name changed; “asset” became “folder,” “subset” became “product” (and “family” became “productType”).
Most of the classes and functions related to OpenPype were removed, as it would be almost impossible to support their backward compatibility.

In OpenPype, an asset is the source of most data used for scene setting and publishing, but with AYON, a task is a full entity that can be used for that from now on.

The most noticeable change is to use ayon_core imports instead of openpype. At the end is a list of all known changes in the code; if you encounter any that are not mentioned, please report them on the discord channel.

There are no system settings; there are only project settings, which have studio overrides and project overrides. Local settings are now site settings and are more limited.

How to convert addon to AYON 0.3.0

Guide to help you update your addon to AYON 0.3.0. This guide might not cover everything, and you might need to make extra changes. It also includes some notes for converting from OpenPype to AYON.

First steps

First, replace all openpype imports with ayon_core. Then, search the entire codebase for “openpype”. Replace any related classes and functions with AYON equivalents.

Changes in environment variables

OpenPype	AYON	Description
`OPENPYPE_LOG_LEVEL`	`AYON_LOG_LEVEL`	Sets the log level.
`OPENPYPE_DEBUG`	`AYON_DEBUG`	Turns on debug mode (simpler than `AYON_LOG_LEVEL`).
`OPENPYPE_HEADLESS_MODE`	`AYON_HEADLESS_MODE`	Starts process without a user interface.
`OPENPYPE_USE_STAGING`	`AYON_USE_STAGING`	Uses the staging bundle.
	`AYON_BUNDLE_NAME`	The name of the bundle to use.
	`AYON_USE_DEV`	Uses the development bundle.
	`AYON_SERVER_TIMEOUT`	The timeout for server requests.
	`AYON_SERVER_RETRIES`	How many times to retry server requests.
	`AYON_ADDONS_DIR`	Where to find addons.
	`AYON_DEPENDENCIES_DIR`	Where to find dependencies.
`OPENPYPE_EXECUTABLE`	`AYON_EXECUTABLE`	Where the AYON launcher is.
`OPENPYPE_VERSION`	`AYON_VERSION`	The version of the AYON launcher.
`OPENPYPE_ROOT`	`AYON_ROOT`	The root of the AYON launcher.
`AVALON_LABEL`	`AYON_MENU_LABEL`	The menu label in DCCs.
`AVALON_PROJECT`	`AYON_PROJECT_NAME`	The current project’s name.
`AVALON_ASSET`	`AYON_FOLDER_PATH`	The current folder path.
`AVALON_TASK`	`AYON_TASK_NAME`	The current task’s name.
`AVALON_WORKDIR`	`AYON_WORKDIR`	The current work directory.
`AVALON_APP`	`AYON_HOST_NAME`	The name of the host app.
`USE_AYON_SERVER`		Removed.
`OPENPYPE_REPOS_ROOT`		Removed.
`OPENPYPE_MONGO`		Removed.
`OPENPYPE_LOG_TO_SERVER`		Removed.
`OPENPYPE_DATABASE_NAME`		Removed.
`AVALON_DB`		Removed.

Changes in functions, classes and data

OpenPype	AYON	More details
`OpenPypeSecureRegistry`	`AYONSecureRegistry`
`OpenPypeSettingsRegistry`	`AYONSettingsRegistry`
`get_openpype_username`	`get_ayon_username`
`get_openpype_executable`	`get_ayon_launcher_args`
`run_openpype_process`	`run_ayon_launcher_process`
`get_current_asset_name`	`get_current_folder_path`
`get_current_project_asset`	`get_current_folder_entity`
	`get_current_task_entity`	Get current task entity.
	`get_workfile_template_key_from_context`	Get workfile template key from context.
`get_subset_name`	`get_product_name`	Signature of function changed.

Communication api

We removed all functions and classes in openpype.client. They were for talking to mongo and, for a while, for AYON server communication that worked with OpenPype. Since we took out the OpenPype compatibility, these functions aren’t needed anymore.
Now, use ayon_api to talk with the AYON server.

Settings

Function get_system_settings was removed because AYON does not have concept of system settings.

Addons

The content of openpype.modules has been moved to ayon_core.addon; both are working now, but ayon_core.modules is marked as deprecated.
The addon base class OpenPypeModule is now AYONAddon, and ModulesManager has been renamed to AddonsManager. Method and attribute names on the manager and addons have also changed. All instances of the word module have been replaced with addon.
The interface ITrayModule has been renamed to ITrayAddon, and ILaunchHookPaths has been removed.

Applications

The applications logic from openpype.lib is now separated into the addon ayon_applications. This addon will be moved from ayon-core to a separate repository.

This also includes the extractenvironments CLI command. To run the command, use ./ayon addon applications extractenvironments.

Project anatomy

Project anatomy templates now include categories like work, publish, hero, delivery, staging, and others. We updated the code for this change. Before, in OpenPype compatible mode, we used a prefix like "{category name}_{template name}". Now, it’s a nested structure.

To make things easier, we added a get_template_item method to Anatomy.

Module `legacy_io`

The legacy_io module has been removed. It was once needed for OpenPype compatibility but isn’t used in AYON now. It helped define the current context, such as project, asset, and task. Now, functions in ayon_core.pipeline.context_tools serve as its replacement.

Launcher action

Action plugins no longer take “session” as an argument for is_compatible or process. A new class, LauncherActionSelection, replaces the session. This object is backwards compatible but logs a deprecation warning.

Load logic

Methods update, remove, and switch now use “context” instead of “representation,” aligning with the load method. In context, "asset" and "subset" have been renamed to "folder" and "product".

The filtering attribute families changed to product_types.

Create logic

Most changes relate to AYON naming and the new entity structure. The asset is now called a folder, subset is product, and family is product type.

Note that the signature for create plugin classes get_dynamic_data and get_product_name has changed. They now expect a task entity instead of a task name. The methods __init__ and apply_settings no longer expect system settings. Also, the attribute family has been renamed to product_type.

Instances now store "productName", "productType", and "folderPath" instead of "subset", "family", and "asset".

Publish logic

Publish is directly connected to create logic, so most of the changes are related to AYON naming and new entity structure too.

OpenPype	AYON	More details
`"asset"`	`"folderPath"`
`"subset"`	`"productName"`
	`"productType"`	Source value for `"family"`.
`"assetEntity"`	“folderEntity”
	`"taskEntity"`
`"openPypeModules"`	`"ayonAddons"`
	`"ayonAddonsManager"`	Added the manager object in data.
`"project_settings"`	`"projectSettings"`	Both are available now.
`"system_settings"`		Removed.

Editorial related data stored under "hierarchyContext" changed the data structure. All folders now have "entity_type" set to "folder" and "folder_type" with the type of folder. Key "childs" was renamed to "children" and "custom_attributes" was renamed to "attributes".

Hero version

Hero version is now a complete entity with the same data as the “standard” version, making it independent of the “standard” version’s data. It’s identified by a negative version number. The server stores only one version in the database.

Templates data

Templates data structure changed. For now it should support all keys available in OpenPype but new were added.

OpenPype	AYON
`"{asset}"`	`"{folder[name]}"`
`"{subset}"`	`"{product[name]}"`
`"{family}"`	`"{product[type]}"`

Thumbnails

Thumbnails are stored on server instead of filesystem. The server provide the thumbnail url by thumbnail id.

Entity structure changes

OpenPype documents and AYON entities are similar but not the same. For developers, I recommend to get entities from AYON server using ayon-python-api to investagate the structure.

The main changes are that "_id" has been changed to "id". The "parent" field is now called "parentId", "folderId", "taskId", "productId", or "versionId", depending on the child entity. Folders have a new field called "folderType", tasks have "taskType", and products have "productType". Also, the entity data no longer includes the "type" field.

All entities can have attributes and data ("attrib" and "data"). Attributes are not project-specific. Attributes can inherit values across the hierarchy, and API calls will resolve the hierarchy values for you. We have some built-in attributes, which are “the most used,” but you can add more. You can also store any data in the "data" field, but they do not inherit values. The value of "data" is stored as a JSON string, and it is not possible to update a single key. You always have to update the full "data" value.

Tools

Most of the tools and tool utils were re-written to support AYON naming and entity structure.

Summary

We haven’t listed all the changes that have happened or will happen. We’ll update this summary as often as we can. If you have questions or need help with conversion, please ask in the Discord channel.

TODOs

Now we have to convert addons that are not in ayon-core.

🤖 OpenPype to AYON core addon conversion

Introduction

Plan

Initialization of core addon repository - first phase

Remove legacy - second phase

Separate addons - third phase

Plan conclusion

Migration of existing PRs

Conflicts

Possible solution

Day D

Introduction

How to convert addon to AYON 0.3.0

First steps

Changes in environment variables

Changes in functions, classes and data

Communication api

Settings

Addons

Applications

Project anatomy

Module legacy_io

Launcher action

Load logic

Create logic

Publish logic

Hero version

Templates data

Thumbnails

Entity structure changes

Tools

Summary

TODOs

Module `legacy_io`