Backup AYON database/settings

[robert]

Another question.

How do we backup or even transfer the Ayon database/settings.
i couldn’t find anything in the docs

We’re setting up our server soon with Ayon etc.
But i would like to have an autobackup of the database to another location, so in worst case scenario the server would die, we can load it in on another one.

(preffered we’ve a redundant server with autoswitching etc etc etc, but this is something for next year)

[mustafa_jafar]

@robert

How do we backup or even transfer the Ayon database/settings.

As far as I know, it’s possible to dump the entire database using pg_dump utility in the postgres container.
then for restoring from a backup, you just start the postgres container first on a fresh instance (without server) and use the same method as for restoring projects:

docker compose exec -T postgres psql -U ayon ayon < yourbackupfile.sql

[mustafa_jafar]

Quoted from AYON monthly beginner’s topic - December‘23 | Back up Ayon DB

@mustafa_jafar

Back up Ayon DB

This is some how not related to ayon itself but to postgres container that runs in Ayon stack.

• go to your ayon-docker repo clone.
• run
  docker compose exec postgres pg_dump -U ayon > backup.sql
  to create database dump
• copy storage/ , addons/ , docker-compose.yml and backup.sql to the new machine (and other supporting files)
• run
  docker compose up postgres
  on the new machine to start only the database (if you run the entire stack, it will be populated with defaults, we want to keep it empty for now)
• run
  docker compose exec -T postgres psql -U ayon ayon < backup.sql
  to restore the backup
• start everything using
  docker compose up

Maybe @martin.wacker can add more insights.

[martin.wacker]

Yep. I use this daily. But I cannot guarantee the redirect will work properly on Windows - they tend to misbehave and try to buffer everything which may be a problem in case of huge dumps - if that ever happens to you, please let me know and we’ll find a workaround

[Des]

• run
  docker compose exec postgres pg_dump -U ayon > backup.sql

This errors out with:

no configuration file provided: not found
This saves an emtpy file in the directory where you run it.

You should update the guide how to properly do this:
Navigate to your ayon docker folder on disk, then run this command.

Users like who didint drink coffe in the morning could get easily a gray hair

Also mentions to if you have your own postgres you dont need docker compose part.

[mustafa_jafar]

@BigRoy has cool GH notes about Backup the Database. you might would like to check it.

https://gist.github.com/BigRoy/bb3613e49ec421acb9d09613d984d98c#dump-entire-database

[mustafa_jafar]

Tip of the day, These commands should be run in ayon-docker path and make sure you have enough permissions, maybe do it with sudo .

[jm22dogs]

Hello fellow Ayoneers,

I hope it’s OK I bumped this thread back up from its dormant state. I just transferred the DB from one Ayon instance into another and it seemed to be working fine, thank you.
I would like to know what happens to bundles and bundle settings, are those copied over? (I didn’t get the bundles when restoring the DB)
The addons and dependency packages are there because I copied both the addons and the storage folder, so that’s good.

Thank you!

[mustafa_jafar]

Have you checked the data inside the generated sql file?

[jm22dogs]

I did, and the bundles were there in the sql file, problem was that the backup only copied over part of it to the ayon db, some sql statements threw errors. Anyway I run it again and now the bundles are there… thank you!

[jberkheimer]

So has anyone successfully been able to migrate data from an older server to a new server? I’ve been attempting to write a script that will automate the migration of the addon settings from my 1.5.2+202410141409 version server into my 1.11.2+202509052305 version server. But there are a lot of schema differences and it has been very difficult figuring out how to account for all the differences. Even trying to do it manually is a pain in the ass.

So how are you guys handling this?

[mustafa_jafar]

@jberkheimer

I’ve been attempting to write a script that will automate the migration of the addon settings from my 1.5.2+202410141409 version server into my 1.11.2+202509052305 version server. But there are a lot of schema differences and it has been very difficult figuring out how to account for all the differences. Even trying to do it manually is a pain in the ass.

So how are you guys handling this?

I’d say when working with settings, you should depend on the API.
There are further info here
https://help.ayon.app/en/help/articles/4089565-ayon-server-provisioning#zcik4k15ly5

[jberkheimer]

Yeah, that’s what I’ve been using. I’ve been working on a tool that will help me migrate the data between servers. But the problem I’m running into is the settings schemas have changed a bit between the server/addon versions.

For example, in the Maya addon (0.2.14) there is this setting:

    "maya_dirmap": {
      "title": "Maya dirmap Settings",
      "disabled": false,
      "allOf": [
        {
          "$ref": "#/definitions/MayaDirmapModel"
        }
      ],
      "scope": [
        "project",
        "studio"
      ]
    },

But in Maya (0.4.13) that same section is this:

    "dirmap": {
      "title": "Maya dirmap Settings",
      "disabled": false,
      "allOf": [
        {
          "$ref": "#/definitions/DirmapModel"
        }
      ],
      "scope": [
        "project",
        "studio"
      ]
    },

It’s a slight difference. But this is a hard thing to account for in your code because these name changes are arbitrary. After thinking on it over the weekend, I’m resigned to the fact that I need to do this by hand to at least get the settings updated to the current formatting.

[mustafa_jafar]

@jberkheimer

It’s a slight difference. But this is a hard thing to account for in your code because these name changes are arbitrary. After thinking on it over the weekend, I’m resigned to the fact that I need to do this by hand to at least get the settings updated to the current formatting.

Yeah about that, addon settings may change over time and this is handeled per addon where each addon has a conversion script that tells ayon how to re-map settings e.g. check Maya’s conversion.py.

The only catch is these converstions work when applying settings from older versions to newer versions.

Therefore, you only need to backup/restore the values as they are returned form the api.

[jberkheimer]

OK, interesting. Thanks for pointing that out.