Houdini Caches Publish Optimisation

General
,
1

Hi,

Imagine a big size sequence cache; when Ayon publishes something, it actually gets copied from the staging directory to the publish directory.

A single job of publishing (which also has a copying function) with vast data is literally killing a lot of time either in the farm or locally.

Any thoughts on optimizing this workflow?

2

Depends on your needs, what would you like it to do instead?

Server side copy

On our end, it seems our server and the way the copy is handled is actually doing a great job at “server-side copy” and doing a quick copy on the server itself. Looking at our publish times, even on larger jobs - the minority of the time is spent on the file transfers itself. But I’ve seen some of your timings that did perform very badly in that area so it’s still an area to investigate.

How well your server can ‘copy’ and do server-side copy based on a connected client’s request may be up to your IT department to figure out and can heavily depend on the used filesystem or hardware setup.

Do not move files on publish

There is a way to configure anatomy to NOT move files on publish, and use the rendered path as the published file path. I believe it’s the {originalDirname} and {originalFilename} anatomy tokens. However, it’d then be up to you to ensure the path you’re rendering to is actually unique for each version.

Other approaches?

What would be another approach - when using linux a potential approach could be to create a symlink/hardlink to the original file and just unlink the original file (which could be faster than server side copy, again depending on your server setup). But has other potential file lock downsides, especially if Windows is involved.

3

Thanks for writing various options. we wanted to take published files into a different box so, compositors will not have a bottleneck with 3D files/caches/renders

Is there any documentation available for the server-side copy?
Is it an Ayon feature, or did you manage to build something in-house?

Wondering how server side copy is incorporated when we doing farm-render and publish in deadline!

4

It’s a feature of the network share. “Copy” file requests then would not go through the connected client but would fully occur on the server without needing to go over ethernet. It’s like saying: “Copy X to Y” to the server and instead of the connected client loading file X and copying the file to Y it doesn’t hit the client at all, it’s just the server doing it locally.

I believe It’s only supported on hardware and server if the source and destination are on the same partition on the server.

It would occur on any file copy on the server (if the lower level code, e.g. os module in python does the correct request to the server - which I believe should be the correct default in Py3+ at least.) I believe what speedcopy does also triggers it by default.

You may be able to identify if it works by checking that a copy like that involves close to no ethernet bandwidth for large files on the client machine ↔ server, because it’d all happen on the server.

See e.g. server side copy in Samba.

1 Like
5

Makes sense! Our cache and publish folder are in two different storage boxes, so we can’t take advantage of server copy!

Here is another idea! Can the publishing process be split into two different jobs in Deadline? So the process happens like below with dependencies:

  1. USD Render
  2. Copying job-—Multiple tasks (This happening inside publish rn with 1 task. I believe this would be beneficial for many studios. single machine copy bottle neck will get omitted)
  3. Publish in ayon database

Let me know your thoughts on this!

6

Copying is part of the integration process which happens on AYON publish job.

So, what about using a dedicated machine for AYON publish jobs?

Quoting from AYON publish deadline job's log 
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Integrating source files to destination ...
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1001.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1001.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1002.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1002.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1003.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1003.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1004.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1004.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1005.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1005.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1006.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1006.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1007.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1007.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1008.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1008.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1009.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1009.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings_h264burnin.mp4 -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review_h264.mp4
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Checking file ... C:\Users\MUSTAF~1\AppData\Local\Temp\pyblish_tmp_8420j48k\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1005_thumb.jpg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_thumbnail.jpg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1001.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1001.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1002.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1002.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1003.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1003.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1004.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1004.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1005.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1005.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1006.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1006.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1007.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1007.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1008.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1008.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1009.jpeg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review.1009.jpeg
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\ay_tmp_q7wifmge\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings_h264burnin.mp4 -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_review_h264.mp4
2025-04-11 16:10:17:  0: STDOUT: DEBUG:pyblish.IntegrateAsset:Copying file ... C:\Users\MUSTAF~1\AppData\Local\Temp\pyblish_tmp_8420j48k\ALA_decor_sheets_projector01_lookdev_v003.karmarendersettings.1005_thumb.jpg -> H:\AYON\projects\Animal_Logic_ALab\assets\decor_sheets_projector01\publish\render\renderLookdevMain_Beauty\v001\ALA_decor_sheets_projector01_renderLookdevMain_Beauty_v001_thumbnail.jpg
7

Yes, You are correct.

It usually takes lot of time with one machine when the cache size is huge for copying. The bottleneck will be single PC bandwidth.

The idea that I have is to split the copy process to multiple machines, In this case bottleneck will be server bandwidth, usually studio servers have lot of bandwidth!

8

I’m not entirely convinced it’s the correct solution.

  • It only helps for many frames (more splitting) and does not do much for fewer but much larger files.
  • By splitting into many small tasks you’re ignoring the fact that the actual starting of that task (and management through e.g. deadline) comes with some overhead. So each transfer may incur say a 1s overhead per frame.
  • It’d mean that many machines will suddenly start to try and access files randomly on one server, to then connect to another server.

You may be better off finding a way to increase the speed from one server to the other from the involved client. Either:

  • re-write the transfer logic to be a straight server->server copy (more complicated, but most likely best transfer speeds).
  • increase bandwidth from the clients (10gbe+ minimum isn’t odd nowadays for workstations, but 25gbe may be worth it if you’re losing so much purely on bandwidth issues to the server).

This increased bandwidth could e.g. be only on a select machine that’d purely do publish jobs. Or you could even run the publish jobs from a server itself (so that it’s actually dealing with local files mostly).

I really feel like if you’re dealing with server->server copies that are this massive and take minutes or hours you may be best off looking at strategies to improve that instead of throwing many clients at it that start individually downloading the file locally to then upload to the other server (which is what such a file copy essentially is?).

2 Likes