Some scenarios where FX caches and its iterations are stored, later get published. Here we can take advantage of deleting published files, but the original FX caches still remain the same, and they occupy so much space for each shot. Just curious if openpype can do something about that(maybe deleting unpublished files as well?)
Could you elaborate on the workflow you are using?
Ideally the unpublished files should be part of a publishing process similar to how renders work, so we can help with cleaning up the temporary caches.
I agree with Toke here, I’d imagine it’s better to delete the intermediate unpublished (cache) files and as long as you can keep all the published files. If you have a very strict cleaning policy then potentially you can also go into deleting some older published versions but since then you’re potentially deleting loaded content.
What’s the reason to want to do the opposite of that?
I’ll try to explain the workflow for water FX as an example here:
Load published assets
Run simulations (Bgeo.sc)
build mesh from simulation data
publish the mesh for lighting
In steps 2–3, it happens in the same scene file with the same artist, there is no reason to publish the huge set of cache (usually water takes 50+ GB), which we don’t pass to anyone. When we have a water sequence, it stores TBs of data.
As an example, I’ve mentioned WaterFx here, it’s similar to other FX tasks as well.
another way I’m thinking is to archieve(zip only the latest published workfiles, geometry’s and save them as a backup)
Like I described, this is usually the first data we throw out when “cleaning a project” instead of the published meshes for lighting.
A few reasons to prefer deleting that over the published lighting meshes:
There’s much more disk benefit, because it’s usually larger.
It’s basically a cache, so you can easily regenerate it. (Even though in some cases, recreation could take a long time!)
Usually no one except the artist working on that one FX/scene relies on the cache’s existence, it doesn’t influence others down the pipeline.
But reading your last comment I feel that’s also what you’re asking - how can we easily delete the caches and not the publishes?
As far as I know there’s currently no tooling that comes with OpenPype itself which deletes caches like these (I wouldn’t delete the workfiles because there’s pretty much no gain and it breaks your ability to potentially regenerate the data if you need it, but just delete the caches). Since how you work in the DCC with intermediate data like a cache like this is pretty much unmanaged by the pipeline (you can do whatever you want) it’s also hard to create something that reliably and safely deletes (only) the caches you actually want to delete.