This proposal aims to enhance the existing host integration testing for Ayon (formerly known as OpenPype). The primary goal is to enable Studios and the Client Success team to conduct tests independently before considering a release version suitable for production deployment.
Integration Testing Triggers
To initiate the new integration testing, the following triggers are proposed:
- Ayon Client App (Staging Mode): Starting the Ayon client app in “staging” mode will display a “Testing” section within the Ayon Tray menu. This section will offer a hierarchy of options:
- Running host-specific tests in headless mode.
- Executing all tests or a specific testing unit under the chosen host.
- Conducting a test using the code’s default settings.
- Performing a test with Studio default overrides.
- Running a test with Project overrides.
- App Launcher (Staging Mode): Using the App Launcher in “staging” mode allows users to open a host in a project-specific context. The host launcher will offer testing options, enabling the creation of a test project with a cloned hierarchy and connected project settings overrides.
- CI Headless Mode: The existing headless mode used in the TeamCity framework.
+---------------------+ +------------------+ +-----------------+
| Ayon Client App | | App Launcher | | CI Headless Mode |
| (Staging Mode) | | (Staging Mode) | | (TeamCity) |
+----------+----------+ +---------+--------+ +--------+--------+
| | |
+------------------+-------+------------------+-------+
|
v
+---------+--------+
| Integration Tests |
+-------------------+
Types of Tests
There are two ways to perform tests:
-
Deterministic Testing (Code Released Studio Setting Defaults): Deterministic testing uses predefined metrics to compare expected outputs. These tests are mostly used in CI testing and headless (non-interactive) testing initiated from the Ayon Tray submenu. However, these tests are not applicable when using studio project settings overrides. In such cases, non-deterministic testing is employed.
- Advantages: Measurable and suitable for CI since results are comparable
- Disadvantages: Limited testing scope due to the use of predefined database dumps
-
Non-Deterministic Testing (Studio Project Settings Override): Non-deterministic testing is a simplified method for assessing outputs. It is only used when a user initiates tests with selected studio project settings overrides. These tests confirm that the code does not disrupt or hinder production, making them a valuable tool for the Client Success team. Non-deterministic user tests allow users to examine output data and visually check the results (e.g., comparing output with or without Burn-ins).
- Advantages: Testable in production, can better reveal edge cases
- Disadvantages: Not measurable since outputs cannot be compared with predefined expected data
Testing with Studio Default Settings or Project Overrides
To run tests with Studio default settings or project overrides, a new testing project should be created in MongoDB or PostgreSQL. The selected overrides should then be copied into the testing database and connected to the testing project, ensuring that no studio projects are altered during testing.
Current Datasets for Deterministic Testing
At present, each integration test sources its dataset package from cloud storage (specifically, Google Drive). These packages must be prepared for each testing unit and may include an expected folder containing the expected folder hierarchy, file naming conventions, formats, and properties (e.g., bit depth, bit size, hash, layers, parts, etc.). Due to the large size of some expected files, they are packaged as .zip files and stored separately from the main repository.
For a detailed description of the .zip package contents, visit Ayon Integration Testing.
Improving the Workflow
To enhance the deterministic testing workflow, the expected folder and physical data do not need to be included in the dataset. Instead, a tool can be developed to convert this data into a JSON format. This modification would significantly reduce the dataset size, enabling their inclusion in addons and eliminating the need for separate distribution.
Additional Recommendations
- Test Coverage and Documentation: Ensure comprehensive test coverage for all host integrations, including edge cases. Maintain clear and concise documentation for all tests, detailing their purpose, functionality, and any required setup or prerequisites.
- Automated Test Execution and Reporting: Implement automated test execution to streamline the testing process and generate reports that highlight any issues or discrepancies. This can help the Client Success