Testing in Workflows

Testing can be challenging when not done properly, which is why we recommend a practical approach for testing blocks that you create. Since a block is not a standalone element in the ecosystem, testing might seem complex, but with the right methodology, it becomes manageable.

We suggest the following approach when adding a new block:

Unit Tests

Unit tests should cover:

• Parsing of the manifest, especially when aliases are in use.
• Utility functions within the block module. If written correctly, these functions should simply transform input data into output data, making them easy to test.
• The run(...) method should be tested unit-wise only if assembling the test is straightforward. Otherwise, we recommend focusing on integration tests for Workflow definitions that include the block.

Examples can be found in the inference repository unit tests.

Integration Tests

Integration tests should contain:

• Practical use cases where the block is used in collaboration with others.
• Assertions for results, particularly for model predictions. These assertions should be based on empirical verification, such as by visualizing and inspecting predictions to ensure they are accurate.
• When adopting models or inference techniques from external sources (e.g., open-source models), assertions should confirm that the results are consistent with what you would get outside the Workflows ecosystem, ensuring compatibility and correctness.

Examples can be found in the inference repository integration tests.