How Workflow Execution Works

Workflow execution is a complex subject, but you don't need to understand every detail to get started effectively. Grasping some basic concepts can significantly speed up your learning process with the Workflows ecosystem. This document provides a clear and straightforward overview, designed to help you quickly understand the fundamentals and build more powerful applications.

For those interested in a deeper technical understanding, explore the compiler and execution engine documentation.

Compilation

Workflow execution begins with compiling the Workflow definition. A Workflow definition is a JSON document that outlines inputs, steps, outputs, and connections between elements. To turn this document into an executable format, it must be compiled.

From the Execution Engine's perspective, this process involves creating a computation graph and checking its integrity and correctness. This verification step is crucial because it helps identify and alert you to errors early on, making it easier and faster to debug issues.

Once the compilation is complete, it means your Workflow is ready to run. This confirms that:

• Your Workflow is compatible with the version of the Execution Engine in your environment.
• All blocks in your Workflow were successfully loaded and initialized.
• The connections between blocks are valid.
• The input data you provided for the Workflow has been validated.

Data in Workflow Execution

When you run a Workflow, you provide input data each time. Just like a function in programming that can handle different input values, a Workflow can process different pieces of data each time you run it.

You provide input data substituting inputs' placeholders defined in the Workflow. These placeholders are referenced by steps of your Workflow using selectors. When a step runs, the actual piece of data you provided at that moment is used to make the computation. Its outputs can be later used by other steps, continuing this process until the Workflow completes and all outputs are generated.

What is the Data?

Input data in a Workflow can be divided into two types:

• Batch-Oriented Data: Main data to be processed, which you expect to derive results from (for instance: making inference with your model).
• Scalars: Single values used for specific settings or configurations.

Workflow definitions hold inputs of the two categories:

• Scalar inputs — like WorkflowParameter
• Batch inputs — like WorkflowImage, WorkflowVideoMetadata, or WorkflowBatchInput

Steps Interactions with Data

If we asked you about the nature of step outputs in these scenarios:

• A: The step receives only scalar parameters as input.
• B: The step receives batch-oriented data as input.
• C: The step receives both scalar parameters and batch-oriented data as input.

You would likely say:

• In option A, the output will be non-batch.
• In options B and C, the output will be a batch. In option C, the non-batch-oriented parameters will be broadcast to match the batch size of the data.

And you'd be correct.

Dimensionality Level {#dimensionality-level}

Let's say you want to create a Workflow with these steps:

1. Detect objects in a batch of input images.
2. Crop each detected object from the images.
3. Classify each cropped object with a second model to add detailed labels.

Here is what happens with the data in the cropping step:

1. You start with a batch of images, let's say you have n images.
2. The object detection model finds a different number of objects in each image.
3. The cropping step then creates a new image for each detected object, resulting in a new batch of images for each original image.

So, you end up with a nested list of images. The second model (classifier) will process these nested batches of cropped images.

The Execution Engine manages the nesting of data virtually, so blocks always receive data in a flattened, non-nested format. This makes it easier to apply the same block, like an object detection model or classifier, regardless of how deeply nested your data is. But there is a price — the notion of dimensionality level which dictates which steps may be connected.

dimensionality level refers to the level of nesting of batch. Batch-oriented Workflow inputs have dimensionality level 1, crops have dimensionality level 2, and so on.

What matters from the perspective of plugging inputs to a specific step is:

• The difference in dimensionality level across step inputs.
• The impact of the step on dimensionality level of output (a step may decrease, keep the same, or increase dimensionality).

The majority of blocks are designed to work with inputs at the same dimensionality level, not changing dimensionality of its outputs, with some being exceptions to that rule.

Conditional Execution

Some Workflows blocks can impact execution flow — steps made out of those blocks will be specified a bunch of step selectors, dictating possible next steps to be decided for each element of batch. Key points:

• Once a data element is discarded from a batch by conditional execution, it will be hidden from all affected steps down the processing path and denoted in outputs as None.
• Multiple flow-control steps may affect a single next step; a union of conditional execution masks will be created and dynamically applied.
• A step may not be executed if there are no inputs left after conditional execution logic evaluation.
• There are special blocks capable of merging alternative execution branches, such as First Non Empty Or Default.
• Conditional execution usually impacts Workflow outputs — all values that are affected by branching are in fact optional.

Output Construction

The most important thing to understand is that a Workflow's output is aligned with its input regarding batch elements order. This means the output will always be a list of dictionaries, with each dictionary corresponding to an item in the input batch.

input_images = [...]
workflow_results = execution_engine.run(
    runtime_parameters={"images": input_images}
)

for image, result in zip(input_images, workflow_results):
    pass

Elements at dimensionality level 1 will be distributed evenly. Elements at higher dimensionality levels will be embedded into lists of objects of types specific to the step output being referred.

Some outputs require serialization when the Workflows Execution Engine runs behind HTTP API:

• Images are serialized into base64.
• Numpy arrays are serialized into lists.
• sv.Detections are serialized into inference format which can be decoded using sv.Detections.from_inference(...).