workflow.proto

flyteidl.core.IfBlock

[flyteidl.core.IfBlock proto]

Defines a condition and the execution unit that should be executed if the condition is satisfied.

{
  "condition": "{...}",
  "then_node": "{...}"
}
condition

(flyteidl.core.BooleanExpression)

then_node

(flyteidl.core.Node)

flyteidl.core.IfElseBlock

[flyteidl.core.IfElseBlock proto]

Defines a series of if/else blocks. The first branch whose condition evaluates to true is the one to execute. If no conditions were satisfied, the else_node or the error will execute.

{
  "case": "{...}",
  "other": [],
  "else_node": "{...}",
  "error": "{...}"
}
case

(flyteidl.core.IfBlock) required. First condition to evaluate.

other

(flyteidl.core.IfBlock) optional. Additional branches to evaluate.

else_node

(flyteidl.core.Node) The node to execute in case none of the branches were taken.

required.

Only one of else_node, error may be set.

error

(flyteidl.core.Error) An error to throw in case none of the branches were taken.

required.

Only one of else_node, error may be set.

flyteidl.core.BranchNode

[flyteidl.core.BranchNode proto]

BranchNode is a special node that alter the flow of the workflow graph. It allows the control flow to branch at runtime based on a series of conditions that get evaluated on various parameters (e.g. inputs, primtives).

{
  "if_else": "{...}"
}
if_else

(flyteidl.core.IfElseBlock) required

flyteidl.core.TaskNode

[flyteidl.core.TaskNode proto]

Refers to the task that the Node is to execute.

{
  "reference_id": "{...}"
}
reference_id

(flyteidl.core.Identifier) A globally unique identifier for the task.

flyteidl.core.WorkflowNode

[flyteidl.core.WorkflowNode proto]

Refers to a the workflow the node is to execute.

{
  "launchplan_ref": "{...}",
  "sub_workflow_ref": "{...}"
}
launchplan_ref

(flyteidl.core.Identifier) A globally unique identifier for the launch plan.

Only one of launchplan_ref, sub_workflow_ref may be set.

sub_workflow_ref

(flyteidl.core.Identifier) Reference to a subworkflow, that should be defined with the compiler context

Only one of launchplan_ref, sub_workflow_ref may be set.

flyteidl.core.NodeMetadata

[flyteidl.core.NodeMetadata proto]

Defines extra information about the Node.

{
  "name": "...",
  "timeout": "{...}",
  "retries": "{...}",
  "interruptible": "..."
}
name

(string) A friendly name for the Node

timeout

(google.protobuf.Duration) The overall timeout of a task.

retries

(flyteidl.core.RetryStrategy) Number of retries per task.

interruptible

(bool) Identify whether node is interruptible

flyteidl.core.Alias

[flyteidl.core.Alias proto]

Links a variable to an alias.

{
  "var": "...",
  "alias": "..."
}
var

(string) Must match one of the output variable names on a node.

alias

(string) A workflow-level unique alias that downstream nodes can refer to in their input.

flyteidl.core.Node

[flyteidl.core.Node proto]

A Workflow graph Node. One unit of execution in the graph. Each node can be linked to a Task, a Workflow or a branch node.

{
  "id": "...",
  "metadata": "{...}",
  "inputs": [],
  "upstream_node_ids": [],
  "output_aliases": [],
  "task_node": "{...}",
  "workflow_node": "{...}",
  "branch_node": "{...}"
}
id

(string) A workflow-level unique identifier that identifies this node in the workflow. “inputs” and “outputs” are reserved node ids that cannot be used by other nodes.

metadata

(flyteidl.core.NodeMetadata) Extra metadata about the node.

inputs

(flyteidl.core.Binding) Specifies how to bind the underlying interface’s inputs. All required inputs specified in the underlying interface must be fullfilled.

upstream_node_ids

(string) optional Specifies execution depdendency for this node ensuring it will only get scheduled to run after all its upstream nodes have completed. This node will have an implicit depdendency on any node that appears in inputs field.

output_aliases

(flyteidl.core.Alias) optional. A node can define aliases for a subset of its outputs. This is particularly useful if different nodes need to conform to the same interface (e.g. all branches in a branch node). Downstream nodes must refer to this nodes outputs using the alias if one’s specified.

task_node

(flyteidl.core.TaskNode) Information about the Task to execute in this node.

Information about the target to execute in this node.

Only one of task_node, workflow_node, branch_node may be set.

workflow_node

(flyteidl.core.WorkflowNode) Information about the Workflow to execute in this mode.

Information about the target to execute in this node.

Only one of task_node, workflow_node, branch_node may be set.

branch_node

(flyteidl.core.BranchNode) Information about the branch node to evaluate in this node.

Information about the target to execute in this node.

Only one of task_node, workflow_node, branch_node may be set.

flyteidl.core.WorkflowMetadata

[flyteidl.core.WorkflowMetadata proto]

This is workflow layer metadata. These settings are only applicable to the workflow as a whole, and do not percolate down to child entities (like tasks) launched by the workflow.

{
  "quality_of_service": "{...}",
  "on_failure": "..."
}
quality_of_service

(flyteidl.core.QualityOfService) Indicates the runtime priority of workflow executions.

on_failure

(flyteidl.core.WorkflowMetadata.OnFailurePolicy) Defines how the system should behave when a failure is detected in the workflow execution.

Enum flyteidl.core.WorkflowMetadata.OnFailurePolicy

[flyteidl.core.WorkflowMetadata.OnFailurePolicy proto]

Failure Handling Strategy

FAIL_IMMEDIATELY

(DEFAULT) ⁣FAIL_IMMEDIATELY instructs the system to fail as soon as a node fails in the workflow. It’ll automatically abort all currently running nodes and clean up resources before finally marking the workflow executions as failed.

FAIL_AFTER_EXECUTABLE_NODES_COMPLETE

⁣FAIL_AFTER_EXECUTABLE_NODES_COMPLETE instructs the system to make as much progress as it can. The system will not alter the dependencies of the execution graph so any node that depend on the failed node will not be run. Other nodes that will be executed to completion before cleaning up resources and marking the workflow execution as failed.

flyteidl.core.WorkflowMetadataDefaults

[flyteidl.core.WorkflowMetadataDefaults proto]

The difference between these settings and the WorkflowMetadata ones is that these are meant to be passed down to a workflow’s underlying entities (like tasks). For instance, ‘interruptible’ has no meaning at the workflow layer, it is only relevant when a task executes. The settings here are the defaults that are passed to all nodes unless explicitly overridden at the node layer. If you are adding a setting that applies to both the Workflow itself, and everything underneath it, it should be added to both this object and the WorkflowMetadata object above.

{
  "interruptible": "..."
}
interruptible

(bool) Whether child nodes of the workflow are interruptible.

flyteidl.core.WorkflowTemplate

[flyteidl.core.WorkflowTemplate proto]

Flyte Workflow Structure that encapsulates task, branch and subworkflow nodes to form a statically analyzable, directed acyclic graph.

{
  "id": "{...}",
  "metadata": "{...}",
  "interface": "{...}",
  "nodes": [],
  "outputs": [],
  "failure_node": "{...}",
  "metadata_defaults": "{...}"
}
id

(flyteidl.core.Identifier) A globally unique identifier for the workflow.

metadata

(flyteidl.core.WorkflowMetadata) Extra metadata about the workflow.

interface

(flyteidl.core.TypedInterface) Defines a strongly typed interface for the Workflow. This can include some optional parameters.

nodes

(flyteidl.core.Node) A list of nodes. In addition, “globals” is a special reserved node id that can be used to consume workflow inputs.

outputs

(flyteidl.core.Binding) A list of output bindings that specify how to construct workflow outputs. Bindings can pull node outputs or specify literals. All workflow outputs specified in the interface field must be bound in order for the workflow to be validated. A workflow has an implicit dependency on all of its nodes to execute successfully in order to bind final outputs. Most of these outputs will be Binding’s with a BindingData of type OutputReference. That is, your workflow can just have an output of some constant (Output(5)), but usually, the workflow will be pulling outputs from the output of a task.

failure_node

(flyteidl.core.Node) optional A catch-all node. This node is executed whenever the execution engine determines the workflow has failed. The interface of this node must match the Workflow interface with an additional input named “error” of type pb.lyft.flyte.core.Error.

metadata_defaults

(flyteidl.core.WorkflowMetadataDefaults) workflow defaults