tasks.proto

For interruptible we will populate it at the node level but require it be part of TaskMetadata for a user to set the value. We are using oneof instead of bool because otherwise we would be unable to distinguish between value being set by the user or defaulting to false. The logic of handling precedence will be done as part of flytepropeller.

flyteidl.core.Resources

[flyteidl.core.Resources proto]

A customizable interface to convey resources requested for a container. This can be interpretted differently for different container engines.

{
  "requests": [],
  "limits": []
}
requests

(flyteidl.core.Resources.ResourceEntry) The desired set of resources requested. ResourceNames must be unique within the list.

limits

(flyteidl.core.Resources.ResourceEntry) Defines a set of bounds (e.g. min/max) within which the task can reliably run. ResourceNames must be unique within the list.

flyteidl.core.Resources.ResourceEntry

[flyteidl.core.Resources.ResourceEntry proto]

Encapsulates a resource name and value.

{
  "name": "...",
  "value": "..."
}
name

(flyteidl.core.Resources.ResourceName) Resource name.

value

(string) Value must be a valid k8s quantity. See https://github.com/kubernetes/apimachinery/blob/master/pkg/api/resource/quantity.go#L30-L80

Enum flyteidl.core.Resources.ResourceName

[flyteidl.core.Resources.ResourceName proto]

Known resource names.

UNKNOWN

(DEFAULT)

CPU

GPU

MEMORY

STORAGE

flyteidl.core.RuntimeMetadata

[flyteidl.core.RuntimeMetadata proto]

Runtime information. This is losely defined to allow for extensibility.

{
  "type": "...",
  "version": "...",
  "flavor": "..."
}
type

(flyteidl.core.RuntimeMetadata.RuntimeType) Type of runtime.

version

(string) Version of the runtime. All versions should be backward compatible. However, certain cases call for version checks to ensure tighter validation or setting expectations.

flavor

(string) optional It can be used to provide extra information about the runtime (e.g. python, golang… etc.).

Enum flyteidl.core.RuntimeMetadata.RuntimeType

[flyteidl.core.RuntimeMetadata.RuntimeType proto]

OTHER

(DEFAULT)

FLYTE_SDK

flyteidl.core.TaskMetadata

[flyteidl.core.TaskMetadata proto]

Task Metadata

{
  "discoverable": "...",
  "runtime": "{...}",
  "timeout": "{...}",
  "retries": "{...}",
  "discovery_version": "...",
  "deprecated_error_message": "...",
  "interruptible": "..."
}
discoverable

(bool) Indicates whether the system should attempt to lookup this task’s output to avoid duplication of work.

runtime

(flyteidl.core.RuntimeMetadata) Runtime information about the task.

timeout

(google.protobuf.Duration) The overall timeout of a task including user-triggered retries.

retries

(flyteidl.core.RetryStrategy) Number of retries per task.

discovery_version

(string) Indicates a logical version to apply to this task for the purpose of discovery.

deprecated_error_message

(string) If set, this indicates that this task is deprecated. This will enable owners of tasks to notify consumers of the ending of support for a given task.

interruptible

(bool) Identify whether task is interruptible

flyteidl.core.TaskTemplate

[flyteidl.core.TaskTemplate proto]

A Task structure that uniquely identifies a task in the system Tasks are registered as a first step in the system.

{
  "id": "{...}",
  "type": "...",
  "metadata": "{...}",
  "interface": "{...}",
  "custom": "{...}",
  "container": "{...}"
}
id

(flyteidl.core.Identifier) Auto generated taskId by the system. Task Id uniquely identifies this task globally.

type

(string) A predefined yet extensible Task type identifier. This can be used to customize any of the components. If no extensions are provided in the system, Flyte will resolve the this task to its TaskCategory and default the implementation registered for the TaskCategory.

metadata

(flyteidl.core.TaskMetadata) Extra metadata about the task.

interface

(flyteidl.core.TypedInterface) A strongly typed interface for the task. This enables others to use this task within a workflow and gauarantees compile-time validation of the workflow to avoid costly runtime failures.

custom

(google.protobuf.Struct) Custom data about the task. This is extensible to allow various plugins in the system.

container

(flyteidl.core.Container) Known target types that the system will guarantee plugins for. Custom SDK plugins are allowed to set these if needed. If no corresponding execution-layer plugins are found, the system will default to handling these using built-in handlers.

flyteidl.core.ContainerPort

[flyteidl.core.ContainerPort proto]

Defines port properties for a container.

{
  "container_port": "..."
}
container_port

(uint32) Number of port to expose on the pod’s IP address. This must be a valid port number, 0 < x < 65536.

flyteidl.core.Container

[flyteidl.core.Container proto]

{
  "image": "...",
  "command": [],
  "args": [],
  "resources": "{...}",
  "env": [],
  "config": [],
  "ports": [],
  "data_config": "{...}"
}
image

(string) Container image url. Eg: docker/redis:latest

command

(string) Command to be executed, if not provided, the default entrypoint in the container image will be used.

args

(string) These will default to Flyte given paths. If provided, the system will not append known paths. If the task still needs flyte’s inputs and outputs path, add $(FLYTE_INPUT_FILE), $(FLYTE_OUTPUT_FILE) wherever makes sense and the system will populate these before executing the container.

resources

(flyteidl.core.Resources) Container resources requirement as specified by the container engine.

env

(flyteidl.core.KeyValuePair) Environment variables will be set as the container is starting up.

config

(flyteidl.core.KeyValuePair) Allows extra configs to be available for the container. TODO: elaborate on how configs will become available.

ports

(flyteidl.core.ContainerPort) Ports to open in the container. This feature is not supported by all execution engines. (e.g. supported on K8s but not supported on AWS Batch) Only K8s

data_config

(flyteidl.core.DataLoadingConfig) BETA: Optional configuration for DataLoading. If not specified, then default values are used. This makes it possible to to run a completely portable container, that uses inputs and outputs only from the local file-system and without having any reference to flyteidl. This is supported only on K8s at the moment. If data loading is enabled, then data will be mounted in accompanying directories specified in the DataLoadingConfig. If the directories are not specified, inputs will be mounted onto and outputs will be uploaded from a pre-determined file-system path. Refer to the documentation to understand the default paths. Only K8s

flyteidl.core.IOStrategy

[flyteidl.core.IOStrategy proto]

Strategy to use when dealing with Blob, Schema, or multipart blob data (large datasets)

{
  "download_mode": "...",
  "upload_mode": "..."
}
download_mode

(flyteidl.core.IOStrategy.DownloadMode) Mode to use to manage downloads

upload_mode

(flyteidl.core.IOStrategy.UploadMode) Mode to use to manage uploads

Enum flyteidl.core.IOStrategy.DownloadMode

[flyteidl.core.IOStrategy.DownloadMode proto]

Mode to use for downloading

DOWNLOAD_EAGER

(DEFAULT) ⁣All data will be downloaded before the main container is executed

DOWNLOAD_STREAM

⁣Data will be downloaded as a stream and an End-Of-Stream marker will be written to indicate all data has been downloaded. Refer to protocol for details

DO_NOT_DOWNLOAD

⁣Large objects (offloaded) will not be downloaded

Enum flyteidl.core.IOStrategy.UploadMode

[flyteidl.core.IOStrategy.UploadMode proto]

Mode to use for uploading

UPLOAD_ON_EXIT

(DEFAULT) ⁣All data will be uploaded after the main container exits

UPLOAD_EAGER

⁣Data will be uploaded as it appears. Refer to protocol specification for details

DO_NOT_UPLOAD

⁣Data will not be uploaded, only references will be written

flyteidl.core.DataLoadingConfig

[flyteidl.core.DataLoadingConfig proto]

This configuration allows executing raw containers in Flyte using the Flyte CoPilot system. Flyte CoPilot, eliminates the needs of flytekit or sdk inside the container. Any inputs required by the users container are side-loaded in the input_path Any outputs generated by the user container - within output_path are automatically uploaded.

{
  "enabled": "...",
  "input_path": "...",
  "output_path": "...",
  "format": "...",
  "io_strategy": "{...}"
}
enabled

(bool) Flag enables DataLoading Config. If this is not set, data loading will not be used!

input_path

(string) File system path (start at root). This folder will contain all the inputs exploded to a separate file. Example, if the input interface needs (x: int, y: blob, z: multipart_blob) and the input path is “/var/flyte/inputs”, then the file system will look like /var/flyte/inputs/inputs.<metadata format dependent -> .pb .json .yaml> -> Format as defined previously. The Blob and Multipart blob will reference local filesystem instead of remote locations /var/flyte/inputs/x -> X is a file that contains the value of x (integer) in string format /var/flyte/inputs/y -> Y is a file in Binary format /var/flyte/inputs/z/… -> Note Z itself is a directory More information about the protocol - refer to docs #TODO reference docs here

output_path

(string) File system path (start at root). This folder should contain all the outputs for the task as individual files and/or an error text file

format

(flyteidl.core.DataLoadingConfig.LiteralMapFormat) In the inputs folder, there will be an additional summary/metadata file that contains references to all files or inlined primitive values. This format decides the actual encoding for the data. Refer to the encoding to understand the specifics of the contents and the encoding

io_strategy

(flyteidl.core.IOStrategy)

Enum flyteidl.core.DataLoadingConfig.LiteralMapFormat

[flyteidl.core.DataLoadingConfig.LiteralMapFormat proto]

LiteralMapFormat decides the encoding format in which the input metadata should be made available to the containers. If the user has access to the protocol buffer definitions, it is recommended to use the PROTO format. JSON and YAML do not need any protobuf definitions to read it All remote references in core.LiteralMap are replaced with local filesystem references (the data is downloaded to local filesystem)

JSON

(DEFAULT) ⁣JSON / YAML for the metadata (which contains inlined primitive values). The representation is inline with the standard json specification as specified - https://www.json.org/json-en.html

YAML

PROTO

⁣Proto is a serialized binary of core.LiteralMap defined in flyteidl/core