flytekit.configuration package

Submodules

flytekit.configuration.auth module

flytekit.configuration.auth.ASSUMABLE_IAM_ROLE = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is the role the SDK will use by default to execute workflows. For example, in AWS this should be an IAM role string.

flytekit.configuration.auth.KUBERNETES_SERVICE_ACCOUNT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is the kubernetes service account that will be passed to workflow executions.

flytekit.configuration.auth.RAW_OUTPUT_DATA_PREFIX = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is not output metadata but rather where users can specify an S3 or gcs path for offloaded data like blobs and schemas.

The reason this setting is in this file is because it’s inextricably tied to a workflow’s role or service account, since that is what ultimately gives the tasks the ability to write to certain buckets.

flytekit.configuration.aws module

flytekit.configuration.common module

class flytekit.configuration.common.FlyteBoolConfigurationEntry(section, key, default=None, validator=None, fallback=None)[source]

Bases: flytekit.configuration.common._FlyteConfigurationEntry

class flytekit.configuration.common.FlyteConfigurationFile(location=None)[source]

Bases: object

get_bool(section, key, default=None)[source]
Parameters
  • section (Text) –

  • key (Text) –

  • default (bool) –

Return type

bool

get_int(section, key, default=None)[source]
Parameters
  • section (Text) –

  • key (Text) –

  • default (int) –

Return type

int

get_string(section, key, default=None)[source]
Parameters
  • section (Text) –

  • key (Text) –

  • default (Text) –

Return type

Text

reset_config(location)[source]
Parameters

location (Text) –

class flytekit.configuration.common.FlyteIntegerConfigurationEntry(section, key, default=None, validator=None, fallback=None)[source]

Bases: flytekit.configuration.common._FlyteConfigurationEntry

class flytekit.configuration.common.FlyteRequiredBoolConfigurationEntry(section, key, validator=None)[source]

Bases: flytekit.configuration.common._FlyteRequiredConfigurationEntry

class flytekit.configuration.common.FlyteRequiredIntegerConfigurationEntry(section, key, validator=None)[source]

Bases: flytekit.configuration.common._FlyteRequiredConfigurationEntry

class flytekit.configuration.common.FlyteRequiredStringConfigurationEntry(section, key, validator=None)[source]

Bases: flytekit.configuration.common._FlyteRequiredConfigurationEntry

class flytekit.configuration.common.FlyteRequiredStringListConfigurationEntry(section, key, validator=None)[source]

Bases: flytekit.configuration.common._FlyteRequiredConfigurationEntry

class flytekit.configuration.common.FlyteStringConfigurationEntry(section, key, default=None, validator=None, fallback=None)[source]

Bases: flytekit.configuration.common._FlyteConfigurationEntry

class flytekit.configuration.common.FlyteStringListConfigurationEntry(section, key, default=None, validator=None, fallback=None)[source]

Bases: flytekit.configuration.common._FlyteConfigurationEntry

flytekit.configuration.common.format_section_key(section, key)[source]
Parameters
  • section (Text) –

  • key (Text) –

Return type

Text

flytekit.configuration.creds module

flytekit.configuration.creds.AUTHORIZATION_METADATA_KEY = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

The authorization metadata key used for passing access tokens in gRPC requests. Traditionally this value is ‘authorization’ however it is made configurable.

flytekit.configuration.creds.AUTH_MODE = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

The auth mode defines the behavior used to request and refresh credentials. The currently supported modes include: - ‘standard’ This uses the pkce-enhanced authorization code flow by opening a browser window to initiate credentials

access.

  • ‘basic’ This uses cert-based auth in which the end user enters his/her username and password and public key encryption

    is used to facilitate authentication.

  • None: No auth will be attempted.

flytekit.configuration.creds.CLIENT_CREDENTIALS_SCOPE = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

Used for basic auth, which is automatically called during pyflyte. This is the scope that will be requested. Because there is no user explicitly in this auth flow, certain IDPs require a custom scope for basic auth in the configuration of the authorization server.

flytekit.configuration.creds.CLIENT_CREDENTIALS_SECRET = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

Used for basic auth, which is automatically called during pyflyte. This will allow the Flyte engine to read the password directly from the environment variable. Note that this is less secure! Please only use this if mounting the secret as a file is impossible.

flytekit.configuration.creds.CLIENT_ID = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is the public identifier for the app which handles authorization for a Flyte deployment. More details here: https://www.oauth.com/oauth2-servers/client-registration/client-id-secret/.

flytekit.configuration.creds.REDIRECT_URI = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is the callback uri registered with the app which handles authorization for a Flyte deployment. Please note the hardcoded port number. Ideally we would not do this, but some IDPs do not allow wildcards for the URL, which means we have to use the same port every time. This is the only reason this is a configuration option, otherwise, we’d just hardcode the callback path as a constant. FYI, to see if a given port is already in use, run sudo lsof -i :<port> if on a Linux system. More details here: https://www.oauth.com/oauth2-servers/redirect-uris/.

flytekit.configuration.gcp module

flytekit.configuration.internal module

flytekit.configuration.internal.look_up_version_from_image_tag(tag)[source]
Looks up the image tag from environment variable (should be set from the Dockerfile).

FLYTE_INTERNAL_IMAGE should be the environment variable.

This function is used when registering tasks/workflows with Admin. When using the canonical Python-based development cycle, the version that is used to register workflows and tasks with Admin should be the version of the image itself, which should ideally be something unique like the sha of the latest commit.

Parameters

tag (Text) – e.g. somedocker.com/myimage:someversion123

Return type

Text

flytekit.configuration.platform module

flytekit.configuration.platform.AUTH = <flytekit.configuration.common.FlyteBoolConfigurationEntry object>

This config setting should not normally be filled in. Whether or not an admin server requires authentication should be something published by the admin server itself (typically by returning a 401). However, to help with migration, this config object is here to force the SDK to attempt the auth flow even without prompting by Admin.

flytekit.configuration.platform.HTTP_URL = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not starting with either http or https, this setting should begin with // as per the urlparse library and https://tools.ietf.org/html/rfc1808.html, otherwise the netloc will not be properly parsed.

Currently the only use-case for this configuration setting is for Auth discovery. This setting supports the case where Flyte Admin’s gRPC and HTTP points are deployed on different ports.

flytekit.configuration.resources module

flytekit.configuration.resources.DEFAULT_CPU_LIMIT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this limit will be applied as the default. Follows Kubernetes CPU request/limit format.

flytekit.configuration.resources.DEFAULT_CPU_REQUEST = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this request will be applied as the default. Follows Kubernetes CPU request/limit format.

flytekit.configuration.resources.DEFAULT_GPU_LIMIT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this limit will be applied as the default. Follows Kubernetes GPU request/limit format.

flytekit.configuration.resources.DEFAULT_GPU_REQUEST = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this request will be applied as the default. Follows Kubernetes GPU request/limit format.

flytekit.configuration.resources.DEFAULT_MEMORY_LIMIT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this limit will be applied as the default. Follows Kubernetes memory request/limit format.

flytekit.configuration.resources.DEFAULT_MEMORY_REQUEST = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this request will be applied as the default. Follows Kubernetes memory request/limit format.

flytekit.configuration.resources.DEFAULT_STORAGE_LIMIT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this limit will be applied as the default. Follows Kubernetes storage request/limit format.

flytekit.configuration.resources.DEFAULT_STORAGE_REQUEST = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

If not specified explicitly when constructing a task, this request will be applied as the default. Follows Kubernetes storage request/limit format.

flytekit.configuration.sdk module

flytekit.configuration.sdk.EXECUTION_ENGINE = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is a comma-delimited list of package strings, in order, for resolving execution behavior.

TODO: Explain how this would be used to extend the SDK

flytekit.configuration.sdk.LAUNCH_PLAN_NAME_FORMAT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is a Python format string which the SDK will use to generate names for launch plans. Any template portion of the string can only include ‘module’ or ‘name’. So ‘{name}’ is valid, but ‘{key}’ is not. If not specified, we fall back to the configuration for flytekit.configuration.sdk.NAME_FORMAT

flytekit.configuration.sdk.LOCAL_SANDBOX = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is the path where SDK will place files during local executions and testing. The SDK will not automatically clean up data in these directories.

flytekit.configuration.sdk.LOGGING_LEVEL = <flytekit.configuration.common.FlyteIntegerConfigurationEntry object>

This is the default logging level for the Python logging library and will be set before user code runs. Note that this configuration is special in that it is a runtime setting, not a compile time setting. This is the only runtime option in this file.

flytekit.configuration.sdk.NAME_FORMAT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is a Python format string which the SDK will use to generate names for discovered entities. The default is ‘{module}.{name}’ which will result in strings like ‘package.module.name’. Any template portion of the string can only include ‘module’ or ‘name’. So ‘{name}’ is valid, but ‘{key}’ is not.

flytekit.configuration.sdk.PARQUET_ENGINE = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is the parquet engine to use when reading data from parquet files.

flytekit.configuration.sdk.ROLE = <flytekit.configuration.common.FlyteRequiredStringConfigurationEntry object>

This is the role the SDK will use by default to execute workflows. For example, in AWS this should be an IAM role string.

flytekit.configuration.sdk.SDK_PYTHON_VENV = <flytekit.configuration.common.FlyteStringListConfigurationEntry object>

This is a list of commands/args which will be prefixed to the entrypoint command by SDK.

flytekit.configuration.sdk.TASK_NAME_FORMAT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is a Python format string which the SDK will use to generate names for tasks. Any template portion of the string can only include ‘module’ or ‘name’. So ‘{name}’ is valid, but ‘{key}’ is not. If not specified, we fall back to the configuration for flytekit.configuration.sdk.NAME_FORMAT

flytekit.configuration.sdk.TYPE_ENGINES = <flytekit.configuration.common.FlyteStringListConfigurationEntry object>

This is a comma-delimited list of package strings, in order, for resolving type behavior.

TODO: Explain how this would be used to extend the SDK

flytekit.configuration.sdk.WORKFLOW_NAME_FORMAT = <flytekit.configuration.common.FlyteStringConfigurationEntry object>

This is a Python format string which the SDK will use to generate names for workflows. Any template portion of the string can only include ‘module’ or ‘name’. So ‘{name}’ is valid, but ‘{key}’ is not. If not specified, we fall back to the configuration for flytekit.configuration.sdk.NAME_FORMAT

flytekit.configuration.sdk.WORKFLOW_PACKAGES = <flytekit.configuration.common.FlyteStringListConfigurationEntry object>

This is a comma-delimited list of packages that SDK tools will use to discover entities for the purpose of registration and execution of entities.

flytekit.configuration.statsd module

Module contents

class flytekit.configuration.TemporaryConfiguration(new_config_path, internal_overrides=None)[source]

Bases: object

flytekit.configuration.set_flyte_config_file(config_file_path)[source]
Parameters

config_file_path (Text) –