Common configuration across all backend components

All backend Flyte components are written in GoLang and use flytestdlib. Flytestdlib uses the common configuration system and is shared across the entire flyte-backend. The configuration used to configure the logging and metadata storage systems.

Logger Configuration

Logger can be configured using a common configuration that is shared across all components. The variables available for configuration are described in the Config struct in the source code. Documentation for this is available @ godoc:flytestdlib/logger.

Config Val

Type

Description

logger.formatter.type

string

Sets logging format type. (default “json”)

logger.level

int

Sets the minimum logging level. (default 4)

logger.mute

Bool

Mutes all logs regardless of severity. Intended for benchmarks/tests only.

logger.show-source

Bool

Includes source code location in logs.

Metadata storage configuration

Metadata Storage can be configured using a common configuration that is shared across all components. The variables available for configuration are described in the Config struct in the code. Documentation for this is available @ godoc:flytestdlib/storage

Config Val

Type

Description

storage.cache.max_size_mbs

int

Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0, cache is not used

storage.cache.target_gc_percent

int

Sets the garbage collection target percentage.

storage.connection.access-key

string

Access key to use. Only required when authtype is set to accesskey.

storage.connection.auth-type

string

Auth Type to use [iam, accesskey]. (default “iam”)

storage.connection.disable-ssl

Disables

SSL connection. Should only be used for development.

storage.connection.endpoint

string

URL for storage client to connect to.

storage.connection.region

string

Region to connect to. (default “us-east-1”)

storage.connection.secret-key

string

Secret to use when accesskey is set.

storage.container

string

Initial container to create -if it doesn’t exist-.’

storage.limits.maxDownloadMBs

int

Maximum allowed download size (in MBs) per call. (default 2)

storage.type

string

Sets the type of storage to configure [s3/minio/local/mem]. (default “s3”)