Product Documentation

Appearance

All platforms

Object Storage Options ​

The following options are used in ybload, ybunload, ybbackup, ybrestore, and ybbackupctl commands. These options specify connectivity, credentials, and configuration settings for object storage systems, including Azure Blob Storage and S3 object stores. The usage for these options depends on the object store provider.

For more details about supported object storage authentication methods and URIs, see Loading from Amazon S3 and Loading from Azure Blob Storage.

--object-store-anonymous, --no-object-store-anonymous

Connect to the object store anonymously, instead of providing a credential. For example, you can connect anonymously when you are accessing an AWS S3 bucket that is publicly available. You cannot specify --object-store-anonymous and credential values in the same command.

--object-store-credential STRING

The credential to use when connecting to an object store.

For AWS S3, this credential corresponds to your AWS Secret Access Key.

For Azure, this credential is a storage account access key or a generated SAS token.

--object-store-endpoint VALUE

The endpoint URI to use when connecting to an object store.

For AWS S3, if you specify the --object-store-region option, you can omit the --object-store-endpoint option. An example of an AWS S3 endpoint is:

https://s3.us-east-2.amazonaws.com

An example of an on-premises S3-compatible endpoint is:

http://ybload1.yellowbrick.io:9000

where the host name is a system where the S3 provider is installed. (The port number varies by S3 provider.)

For Azure, if you specify the --object-store-identity option, you can omit the --object-store-endpoint option. An example of an Azure endpoint is:

https://ybbobr.blob.core.windows.net

where ybbobr is the storage account name.

You can only access one object store per operation. For example, you cannot specify connection details for both an Azure client and an S3 client in the same command, and you cannot load or unload data using multiple Azure endpoints or storage accounts.

--object-store-identity STRING

The identity to use when connecting to an object store.

For AWS S3, this identity corresponds to your AWS Access Key ID. For example:

ABCD1EF2GHIJKLMN3OPQ

For Azure, this identity is a storage account name. If you specify this option, you can omit the --object-store-endpoint option because the storage account name is always part of the standard endpoint pattern (https://<storage account>.blob.core.windows.net). For example:

https://ybbobr.blob.core.windows.net

where ybbobr is the storage account.

--object-store-profile STRING

The credential profile to use when connecting to an object store. This option overrides the settings for --object-store-identity and --object-store-credential.

This option is not supported for Azure object stores.

--object-store-provider-config JAVA_PROPERTIES_FILE

The name of a Java properties file that contains provider-specific configuration settings for an object store. For example: ybbobr_az.properties or ybbobr_s3.properties. Default: {}.

--object-store-region STRING

The region to use when connecting to an object store.

For AWS S3, an example of a region is us-east-2.

For on-premises, S3-compatible object storage providers, a region value must be specified even if the provider does not support or require regions. You can specify any string for the region, such as noregion or dummy. Alternatively, you can set the region in an AWS configuration file:

~/.aws/config

[default]
region = dummy

The region option is not supported for Azure Blob storage.

Java Properties File ​

If you choose to specify object storage options in a properties file (--object-store-provider-config option), drop the --object-store- prefix from the option names. You can either add a yb.object_store. prefix to the entries in the file, or you can specify the entries without a prefix.

Do not quote any of the values in the file. For example, specify us-east-2, not 'us-east-2'.

AWS S3

For example:

$ more ybbobrS3.properties

yb.object_store.region = us-east-2
yb.object_store.identity = ABCD1EF2GHIJKLMN3OPQ
yb.object_store.credential = ****************************************

Alternatively, you can specify:

region = us-east-2
identity = ABCD1EF2GHIJKLMN3OPQ
credential = ****************************************

This properties file is distinct from the ~/.aws/* configuration files that the S3 SDK consults.

Azure Blob Storage, Azure Data Lake Gen 2

endpoint = https://mystorageaccount.blob.core.windows.net
identity = mystorageaccount
credential = myStorageAccountKey

For example:

$ more ybbobrAz.properties

yb.object_store.endpoint = https://ybbobr.blob.core.windows.net
yb.object_store.identity = ybbobr
yb.object_store.credential = ****************************************

S3 Pass-Through Options ​

The following pass-through options are supported for S3 clients. You can specify them either in a Java properties file (referenced by --object-store-provider-config) or as URI parameters in the first S3 URI passed to the ybload command. (The defaults listed here are as defined by the AWS Java SDK 2.7.31.)

accelerateModeEnabled
Defaults to false.
checksumValidationEnabled
Defaults to true.
chunkedEncodingEnabled
Defaults to true.
dualStackEnabled
Defaults to false.